Leverage the power of Artificial Intelligence to enhance & restyle texture maps on 3D models.

Microsoft
Self-Initiated Project
Role: Experience Design Lead

Remix 3D is an online community for sharing 3D models, focused on user-generated content and has an embedded experience in Paint 3D, PowerPoint, and even Minecraft. It's part of an effort to bring the awesomeness of 3D expression to anyone and everyone.

See an overview of this product here and take a look at a case study on a feature I like here.

Neural style transfer is a deep learning algorithm that has been a huge success in the world of 2D art (eg. deepart.io & Prisma).

My team and I wanted to combine the power of AI and the power of 3D!

The prototype we built has the ability to give a 3D model (hosted on Remix 3D) an artistic paint job in the style of any image.

What this means in Remix 3D…

When a user publishes their model, they get to choose one or more additional styles to publish it in. Viewers can switch between styles as they view the model. Have an idea for a style that would look really cool on an the model someone published? Upload your own style image to see how it looks, and publish your new styled version as a remix!

Technology we used:

Existing implementation of neural style transfer based on original 2015 paper from Gatys et. al
Fast neural style transfer implementation for pre selected styles.
An NC-series Azure VM with an NVIDIA K80 GPU

Future work:

Productionize this feature, and integrate it into the right place in our 3D app ecosystem
Tweak the algorithm to minimize artifacts due to the 2D texture seams
Explore the possibilities of neural style transfer of the geometry of 3D models!