At GTC, Nvidia’s research department has presented the new deep learning model — GANverse3D — capable of creating interactive 3D objects through standard 2D images in just a few seconds.
The GANverse3D application, developed at Nvidia’s AI Research Lab in Toronto, was built using what’s called Generative Adversarial Network (GAN), which can convert static photos into dynamic 3D models and can be imported into the Nvidia’s collaboration platform, Omniverse for use in various 3D development software.
GANverse3D is still in the initial development stage. AI training has been carried out for vehicles, birds, horses, and other objects. In particular, the vehicle part uses up to 55,000 photos for training. Users only need to import the photos into the application, and the program will use AI to inferring and predicting the three-dimensional information. The image is automatically converted into a 3D mesh, and a texture map is generated. It can also generate a segmentation mask, which can distinguish different parts of the object.
The system is now available within Nvidia Omniverse, which works on the company’s RTX graphics cards.
According to NVIDIA, game creators, architects, or designers can use this system to test new ideas and visualize prototypes before creating their final products. Also including simulation in complex virtual environments with real-time ray tracing.
For more technical details of GANverse3D, you can refer to papers such as “Image Gans Meet Differentiable Rendering For Inverse Graphics And Interpretable 3D Neural Rendering” and “Datasetgan: Efficient Labeled Data Factory With Minimal Human Effort.”
