Neuralange: How Nvidia’s AI Model Creates 3D Scenes from 2D Videos

Introduction

Nvidia, the leading company in artificial intelligence (AI) and graphics processing units (GPUs), has recently unveiled a new AI model that can turn 2D video clips into detailed 3D structures. This new model is inspired by the famous sculptor and painter Michelangelo, who created stunning, life-like visions from blocks of marble.

This new model is based on a paper by Nvidia Research and Johns Hopkins University, which has been accepted to the Conference on Computer Vision and Pattern Recognition (CVPR) 20231. The paper provides more technical details and evaluations of Neuralange’s performance and limitations. This new model is called 'Neuralange'.

What is Neuralange?

Neuralange is an AI model that uses neural rendering to reconstruct 3D scenes from 2D video clips. Neural rendering is a technique that combines computer graphics and deep learning to synthesize realistic images and videos. Neuralange adopts instant neural graphics primitives, the technology behind Nvidia’s Instant NeRF, to capture the finer details and textures of complex materials, such as roof shingles, panes of glass, and smooth marble. Neuralange can also handle repetitive texture patterns, homogenous colors, and strong color variations, which are challenging for previous methods.

Neuralange is not just a useful tool for creative professionals, but also a novel approach to 3D reconstruction research. Neuralange is one of the first models that can reconstruct large-scale scenes and small objects with high fidelity using neural rendering. 

Key Features of Neuralange

Neuralange has several key features that make it stand out from other 3D reconstruction methods:

• It can reconstruct large-scale scenes, such as building interiors and exteriors, as well as small objects, such as statues and trucks.
• It can preserve the original lighting and shading effects of the 2D video clips, making the 3D structures more realistic and consistent.
• It can generate high-fidelity 3D models with intricate details and textures that can be imported into design applications for further editing and refinement.
• It can work with any 2D video clip filmed from various angles, without requiring special cameras or sensors.
• It can even generate lifelike virtual replicas of buildings, sculptures, and other real-world objects from 2D footage captured by smartphones.

Use Cases of Neuralange

Neuralange has many potential applications in various domains, such as:

• Art: Neuralange can help artists create digital replicas of their physical artworks or sculptures or generate new 3D artworks from 2D sketches or photos.
• Video game development: Neuralange can help game developers create immersive virtual environments and characters from real-world scenes and objects.
• Robotics: Neuralange can help robotics researchers build accurate 3D maps and models of their surroundings for navigation and manipulation tasks.
• Industrial digital twins: Neuralange can help engineers create virtual copies of physical assets, such as buildings, factories, or machines, for monitoring and optimization purposes.

How does Neuralange work?

Neuralange works by following these steps:

1. It selects several frames from the 2D video clip that capture different viewpoints of the object or scene.
2. It determines the camera position of each frame using a neural network.
3. It creates a rough 3D representation of the scene using instant neural graphics primitives, which are learned functions that map 3D coordinates to colors and densities.
4. It optimizes the render using another neural network that refines the details and textures of the 3D structure.

Neuralange: A Visual Showcase of 3D Reconstruction

One of the most impressive aspects of Neuralange is its ability to reconstruct 3D scenes with high visual quality and realism. To showcase this, Nvidia Research has created a video comparison between the original sculptures by Michelangelo and the 3D models by Neuralange. The video shows how Neuralange can capture the fine details, textures, and expressions of the sculptures, such as David’s curly hair, Moses’ beard, and Pieta’s drapery. The video also shows how Neuralange can preserve the original lighting and shading effects of the sculptures, such as David’s contrast between light and shadow, Moses’ reflection on the marble surface, and Pieta’s soft glow. The video comparison can be seen on Nvidia’s blog or on YouTube.

How to access and use Neuralange?

Neuralange is currently a research project by Nvidia Research and Johns Hopkins University. It has not been released to the public yet. However, you can watch a demo video of Neuralange on Nvidia’s blog or read the paper on arXiv. You can also explore other generative AI models by Nvidia on their website or their developer portal. Some of these models are open-source and/or commercially usable, depending on their licensing structure.

If you are interested in learning more about this AI model, you can find all the links that are referenced in this blog post under the ‘source’ section at the end of this article.

Limitations

Neuralange is still a work in progress and has some limitations, such as:

• It requires a sufficient number of input frames that cover the object or scene from different angles. If the input frames are too few or too similar, the 3D reconstruction may be incomplete or inaccurate.
• It may not be able to handle dynamic scenes or objects that change their shape or appearance over time, such as people or animals.
• It may not be able to capture the fine-grained details or textures of some materials, such as hair, fur, or feathers.

Conclusion

Neuralange is part of Nvidia’s vision to advance generative AI, which is the branch of AI that can create new content from existing data. Nvidia is at the forefront of generative AI research, launching groundbreaking models like StyleGAN, GauGAN, eDiff-I, and many more. These generative models are pretrained for efficient enterprise application development. Neuralange is another example of how Nvidia is using generative AI to bridge the gap between the real world and the digital world. 

Source
Blog post - https://blogs.nvidia.com/blog/2023/06/01/neuralangelo-ai-research-3d-reconstruction/
Generative AI - https://www.nvidia.com/en-us/ai-data-science/generative-ai/
Ai models - https://developer.nvidia.com/ai-models
research paper - https://arxiv.org/abs/2306.03092
research document - https://arxiv.org/pdf/2306.03092.pdf
research lab - https://research.nvidia.com/labs/dir/neuralangelo/