At GTC 2022, the NVIDIA Research Team unveiled an impressive technology called Instant NeRF that could turn 2D photos into 3D scenes in mere seconds.
To do this, the researchers leveraged the power of AI, more specifically through the so-called neural radiance fields. NeRFs use neural networks to reconstruct the scene from the images by predicting the color of light radiating in any direction. NVIDIA claims Instant NeRF to be the fastest technology of its kind yet, allowing for a multiplicative speed-up of over 1,000x in some instances so that rendering at 1080p occurs in mere milliseconds.
As discussed by Principal Research Scientist Thomas Muller in a GDC 2022 talk titled Instant Neural Graphics Primitives, this compounded effect is the result of three main improvements: a task-specific GPU implementation of the rendering/training algorithm, which uses the GPU’s fine-grain control flow capabilities to be much faster than dense tensors; a fully fused implementation of a small neural network, which is faster than general purpose matrix multiplication routines; lastly, NVIDIA developed a technique called multiresolution hash grid encoding which is task agnostic and has better speed/quality tradeoff than preexisting work.
Unsurprisingly, the model for Instant NeRF was developed with the CUDA Toolkit and the Tiny CUDA Neural Networks library. You can dive deeper into the research and even access its code on this page; according to NVIDIA, the neural network is lightweight enough to run easily on a single GPU, especially if it is equipped with Tensor Cores.
David Luebke, vice president for graphics research at NVIDIA, said in a statement:
If traditional 3D representations like polygonal meshes are akin to vector images, NeRFs are like bitmap images: they densely capture the way light radiates from an object or within a scene. In that sense, Instant NeRF could be as important to 3D as digital cameras and JPEG compression have been to 2D photography — vastly increasing the speed, ease and reach of 3D capture and sharing.
The applications for the Instant NeRF technology can be many, from quickly scanning real environments or persons so that game creators can then use the digital scans in their projects to training self-driving cars or robots to understand the shape and size of real objects.
