How to use NVIDIA Instant NeRF on Nebula Cloud

How to use NVIDIA Instant NeRF on Nebula Cloud

GPU Computing Cloud Workstations Digital Twin 3D Modeling Virtual Reality Metaverse

 

 

What Is a NVIDIA Instant NeRF?

NVIDIA Instant NeRF is a neural rendering model that learns a high-resolution 3D scene in seconds — and can render images of that scene in a few milliseconds. NeRFs use neural networks to represent and render realistic 3D scenes based on an input collection of 2D images.

Collecting data to feed a NeRF is a bit like being a red-carpet photographer trying to capture a celebrity’s outfit from every angle — the neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of those shots.

In a scene that includes people or other moving elements, the quicker these shots are captured, the better. If there’s too much motion during the 2D image capture process, the AI-generated 3D scene will be blurry.

From there, a NeRF essentially fills in the blanks, training a small neural network to reconstruct the scene by predicting the color of light radiating in any direction, from any point in 3D space. The technique can even work around occlusions — when objects seen in some images are blocked by obstructions such as pillars in other images.

Accelerating 1,000x With Instant NeRF

While estimating the depth and appearance of an object based on a partial view is a natural skill for humans, it’s a demanding task for AI.

Creating a 3D scene with traditional methods takes hours or longer, depending on the complexity and resolution of the visualization. Bringing AI into the picture speeds things up. Early NeRF models rendered crisp scenes without artifacts in a few minutes, but still took hours to train.

Instant NeRF, however, cuts rendering time by several orders of magnitude. It relies on a technique developed by NVIDIA called multi-resolution hash grid encoding, which is optimized to run efficiently on NVIDIA GPUs. Using a new input encoding method, researchers can achieve high-quality results using a tiny neural network that runs rapidly.

The model was developed using the NVIDIA CUDA Toolkit and the Tiny CUDA Neural Networks library. Since it’s a lightweight neural network, it can be trained and run on a single NVIDIA GPU — running fastest on cards with NVIDIA Tensor Cores.

The technology could be used to train robots and self-driving cars to understand the size and shape of real-world objects by capturing 2D images or video footage of them. It could also be used in architecture and entertainment to rapidly generate digital representations of real environments that creators can modify and build on.

Beyond NeRFs, NVIDIA researchers are exploring how this input encoding technique might be used to accelerate multiple AI challenges including reinforcement learning, language translation and general-purpose deep learning algorithms.

For the first time in the world, Nebula Cloud has enabled NVIDIA Instant NeRF on Cloud with the choice of GPU configurations and storage options (both SSD and HDD). It removes the burden on the content developers and business users for installation, configuration and administration of the Instant NeRF software. The platform significantly speeds-up the setup time and the use of software for various content development work in disciplines like photogrammetry, 3D modelling, 3D rendering, Digital Twins and other industrial use cases.

Key Features of NVIDIA Instant NeRF on Nebula Cloud:

1.      Virtual Windows 10 Professional Workstation comes preconfigured with NVIDIA Instant NeRF with the choice of GPU (Tesla T4) and hardware configurations.
2.      Start using preconfigured NVIDIA Instant NeRF in less than 5 minutes on Nebula Cloud.
3.      No hassles of installation, configuration and administration of the software and hardware.
4.      In-built data migration tool to move the data from on-premise IT systems to the cloud workstation and vice-versa.
5.      Scale-up/Scale-down the hardware and storage depending on the changes in the project and volume of the data.
6.      In-built SSD storage (Upto 5TB) for effecient data processing of imagery and videos.
7.      In-built HDD storage (Upto 5TB) for data storage and archival.
8.  Comes with NICE DCV, a high-performance remote display protocol that provides customers with a secure way to deliver remote desktops and application streaming from any cloud or data center to any device, over varying network conditions.
9.  Comes preconfigured software and tools like WinSCP for FTP file transfer, Adobe Reader and many other.

10.  Pay only for the Cloud resources and use Instant NerF for free.

Currently made available for Australia, India, Singapore, Europe and USA regions. Supported by AWS Cloud.

***************************

Please note NVIDIA Instant NeRF works only on GPU enabled virtual machines so you have to select the correct GPU instance type while creating the virtual machine. Please use g4dn.xlarge (4 Core, 16GB RAM, 1 GPU (16GB Memory) to start with or higher configuration as desired.

***************************

NVIDIA Instant NeRF on Nebula Cloud comes pre-configured and ready to use. You just have to start the session using the following commands:

       ·     Once the machine is deployed as per above recommendations, you will have a GPU enabled virtual machine created and ready for use.
·       You will see the login credentials once the machine comes into running state.
·       Use the login credentials provided and connect to the virtual machine via RDP i.e. Remote Desktop Connection

       ·  Now, open the VS 2019 command line utility available on the desktop to start the Instant NeRF session
·       Change the directory and point to the project folder where NVIDIA Instant NeRF is configured. It is located here: C:\nvidia_nerf\instant-ngp
·       Use the following commands to point to the project folder: cd C:\nvidia_nerf\instant-ngp
·      Multiple sample datasets are preconfigured for testing purposes. You can test the setup by running the following commands:

 “./build/testbed” --scene data/nerf/fox
                                  or  
“./build/testbed” --scene data/nerf/lion 
                                   or 
“./build/testbed” --scene data/nerf/lordkrishna

·       Your test NeRF scene will open for rendering and further processing.

 

·  You can bring your own data to the virtual machine using winscp ftp utility available on the machine or data available in google drive or one drive as applicable.

All scripts are located in the below folder for various image processing activities with Instant NeRF:

C:\nvidia_nerf\instant-ngp\scripts>

·  You can render the video using the run.py python script with below commands:

C:\nvidia_nerf\instant-ngp\scripts>run.py --mode --scene C:\nvidia_nerf\instant-ngp\data\nerf\fox --load_snapshot C:\nvidia_nerf\instant-ngp\data\nerf\fox\base.msgpack --video_camera_path C:\nvidia_nerf\instant-ngp\data\nerf\fox\transforms.json --video_n_seconds 15 --video_fps 24 --video_output fox.mp4 --height 1280 --width 1024