PaperlayerView exact revision on arXiv

Source-linked AI summary

Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction

Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, Yingyan Celine Lin

arXiv:2304.12467v3cs.AR

TL;DR

On-device NeRF training fast enough for instant AR/VR reconstruction remains out of reach for existing training algorithms. Instant-3D co-designs an algorithm and accelerator around embedding-grid interpolation, reducing training time while preserving reconstruction quality and demonstrating sub-five-second reconstruction within the stated AR/VR power constraint.

  • Problem

    Even the most efficient existing NeRF training algorithm cannot achieve instant, under-five-second on-device reconstruction for new AR/VR scenes.

    Source: §1 · pp. 1–2

  • Method

    Instant-3D decomposes the embedding grid into color and density branches with different grid sizes and update frequencies, alongside accelerator mechanisms that reduce grid-memory accesses.

    Source: §1 · p. 2

  • Results

    41×–248× training-time reduction while maintaining the same reconstruction quality; on NeRF-Synthetic, reconstruction takes 1.6 seconds per scene at PSNR 25 while meeting the 1.9 W AR/VR power constraint.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Instant-3D demonstrates instant on-device NeRF-based 3D reconstruction and, the authors suggest, opens a perspective toward on-device 3D reconstruction for AR/VR.

    Source: §1 · p. 2, §7 · p. 12

Abstract

from arXiv · show

Neural Radiance Field (NeRF) based 3D reconstruction is highly desirable for immersive Augmented and Virtual Reality (AR/VR) applications, but achieving instant (i.e., < 5 seconds) on-device NeRF training remains a challenge. In this work, we first identify the inefficiency bottleneck: the need to interpolate NeRF embeddings up to 200,000 times from a 3D embedding grid during each training iteration. To alleviate this, we propose Instant-3D, an algorithm-hardware co-design acceleration framework that achieves instant on-device NeRF training. Our algorithm decomposes the embedding grid representation in terms of color and density, enabling computational redundancy to be squeezed out by adopting different (1) grid sizes and (2) update frequencies for the color and density branches. Our hardware accelerator further reduces the dominant memory accesses for embedding grid interpolation by (1) mapping multiple nearby points' memory read requests into one during the feed-forward process, (2) merging embedding grid updates from the same sliding time window during back-propagation, and (3) fusing different computation cores to support the different grid sizes needed by the color and density branches of Instant-3D algorithm. Extensive experiments validate the effectiveness of Instant-3D, achieving a large training time reduction of 41x - 248x while maintaining the same reconstruction quality. Excitingly, Instant-3D has enabled instant 3D reconstruction for AR/VR, requiring a reconstruction time of only 1.6 seconds per scene and meeting the AR/VR power consumption constraint of 1.9 W.

1 INTRODUCTION

Instant on-device NeRF reconstruction is desirable for AR/VR, but even the most efficient existing training algorithm could not achieve it. Instant-3D addresses this gap with an algorithm-hardware co-design targeting embedding-grid interpolation, reporting faster training while maintaining reconstruction quality.

1 INTRODUCTION

  • Embedding-grid interpolation and its back-propagation dominate around 80% of training runtime and are executed more than 200,000 times per iteration.The interpolation step generates color and density embeddings for scene points.Source: §1 · p. 2
  • 41×–248× training-time reduction while maintaining the same reconstruction quality; Instant-3D also reconstructs a NeRF-Synthetic scene in 1.6 seconds at PSNR 25 while meeting the 1.9 W AR/VR power constraint.The 1.6-second, PSNR 25 result is reported on NeRF-Synthetic.Source: §1 · p. 2
  • Instant-3D separates the embedding grid into color and density branches, using different grid sizes and update frequencies to reduce computational redundancy without compromising reconstruction quality.The decomposition leverages the branches’ different sensitivities when compressing NeRFs.Source: §1 · p. 2
  • Its accelerator maps nearby points’ feed-forward reads into one request, merges updates from the same sliding time window during back-propagation, and fuses computation cores for different grid sizes.These hardware mechanisms target memory accesses during embedding-grid interpolation.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

This section reviews vanilla NeRF’s training pipeline and why its cost precludes instant on-device reconstruction, then identifies embedding-grid interpolation and its back-propagation as Instant-NGP’s runtime bottleneck.

2.1 Preliminaries of NeRFs

  • Vanilla NeRF samples pixels, casts rays, queries per-point density and color, renders ray colors, computes squared color loss, and backpropagates through the differentiable pipeline.Source: §2.1 · pp. 2–3
  • Vanilla NeRF typically needs 150,000 iterations and 353,895 trillion FLOPs per scene, taking more than one day on a V100 GPU.Source: §2.1 · p. 3
  • Instant-NGP replaces vanilla NeRF’s large feature-query MLP with hash-grid embedding interpolation and a small MLP, yet still needs minutes to days on edge GPUs versus the <5-second instant target.Source: §2.1 · pp. 3–4

2.2 Profiling Analysis of the SOTA Efficient NeRF Training Process

  • Across eight NeRF-Synthetic scenes and commercial devices, embedding-grid interpolation and its corresponding back-propagation dominate Instant-NGP training runtime.The profiling covered devices consuming 10–20 W.Source: Figure 4 · p. 4, §2.2 · p. 4

3 INSTANT-3D: PROPOSED ALGORITHM

Instant-3D separates the color and density embedding grids because their features learn at different rates and have different sensitivities. It exploits this difference with distinct grid sizes and update frequencies to reduce training time while preserving reconstruction quality.

3.1 Different Paces of Color and Density During Training

  • Color features learn faster than density features, supporting separate color and density grids with independently chosen sizes and update frequencies.Color reaches 24 dB PSNR in 160 iterations versus 200 for density; the paper attributes the difference to training loss being based on predicted color.Source: §3.1 · pp. 4–5, §3 · p. 4, Figure 5 · p. 4

3.2 Different Grid Sizes for the Color and Density Branches

  • 12.5% lower training runtime with the same PSNR as Instant-NGP [24] is achieved by using different grid sizes for the color and density grids.When either grid is reduced to 0.25× the vanilla Instant-NGP size, reconstruction PSNR is 26.0 dB for color and 25.4 dB for density.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 Different Update Frequencies for the Color and Density Branches

  • Instant-3D uses a smaller grid and lower update frequency for color than for density, as depicted in the decomposed-branch pipeline.The design sets S_D > S_C and F_D > F_C.Source: §3.2 · p. 5, §3.3 · p. 5, Figure 6 · p. 5
  • 9.7% lower training runtime with similar PSNR to Instant-NGP [24] is achieved by using different update frequencies for the two grids.At 0.5× the vanilla update frequency, reconstruction PSNR is 25.9 dB for color and 24.3 dB for density.Source: §3.3 · p. 5, Table 2 · p. 5

4 INSTANT-3D: PROPOSED ACCELERATOR

After the algorithm alone leaves embedding-grid interpolation as the dominant training cost, Instant-3D adds an accelerator that exploits feed-forward address locality and back-propagation updates to shared embeddings. Its read mapper, update merger, and reconfigurable grid cores target those access patterns and the algorithm’s different grid sizes.

4.1 Motivation: Profiling Instant-NGP with Our Algorithm

  • Embedding-grid interpolation and its back-propagation account for around 80% of runtime, while algorithm-only Instant-NGP training still takes around 60 seconds per scene on Xavier NX.The algorithm improves Instant-NGP by 17.0% on average, but remains beyond the < 5-second instant target.Source: §4.1 · p. 6, Figure 7 · p. 6

4.4 Feed-Forward Read Mapper to Better Use SRAM Arrays

  • The feed-forward read mapper combines nearby embedding reads into fewer requests when bank access collisions are absent, improving utilization of multi-bank SRAM.The motivating address measurements find about 90% of intra-group distances below 5, while inter-group distances average as high as 60,000.Source: §4.2 · p. 6, Figure 8 · p. 6, Figure 9 · p. 7

4.5 Back-Propagation Update Merger Minimizing SRAM Writes

  • Back-propagation can have more than five updates to shared embeddings, so the update merger accumulates same-address updates before writing them to SRAM.Within 1,000 continuous accesses, about 200 unique addresses are observed in back-propagation; feed-forward accesses in the window are unique.Source: §4.2 · p. 7, §4.5 · p. 9

4.3 Proposed Instant-3D Accelerator

  • The accelerator assigns grid interpolation and its gradients to four grid cores, with FRM and BUM units supporting read mapping and update merging.The host SoC performs pixel sampling, ray mapping, pixel prediction, and reconstruction loss computation.Source: §4.3 · pp. 7–8

4.6 Reconfigurable Scheme Supporting Different Grid Sizes

  • Multi-core fusion supports grid sizes of 256 KB, 512 KB, and 1 MB using standalone, paired-core, and four-core modes, respectively.The modes use 8-, 16-, and 32-bank SRAM access management.Source: Figure 11 · p. 8, §4.6 · p. 9

4.3 Proposed Instant-3D Accelerator

  • The accelerator’s systolic-array and multiplier-adder-tree MLP units target larger and smaller output channels, respectively.The multiplier-adder-tree is used for output channels ≤3; the systolic array is used for channels >3.Source: §4.3 · p. 8

5 EVALUATION

The evaluation compares Instant-3D’s reconstruction quality and training efficiency with NeRF training algorithms and commercial edge devices, then assesses its algorithm and accelerator components. The reported results show faster training and improved energy efficiency, with ablations examining the contributions of the co-design and memory units.

5.1 Evaluation Settings

  • Evaluation uses NeRF-Synthetic, SILVR, and ScanNet, measures reconstruction quality by test-set PSNR, and compares against three commercial devices without a dedicated NeRF-training accelerator baseline.The baselines are Jetson Nano, Jetson TX2, and Xavier NX.Source: §5.1 · p. 10, Table 3 · p. 10
  • The algorithm uses density-to-color grid-size and update-frequency ratios of 1:0.25 and 1:0.5, respectively, selected by grid search while maintaining Instant-NGP reconstruction quality.The density grid updates every iteration and the color grid every two iterations.Source: §5.1 · p. 10
  • The accelerator is implemented in 28nm CMOS and has 6.8 mm2 area and 1.9 W average power consumption.The evaluation estimates training efficiency with a cycle-accurate simulator assuming 59.7 GB/s DRAM bandwidth.Source: §5.1 · p. 10, Figure 15 · p. 10

5.3 Instant-3D Accelerator’s Performance

  • 224×/132×/45× speedups and 1198×/1089×/479× more energy efficiency over Jetson Nano/TX2/Xavier NX, respectively, on NeRF-Synthetic.Figure 16 summarizes normalized speedup and energy efficiency across eight scenes.Source: Figure 16 · p. 11, §5.3 · p. 11

5.2 Instant-3D Algorithm’s Performance

  • 60 seconds versus 72 seconds achieves the same 26.0 PSNR averaged over eight NeRF-Synthetic scenes for Instant-3D versus Instant-NGP on Xavier NX.Table 4 benchmarks the algorithms by PSNR and training runtime.Source: §5.2 · pp. 10–11, Table 4 · p. 11

5.3 Instant-3D Accelerator’s Performance

  • The FRM unit trims runtime by 31.1% on average, and adding the BUM unit increases the runtime reduction to 68.6%.Figure 18 compares normalized runtime with either unit removed.Source: §5.3 · p. 11, Figure 18 · p. 11
  • Instant-3D’s algorithm alone trims runtime by 83.0% versus Instant-NGP on Xavier NX, while the algorithm and accelerator together reduce runtime to 2.2% of that solution.Table 5 compares normalized runtime with algorithm and hardware techniques removed.Source: §5.3 · p. 11, Table 5 · p. 12

6 RELATED WORKS

Prior NeRF accelerators support inference rather than training, while Instant-3D is evaluated against an inference accelerator for rendering. Earlier MLP- and CNN-training accelerators do not support NeRF’s embedding-grid interpolation operations.

6 RELATED WORKS

  • The two existing dedicated NeRF accelerators discussed here support inference only, not the training required for instant on-device reconstruction.The passage identifies them as [15] [33].Source: §6 · p. 12
  • Against the SOTA NeRF inference accelerator [15], Instant-3D provides real-time rendering with 19.5% of its energy per frame and 36% of its chip area.Source: §6 · p. 12
  • Prior MLP- or CNN-training accelerators are not applicable because they do not support embedding-grid interpolation.Source: §6 · p. 12

7 CONCLUSION

The conclusion presents Instant-3D as a co-designed framework for instant on-device NeRF reconstruction. It combines color-density grid decomposition with hardware units intended to reduce memory-access overhead and support the algorithm.

7 CONCLUSION

  • Instant-3D decomposes the embedding grid into color and density branches to squeeze out redundancy in both.Source: §7 · p. 12
  • Its accelerator combines FRM for multi-bank SRAM use, BUM to minimize SRAM writes, and reconfigurable scheduling to support the algorithm.Source: §7 · p. 12
  • The authors identify Instant-3D as the first framework to achieve instant on-device NeRF-based 3D reconstruction, to their knowledge.Source: §7 · p. 12

Related papers

Loading 2304.12467v3…