Source-linked AI summary
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs
Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger
TL;DR
NeRF’s repeated evaluation of a deep MLP makes rendering too slow for interactive use. KiloNeRF replaces it with thousands of spatially local tiny MLPs, using distillation and rendering optimizations, and achieves three orders of magnitude faster synthesis while preserving visual fidelity. The approach remains bounded-scene and has unresolved scaling limits for full-HD or unbounded scenes.
Problem
NeRF’s long rendering time limits interactive applications, despite its high visual quality.
Method
KiloNeRF spatially decomposes scenes into many small MLPs and uses teacher-student distillation, regularization, and rendering optimizations.
Results
Three orders of magnitude faster rendering is achieved than NeRF while maintaining the original model’s visual fidelity.
Takeaways & Limitations
Spatial decomposition enables practical NeRF rendering while retaining excellent render quality and low storage impact.
Takeaways & Limitations
KiloNeRF assumes bounded scenes; unbounded scenes require more networks and larger memory consumption, and current speedups are insufficient for real-time full-HD rendering.
Abstract
from arXiv · showhide
NeRF synthesizes novel views of a scene with unprecedented quality by fitting a neural radiance field to RGB images. However, NeRF requires querying a deep Multi-Layer Perceptron (MLP) millions of times, leading to slow rendering times, even on modern GPUs. In this paper, we demonstrate that real-time rendering is possible by utilizing thousands of tiny MLPs instead of one single large MLP. In our setting, each individual MLP only needs to represent parts of the scene, thus smaller and faster-to-evaluate MLPs can be used. By combining this divide-and-conquer strategy with further optimizations, rendering is accelerated by three orders of magnitude compared to the original NeRF model without incurring high storage costs. Further, using teacher-student distillation for training, we show that this speed-up can be achieved without sacrificing visual quality.
1. Introduction
NeRF renders high-quality novel views but is slowed by repeated evaluations of a deep MLP. KiloNeRF uses many small networks and teacher-guided training to accelerate rendering while preserving visual fidelity.
- NeRF’s long training and rendering times limit interactive applications that require real-time rendering on consumer devices.
- 2548x faster rendering is achieved by replacing one high-capacity MLP with thousands of small MLPs without loss in visual quality.
- NeRF’s deep and wide MLP is queried hundreds of times for millions of pixels, making rendering very slow.
- KiloNeRF assigns independent small MLPs to scene regions, allowing each network to represent only a fraction of the scene.
- KiloNeRF distills a pretrained NeRF teacher into the small networks before fine-tuning on the original training images.
- Three orders of magnitude faster novel-view synthesis is achieved while matching the original NeRF’s visual fidelity.
2. Related Work
Related work improves novel-view synthesis through alternative scene representations and rendering optimizations. KiloNeRF differs by decomposing the scene into thousands of small networks rather than using one feature-conditioned or irregularly partitioned model.
- Mesh- and point-cloud-based methods typically require geometry preprocessing, whereas multi-plane images represent scenes with image or feature-map stacks.
- NeRF’s volumetric rendering avoids training masks and handles semi-transparent objects and thin structures better than surface-rendering approaches.
- NSVF accelerates NeRF with empty-space skipping, early ray termination, and a feature-conditioned network on a uniform 3D grid.
- KiloNeRF uses thousands of small networks for local regions, while NSVF uses one feature-conditioned network for the entire scene.
- Two orders of magnitude faster rendering than NSVF is reported for KiloNeRF.
- KiloNeRF uses thousands of MLPs on a regular 3D grid, contrasting with DeRF’s sixteen irregular Voronoi cells.
- Many NeRF extensions for unbounded scenes, unstructured photo collections, limited views, lighting variation, or video could benefit from faster rendering.
3. Method
KiloNeRF spatially decomposes a bounded scene into grid cells, assigns each cell a tiny MLP, and combines this with sampling and training optimizations. Distillation and regularization preserve quality while reducing computation.
- 3.1. Background: NeRF maps 3D positions and viewing directions to color and density, then repeatedly evaluates these outputs along camera rays.
- 3.1. Background: Rendering requires W × H × K network evaluations for an image with width W, height H, and K ray samples.
- 3.2. KiloNeRF: KiloNeRF subdivides a known axis-aligned bounding box into a uniform 3D grid and assigns each cell an independent tiny MLP.
- 3.2. KiloNeRF: The position-to-cell mapping selects which tiny network evaluates a queried position and viewing direction.
- 3.2. KiloNeRF: KiloNeRF uses 4 hidden layers with 32 hidden units each, compared with NeRF’s 10 hidden layers and wider feature layers.
- 3.3. Training with Distillation: Distillation trains KiloNeRF to match a pretrained NeRF’s density and color outputs before fine-tuning on training images.
- 3.2. KiloNeRF: A forward pass through KiloNeRF’s network requires 1/87th of the original architecture’s FLOPs.
- 3.3. Training with Distillation: L2 regularization on the last two layers imposes NeRF’s view-dependent color inductive bias without sacrificing visual quality.
4. Experimental Evaluation
Across 25 bounded scenes, KiloNeRF matches baseline visual quality while rendering substantially faster, and ablations identify the importance of fine spatial decomposition, distillation, regularization, and optimized execution. Results cover synthetic and real-scene datasets evaluated against NeRF and NSVF.
- Setup: KiloNeRF is evaluated on 25 scenes spanning synthetic-NeRF, synthetic-NSVF, BlendedMVS, and Tanks&Temples datasets.Both KiloNeRF and NSVF assume bounded scenes, enabling direct use of the four NSVF-provided datasets and bounding boxes.
- Setup: The evaluation compares KiloNeRF against the original NeRF model and NSVF, with inference time averaged over all test images on each scene.The measurements use an NVIDIA GTX 1080 Ti, Intel i7-3770k, and 32GB of RAM.
- Results: Three orders of magnitude faster than NeRF and two orders faster than NSVF, KiloNeRF achieves similar visual quality and slightly better LPIPS on three of four datasets.Table 1 reports the principal quality and rendering-speed comparison across all scenes.
- Ablations: A single 6k-parameter MLP cannot represent the entire scene accurately, causing dramatic quality loss compared with thousands of spatially distributed tiny MLPs.The single-network ablation uses KiloNeRF’s tiny-network hyperparameters but instantiates only one network.
- Ablations: Halving grid resolution reduces quality more than halving hidden units, while halving resolution decreases parameters by 8× versus approximately 4× for hidden-unit reduction.For the Lego bulldozer scene, the tested grid changes from 10 × 16 × 10 to 5 × 8 × 5.
- Ablations: Fine-tuning is crucial for high quality, whereas distillation alone mainly provides a good initialization; distillation and L2 regularization also prevent free-space artifacts.Training from scratch can match detail only when distillation avoids artifacts, and omitting L2 regularization produces artifacts in free space.
- Ablations: Reducing network size contributes significantly to KiloNeRF’s overall speedup beyond optimized empty-space skipping and early ray termination.The speedup breakdown compares original-size NeRF with KiloNeRF’s ESS/ERT implementation against the full KiloNeRF technique.
5. Discussion and Future Work
KiloNeRF reaches interactive rates for 800 × 800 images, but does not yet provide sufficient speed for real-time full HD rendering. Scaling to more, smaller networks may improve speed while increasing storage demands, and unbounded scenes remain constrained by memory consumption.
- 800 × 800 images reach interactive frame rates, but KiloNeRF remains insufficiently fast for real-time full HD rendering.The paper identifies higher-resolution real-time rendering as an unresolved limitation.
- Scaling to more and smaller networks could provide further speedups, but naively doing so would increase storage impact.The authors suggest memory-efficient approaches as a possible mitigation.
- KiloNeRF assumes bounded scenes; unbounded scenes require more networks and therefore larger memory consumption.Efficient data structures are suggested for scaling to larger, potentially outdoor scenes.
6. Conclusion
KiloNeRF achieves real-time NeRF rendering by spatially decomposing a scene into a regular grid and assigning a small-capacity network to each cell. The approach retains NeRF’s render quality and low storage impact and may extend to other neural function representations.
- Real-time NeRF rendering is achieved by assigning a small-capacity network to each cell of a regular scene grid.The method uses spatial decomposition to distribute scene representation across many small networks.
- KiloNeRF retains excellent render quality and low storage impact inherited from NeRF.
- The acceleration strategy might also apply to other methods that rely on neural function representations, including implicit surface models.