Source-linked AI summary
FastNeRF: High-Fidelity Neural Rendering at 200FPS
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, Julien Valentin
TL;DR
NeRF enables photorealistic novel-view rendering but remains computationally demanding for interactive use. FastNeRF factorizes the representation into cacheable position and direction components, achieving 200Hz-class rendering and large speedups while retaining visual quality, with some motion-related artifacts and inference limitations.
Problem
More than 100 neural network calls per pixel make NeRF rendering take several seconds for low-resolution images, while prior improvements remain far from real-time consumer-hardware rendering.
Method
FastNeRF factorizes NeRF into a position-conditioned deep radiance map and a direction-conditioned weight function, combining them through an inner product and caching their outputs.
Results
FastNeRF renders photorealistic images at 200Hz and more on consumer hardware, with significant speedups over NeRF and competing methods.
Takeaways & Limitations
Caching makes NeRF-like rendering memory-bound instead of compute-bound, enabling real-time scenarios on consumer hardware.
Takeaways & Limitations
FastNeRF can exhibit more flickering artifacts than the NeRF baseline on the NeRF 360 Synthetic dataset, particularly in motion.
Abstract
from arXiv · showhide
Recent work on Neural Radiance Fields (NeRF) showed how neural networks can be used to encode complex 3D environments that can be rendered photorealistically from novel viewpoints. Rendering these images is very computationally demanding and recent improvements are still a long way from enabling interactive rates, even on high-end hardware. Motivated by scenarios on mobile and mixed reality devices, we propose FastNeRF, the first NeRF-based system capable of rendering high fidelity photorealistic images at 200Hz on a high-end consumer GPU. The core of our method is a graphics-inspired factorization that allows for (i) compactly caching a deep radiance map at each position in space, (ii) efficiently querying that map using ray directions to estimate the pixel values in the rendered image. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF algorithm and at least an order of magnitude faster than existing work on accelerating NeRF, while maintaining visual quality and extensibility.
1. Introduction
NeRF produces high-quality novel views but requires substantial computation, motivating FastNeRF's factorized, cacheable representation for real-time rendering on consumer hardware.
- NeRF renders high-quality novel views while preserving thin structures, view-dependent effects, and partially transparent surfaces.
- More than 100 neural network calls per pixel make NeRF rendering take several seconds for low-resolution images on high-end GPUs.
- Recent NeRF accelerations reduce render time by up to 50× but remain far from real-time rendering on consumer hardware.
- FastNeRF factorizes NeRF into position-dependent and direction-dependent functions whose outputs combine through an inner product to estimate RGB values.
- O(k^3l^2) memory for cached NeRF falls to O(k^3 ∗(1 + D ∗3) + l^2 ∗D) for FastNeRF, enabling caching on a high-end consumer GPU.
- 200FPS rendering of photorealistic novel views is presented as FastNeRF's main system achievement, thousands of times faster than NeRF.
2. Related work
Related work accelerates NeRF through parallelism, domain decomposition, sparsity, depth prediction, or approximate integration, while FastNeRF instead avoids inference-time MLP calls through decomposition and caching.
- FastNeRF differs by discretizing positional and directional components into sparse 3D grids for caching rather than using a grid during training.
- NeRF's implicit model requires hundreds of MLP invocations per image pixel, motivating multiple strategies for faster inference.
- Parallel processing and domain decomposition provide speedups, including up to 3× for decomposed radiance fields.
- Other approaches exploit volumetric sparsity, predict per-pixel depth, or approximate ray integrals, but reported methods remain below interactive rates or reduce quality.
- FastNeRF's decomposition and caching avoid inference-time MLP calls entirely, making it faster in absolute terms even on a single machine.
- FastNeRF does not address training speed, which remains the focus of separate approaches such as meta-learning initialization.
3. Method
FastNeRF factorizes NeRF into position- and direction-dependent networks so their outputs can be cached separately, reducing memory demands and accelerating rendering while preserving NeRF’s runtime interface.
- Overview: 3000 times faster than the original Neural Radiance Fields system, FastNeRF renders high-resolution photorealistic images at over 200Hz on high-end consumer hardware.The method is designed to improve test-time efficiency while preserving NeRF’s visual quality.
- Factorized Neural Radiance Fields: FastNeRF splits NeRF into a position-dependent network producing a deep radiance map and a direction-dependent network producing weights whose inner product estimates color.The factorization changes a function over R5 into functions over R3 and R2, making the outputs cacheable.
- Neural Radiance Fields: NeRF samples many 3D positions along each camera ray, evaluates its network at each position and direction, and integrates the resulting colors and transparencies into a pixel.The default N = 192 samples per pixel require nearly 400 million network calls for one 1080p image.
- Factorized Neural Radiance Fields: The factorized runtime function retains NeRF’s signature, FFastNeRF(p, d) 7→(c, σ), allowing it to serve as a drop-in replacement.The architecture combines the same position and ray-direction inputs into color and density outputs.
- Caching: A naïve NeRF cache stores an output for every position-direction pair, producing O(k3l2) memory complexity and approximately 5600 terabytes for k = l = 1024.The large cache follows from jointly discretizing the five-dimensional input space.
4. Implementation
The implementation trains FastNeRF similarly to NeRF and evaluates rendering speed on the Lego scene at 800×800 pixels against prior methods.
- Training: FastNeRF training is identical to NeRF except for the proposed architecture, using positional encoding and separate MLPs for position and view prediction.The position network uses eight layers with 384 hidden units, while the view-predicting network uses four layers with 128 hidden units.
- Evaluation: The Lego-scene speed evaluation renders 800×800 images and compares FastNeRF with prior work on the Realistic 360 Synthetic dataset.When prior Lego measurements were unavailable, the evaluation used optimistic approximations.
- Test-time rendering: At test time, camera parameters generate one ray per output pixel, samples are integrated along each ray, and hardware-accelerated ray tracing skips empty space.FastNeRF can run as a neural network, but caching substantially improves performance.
5. Experiments
Experiments compare FastNeRF with NeRF across synthetic and forward-facing scenes, cache resolutions, component counts, image-quality metrics, and rendering speed. The results show that caching preserves high visual quality at suitable resolutions while enabling large speedups, with cache size and component count controlling quality, memory, and runtime.
- Evaluation setup: FastNeRF is evaluated quantitatively and qualitatively on the Realistic 360 Synthetic and LLFF datasets used in the original NeRF paper.Quality is assessed with PSNR, SSIM, and LPIPS, while speed comparisons use an Nvidia RTX 3090 GPU.
- Rendering quality: At sufficiently high cache resolutions, FastNeRF achieves the same visual quality as NeRF, while smaller caches retain scene characteristics but can appear pixelated.The cache-resolution trade-off resembles levels of detail in traditional computer graphics.
- Rendering speed: Cached FastNeRF never drops below 100FPS in the Table 2 comparisons, whereas the uncached model can be slower than NeRF because it uses larger MLPs.The highest-resolution cache is not computed for LLFF scenes because their normalized-device-coordinate representation is less sparse.
- Cache resolution: FastNeRF matches or outperforms NeRF on synthetic objects at 1024^3 cache resolution, while a 512^3 cache is identified as a practical synthetic-dataset trade-off.For LLFF, 768^3 works best, although cached results show a slight metric decrease while preserving intricate detail qualitatively.
- Rendering speed: 3000× faster than NeRF on average at 768^3 grid resolution while maintaining largely the same perceptual quality.The comparison reports image-buffer completion time for FastNeRF against the baseline implementation.
- Model configuration: 6 or 8 components are sufficient for most scenes, balancing component count, cache sparsity, memory usage, and image quality.Increasing from 8 to 16 components offers a theoretical improvement of roughly 0.5 dB, while additional components tend to become increasingly sparse when cached.
6. Application
FastNeRF is extended to a dynamic telepresence setting by jointly modeling a canonical scene and expression-conditioned deformation. The proof of concept renders facial expressions in real time and remains compatible with broader NeRF extensions.
- Telepresence: FastNeRF jointly trains a scene model with an expression-conditioned deformation model for non-static facial data.The deformation model updates sample positions into a canonical frame of reference.
- Telepresence: 300×300 pixel facial images render at 30 fps on one Nvidia Tesla V100, about 50× faster than a NeRF setup.The rendered expressions are clearly visible at this resolution and frame rate.
- Telepresence: The telepresence approach can be extended from a multi-camera capture setup to a single camera.
- Extensibility: FastNeRF’s shared inputs and outputs make it applicable to extensions for dynamic scenes, single-image reconstruction, quality improvement, illumination control, and uncertainty.Some extensions require only minor modifications.
7. Conclusion
FastNeRF factorizes NeRF’s function approximator so its outputs can be cached, shifting rendering from compute-bound to memory-bound operation. The resulting system renders photorealistic images at 200Hz or more on consumer hardware.
- FastNeRF enables photorealistic image rendering at 200Hz and more on consumer hardware.The method achieves speedups over NeRF and competing methods while using a cache.
- Factorizing NeRF enables caching that makes rendering memory-bound instead of compute-bound.
8. Overview
The implementation combines sparse caching, ray-based acceleration, and quality-oriented rendering choices, with application-specific deformation methods introducing distinct speed and smoothness trade-offs. Supplementary experiments examine cache configurations, metrics, and practical limitations.
- Rendering trade-offs: Rendering parameters favor the highest possible quality, while sacrificing a small amount of visual quality can significantly increase speed.
- Motion smoothness: FastNeRF can show more flickering artifacts than the baseline on NeRF 360 Synthetic when trained with NeRF’s settings.The artifacts are noticeable only in motion and can be mitigated using a smaller direction cache or modified Fourier encoding.
- Motion smoothness: 29.748dB average PSNR is obtained with Fourier-feature direction encoding L = 1 on synthetic data, compared with 29.646dB for L = 2 and 29.449dB for L = 3.
- Motion smoothness: A smaller direction cache has negligible metric impact and can occasionally improve PSNR.The reported changes include Materials 28.885dB → 28.874dB and Ficus 27.792dB → 28.193dB.
- Rendering acceleration: Sparse octrees accelerate rendering by allowing rays that miss objects or traverse empty space to be skipped efficiently.Collision meshes and ray tracing provide hardware-accelerated intersection and empty-space traversal.
- Memory analysis: The cache-size analysis compares NeRF and FastNeRF using transparency, RGB, directional-value, and weight storage terms.
- Telepresence details: In the telepresence application, a deformation field modifies sample positions using facial expression inputs and is regularized toward identity for neutral expressions.The identity regularizer stabilizes training and improves results.
- Telepresence details: The deformation approach’s main speed limitation is evaluating Fdeform for all input samples.A test-time grid-based sample pruner is introduced to mitigate this cost.