Source-linked AI summary
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
TL;DR
View synthesis requires reconstructing complex scenes from sparse posed images. NeRF optimizes a continuous 5D neural radiance field and renders rays with differentiable volume rendering, achieving photorealistic novel views that outperform prior view-synthesis methods. Its main remaining boundaries are rendering efficiency and interpretability of scenes encoded in network weights.
Problem
View synthesis seeks to render novel views of complex scenes by directly optimizing a continuous scene representation from captured images.
Method
NeRF represents each scene with an MLP mapping 3D location and viewing direction to volume density and view-dependent radiance, then optimizes it through differentiable volume rendering.
Results
NeRF produces better renderings than prior neural and discretized-voxel view-synthesis approaches, including high-resolution photorealistic novel views of real scenes.
Takeaways & Limitations
Continuous 5D radiance fields provide a scene representation for complex geometry and appearance without the prohibitive storage costs of high-resolution discretized voxel grids.
Takeaways & Limitations
Efficient optimization and rendering remain open problems, and the failure modes of scenes encoded in neural-network weights are difficult to analyze.
Abstract
from arXiv · showhide
We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x,y,z)$ and viewing direction $(θ, φ)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.
1 Introduction
NeRF addresses view synthesis by optimizing a continuous 5D scene representation directly from captured images. Its MLP, differentiable volume rendering, positional encoding, and hierarchical sampling enable photorealistic novel views and outperform prior methods.
- Problem and approach: NeRF optimizes a continuous 5D scene representation to minimize rendering error across captured images.The representation models volume density and view-dependent color as functions of spatial location and viewing direction.
- Differentiable rendering: Camera rays are sampled through the scene, an MLP predicts colors and densities, and volume rendering composites them into images for gradient-based optimization.The differentiable rendering process minimizes residuals between observed images and rendered views.
- Technical improvements: Positional encoding enables high-frequency scene content, while hierarchical sampling reduces queries devoted to empty or irrelevant regions.These improvements address insufficient resolution and inefficient sampling in the basic implementation.
- Technical contributions: The approach uses continuous neural radiance fields instead of discretized voxel grids, avoiding prohibitive storage costs for complex high-resolution scenes.It retains the benefits of volumetric representations for complex geometry and appearance.
- Results: The resulting method quantitatively and qualitatively outperforms state-of-the-art view synthesis methods and renders high-resolution photorealistic novel views of real scenes.The paper describes this as the first continuous neural scene representation achieving that capability from RGB images captured in natural settings.
2 Related Work
Related work includes implicit neural shape representations, graphics-function MLPs, mesh and volumetric scene models, and learned view-synthesis systems. NeRF targets their limitations by optimizing 5D radiance fields for complex, high-resolution appearance and geometry.
- Graphics functions: MLPs have also represented images, textured materials, and indirect illumination by mapping low-dimensional coordinates to graphics-function values.These works establish related coordinate-based representations beyond 3D shape modeling.
- Neural 3D shape representations: Prior implicit shape methods map 3D coordinates to signed distance or occupancy fields but often require ground-truth geometry or produce oversmoothed renderings.The discussed models had limited success on scenes with complex geometry.
- NeRF’s position: NeRF uses 5D radiance fields with view-dependent appearance to represent higher-resolution geometry and appearance for photorealistic novel views of complex scenes.This strategy is presented as an alternative to prior continuous shape representations limited to simpler shapes.
- View synthesis: Mesh-based view-synthesis methods represent geometry with diffuse or view-dependent appearance and may use differentiable rasterizers or path tracers.They form one major class of approaches for synthesizing views from observed images.
- Volumetric rendering: Volumetric approaches realistically represent complex shapes and materials, support gradient-based optimization, and often produce fewer distracting artifacts than mesh methods.Earlier systems directly colored voxel grids, while later systems trained networks to predict volumetric representations.
3 Neural Radiance Field Scene Representation
NeRF represents a continuous scene with an MLP that maps spatial location and viewing direction to density and emitted color. Density depends only on location for multiview consistency, while color varies with direction to model non-Lambertian appearance.
- Scene representation: The neural radiance field is a continuous 5D function mapping 3D location and 2D viewing direction to RGB color and volume density.The viewing direction may be expressed as a 3D Cartesian unit vector.
- Rendering pipeline: Rendering samples 5D coordinates along camera rays, feeds them to the MLP, and composites the predicted colors and densities into an image.The procedure is differentiable and supports optimization against observed images.
- Scene representation: An MLP approximates this function and is optimized to predict directional emitted color and density at each queried coordinate.The network is fully connected and contains no convolutional layers.
- View dependence: The network restricts density to depend only on spatial location while allowing color to depend on both location and viewing direction.This design encourages multiview consistency while representing view-dependent effects.
- View dependence: Removing view dependence makes specularities difficult to represent, demonstrating its role in modeling non-Lambertian appearance.The paper points to examples in which view-dependent color captures changing specular effects.
4 Volume Rendering with Radiance Fields
NeRF renders rays using classical volume rendering: densities and directional colors along each ray are integrated into expected pixel color. Differentiable sampling and compositing make the rendering process optimizable from image residuals.
- Volume rendering: The expected color of a camera ray is computed by integrating emitted radiance weighted by volume density and accumulated transmittance.The ray is parameterized between near and far bounds.
- View-dependent appearance: View-dependent radiance lets the renderer model changing specular appearance at fixed spatial points across viewing directions.The Ship examples show directional color distributions for points on the ship and water.
- Volume rendering: Accumulated transmittance represents the probability that a ray travels to a point without hitting another particle.This quantity determines how much each sampled location contributes to the rendered ray color.
- Numerical integration: Stratified sampling partitions the ray interval into evenly spaced bins and draws one sample uniformly from each bin.Evaluating the MLP at continuously varying sampled positions preserves the continuous representation during optimization.
- Numerical integration: The sampled colors and densities are combined with a differentiable quadrature rule that reduces to alpha compositing.The alpha values are α_i = 1 − exp(−σ_iδ_i), where δ_i is the distance between adjacent samples.
5 Optimizing a Neural Radiance Field
Neural radiance field optimization is improved for high-resolution complex scenes by combining positional encoding with hierarchical sampling. The method uses coarse predictions to focus fine samples on regions likely to affect rendered content.
- 5.1 Positional encoding: The basic coordinate-to-network mapping performs poorly on high-frequency color and geometry because deep networks favor lower-frequency functions.Reformulating the network as a learned MLP composed with a fixed encoding significantly improves performance.
- 5.1 Positional encoding: Positional encoding maps continuous input coordinates into a higher-dimensional space, helping the MLP represent high-frequency functions.The encoding is applied separately to spatial coordinates and viewing-direction components; experiments use L = 10 for x and L = 4 for d.
- 5.2 Hierarchical volume sampling: Hierarchical sampling addresses inefficient ray evaluation by allocating more samples to regions expected to influence the rendered image.Dense sampling otherwise repeatedly evaluates free-space and occluded regions that contribute little to the final rendering.
- 5.2 Hierarchical volume sampling: A coarse network first estimates ray weights, which define a piecewise-constant PDF for sampling additional points before evaluating the fine network.The fine network is evaluated on the union of coarse and additional samples, producing the final rendered color.
- 5.3 Implementation details: Training optimizes a separate continuous volume representation for each scene using captured RGB images, camera parameters, and scene bounds.The implementation samples batches of camera rays and applies hierarchical queries to the coarse and fine networks.
- 5.3 Implementation details: Each scene typically uses 4096 rays per batch, with 64 coarse and 128 additional fine samples per ray.Optimization uses Adam with a learning rate decaying from 5 × 10^-4 to 5 × 10^-5 over approximately 100–300k iterations.
6 Results
The method is evaluated on synthetic and real scenes against neural and image-based rendering baselines. It outperforms the compared methods broadly, while trading longer training for substantially lower storage.
- Datasets: The evaluation covers synthetic objects with simple or complex materials and real forward-facing scenes, using matched input views for separate per-scene networks.The synthetic datasets include DeepVoxels and pathtraced objects; the real dataset contains eight handheld captures.
- Overall results: The method quantitatively and qualitatively outperforms prior work across synthetic and real-image datasets.Table 1 reports PSNR, SSIM, and LPIPS; LLFF is slightly better on LPIPS, while the method shows better multiview consistency and fewer artifacts.
- Comparisons: The method outperforms Neural Volumes and SRN in all scenarios and surpasses LLFF qualitatively and quantitatively on all but one metric.Figures show recovery of fine geometry, appearance, and partially occluded content, while baselines exhibit blur, ghosting, banding, or missing details.
- Practical tradeoffs: 5 MB of network weights replaces over 15 GB of storage for one realistic synthetic scene, a 3000× compression relative to LLFF.All compared single-scene methods require at least 12 hours to train, whereas LLFF processes a small input dataset in under 10 minutes.
- Ablation studies: Positional encoding and view dependence provide the largest ablation benefits, followed by hierarchical sampling.The complete model is compared with versions removing these components on eight realistic synthetic scenes.
- Ablation studies: Increasing positional-encoding frequencies beyond 10 does not improve performance, while using only 5 frequencies reduces it.The authors relate the benefit limit to the maximum frequency present in the sampled images, roughly 1024 in their data.
7 Conclusion
The paper concludes that 5D neural radiance fields improve continuous scene representations and rendering quality over discretized voxel-prediction approaches. It also identifies efficient optimization, rendering, and interpretability as open challenges.
- Conclusion: 5D neural radiance fields use an MLP to output volume density and view-dependent radiance from 3D location and 2D viewing direction.This representation is presented as producing better renderings than deep convolutional networks that output discretized voxel representations.
- Conclusion: The approach directly addresses deficiencies in prior MLP-based continuous representations while improving renderings over voxel-based convolutional methods.The conclusion frames this as progress toward continuous scene representations for complex geometry and appearance.
- Future work: Efficient optimization and rendering of neural radiance fields remain important areas for future work.The authors also note that neural-network weight representations make quality and failure-mode analysis less clear than for voxel grids and meshes.
A Additional Implementation Details
The implementation uses a simple fully connected architecture and continuous ray queries, with scene-dependent bounds and regularization. Rendering is computationally expensive despite hierarchical sampling.
- Volume bounds: Synthetic scenes are queried inside a cube of side length 2 centered at the origin, while real scenes use normalized device coordinates for unbounded depth ranges.The volume bounds are chosen differently for synthetic and real-image experiments.
- Training details: For real scenes, unit-variance Gaussian noise is added to density outputs before ReLU during optimization to slightly improve novel-view quality.This regularization is applied only to the real-scene training setup described.
- Rendering details: Rendering uses 64 coarse and 192 fine samples per ray, totaling 256 network queries per ray.The coarse and fine networks use hierarchical sampling at test time.
- Network architecture: The network is fully connected, with color-coded inputs, hidden layers, outputs, activations, and concatenation in the architecture visualization.Figure 7 distinguishes ReLU, linear, sigmoid, and concatenation operations and shows vector dimensions.
- Rendering details: 150–200 million network queries are required per rendered image, taking approximately 30 seconds per frame on an NVIDIA V100.The estimate covers the realistic synthetic and real-scene datasets.
B Additional Baseline Method Details
The evaluation uses authors’ released implementations or pretrained models for NV, SRN, and LLFF, with each baseline trained or applied to individual scenes. SRN’s implementation imposes a lower evaluation resolution because of GPU-memory requirements.
- Baseline implementations: NV and SRN are evaluated using code released by their authors, following each method’s single-scene training procedure.NV is trained without time dependence; SRN is trained on a single scene.
- Baseline implementations: LLFF is evaluated using the pretrained model released by its authors.
- Evaluation constraints: 512 × 512 pixels is the maximum SRN resolution even across 4 NVIDIA V100 GPUs, versus higher resolutions for the other methods.The reported SRN metrics use 512 × 512 pixels on synthetic data and 504 × 376 pixels on real data, while other methods use 800 × 800 and 1008 × 752 respectively.
C NDC ray space derivation
The derivation maps camera-space rays into normalized device coordinates so projected rays trace the same points while using a reparameterized sampling coordinate. For forward-facing real captures, the far bound is taken as infinity, making linear NDC sampling correspond to linear disparity sampling in the original space.
- Projection and ray equivalence: NDC projection maps the viewing frustum into the cube [−1, 1]3 while preserving the projected ray’s traced points under reparameterization.For each original ray o+td, the derivation seeks o′ and d′ such that π(o+td)=o′+t′d′, with rates allowed to differ.
- Ray origin: The NDC ray origin is chosen so t′ = 0 and t = 0 correspond to the same projected point, making o′ equal to the projection of the original ray origin.The derivation substitutes the ray’s initial parameter values into the projection equation to obtain the new origin.
- Camera model: The standard pinhole camera model reparameterizes rays using image width W, height H, and focal length fcam.These quantities enter the camera-to-NDC transformation after applying the perspective projection.
- Sampling in NDC: For forward-facing real captures, assuming an infinite far bound simplifies the NDC z constants because NDC represents inverse depth, or disparity.The implementation shifts the ray origin to the near plane before conversion, then samples t′ linearly from 0 to 1 for linear disparity sampling from the near plane to infinity.
D Additional Results
Additional per-scene results are consistent with the aggregate finding that the method quantitatively outperforms the baselines. Although LLFF can obtain slightly better LPIPS values, the method produces better multiview consistency and fewer artifacts in supplementary-video comparisons.
- Per-scene evaluation: Tables 3–6 provide per-scene quantitative results for DeepVoxels, realistic synthetic scenes, real images, and ablations.The DeepVoxels scenes contain diffuse objects with simple geometry; realistic synthetic scenes use more complex geometry and non-Lambertian materials; real scenes are forward-facing cellphone captures.
- Quantitative results: The per-scene breakdown is consistent with the aggregate metrics, where the method quantitatively outperforms all baselines.
- Qualitative comparison: LLFF achieves slightly better LPIPS metrics, while the method achieves better multiview consistency and fewer artifacts in supplementary-video comparisons.