Source-linked AI summary
NeRF++: Analyzing and Improving Neural Radiance Fields
Kai Zhang, Gernot Riegler, Noah Snavely, Vladlen Koltun
TL;DR
NeRF can face shape-radiance ambiguities and parameterization difficulties in large-scale unbounded 360° scenes. NeRF++ analyzes the ambiguity and introduces inverted sphere parameterization, achieving improved quantitative and qualitative view synthesis results in these challenging captures. The report also identifies substantial computational and calibration challenges that remain.
Problem
NeRF can admit degenerate geometry-radiance solutions and struggles to represent both foreground and distant background content in large-scale unbounded 360° scenes.
Method
NeRF++ analyzes NeRF’s failure modes and uses inverted sphere parameterization with separate foreground and background NeRFs for unbounded scenes.
Results
NeRF++ significantly outperforms NeRF and produces higher-fidelity synthesized images for challenging 360° captures in large-scale unbounded scenes.
Takeaways & Limitations
The analysis explains NeRF’s ability to avoid shape-radiance ambiguity, while inverted sphere parameterization extends it to captures requiring faithful foreground and background representation.
Takeaways & Limitations
Training and rendering remain time-consuming and memory-intensive, and small camera calibration errors may impede photorealistic synthesis.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360 capture of bounded scenes and forward-facing capture of bounded and unbounded scenes. NeRF fits multi-layer perceptrons (MLPs) representing view-invariant opacity and view-dependent color volumes to a set of training images, and samples novel views based on volume rendering techniques. In this technical report, we first remark on radiance fields and their potential ambiguities, namely the shape-radiance ambiguity, and analyze NeRF's success in avoiding such ambiguities. Second, we address a parametrization issue involved in applying NeRF to 360 captures of objects within large-scale, unbounded 3D scenes. Our method improves view synthesis fidelity in this challenging scenario. Code is available at https://github.com/Kai-46/nerfplusplus.
1 INTRODUCTION
NeRF++ analyzes why NeRF can synthesize novel views despite shape-radiance ambiguities and introduces inverted sphere parameterization for 360° captures in unbounded scenes. The method improves fidelity by addressing the difficulty of representing both foreground and distant background content.
- Background: NeRF maps 3D coordinates and viewing directions to opacity and color, then renders novel views using conventional volume rendering.The model is fit to training views using an implicit MLP-based 5D function.
- Shape-Radiance Ambiguity: NeRF++ analyzes potential NeRF failure modes and why NeRF’s specific MLP structure avoids them in practice.The report identifies this analysis as one of its primary contributions.
- Shape-Radiance Ambiguity: NeRF can theoretically fit training images with incorrect geometry by using view-dependent radiance, creating shape-radiance ambiguity.Without regularization, such degenerate solutions may fail to generalize to novel views.
- Parameterization of Unbounded Scenes: Standard parameterizations struggle with unbounded 360° scenes: modeling only part causes background artifacts, while modeling everything reduces detail through finite sampling.The trade-off is illustrated as competing coverage and resolution failures.
- Parameterization of Unbounded Scenes: NeRF++ introduces inverted sphere parameterization to represent background content in large-scale unbounded scenes and reports improved quantitative and qualitative results.The reported evaluations use real-world captures from Tanks and Temples and a light field dataset.
2 PRELIMINARIES
NeRF represents opacity and view-dependent radiance with MLPs, optimizes them against posed training images, and renders pixels through volumetric integration. Positional encoding maps positions and directions to Fourier features to support sharper synthesis.
- NeRF representation: NeRF models opacity from 3D position and radiance from both position and viewing direction using implicit MLPs.The opacity field is σ(x), while radiance is c(x, d).
- Optimization: Given posed training images, stochastic gradient descent minimizes the discrepancy between observed and rendered images by optimizing σ and c.
- Volume rendering: Ray tracing renders each pixel by integrating the opacity-weighted radiance field along the camera ray.The ray is parameterized as r = o + td.
- Positional encoding: NeRF uses positional encoding to map x and d to Fourier features, compensating for spectral bias and enabling sharper images.The encoding includes sinusoidal features such as sin(2^0p), cos(2^0p), through a hyper-parameterized frequency range.
- Motivating ambiguity: A synthetic sphere-based solution can fit 50 training images from hemispherical viewpoints while failing on novel views, illustrating the risk of incorrect geometry.The opacity field models a unit sphere instead of the bulldozer shape, while radiance maps sphere intersections and view directions to pixel colors.
3 SHAPE-RADIANCE AMBIGUITY
NeRF can fit training images with incorrect geometry by encoding the discrepancy in view-dependent radiance, creating a shape-radiance ambiguity. Its architecture helps avoid these solutions by limiting radiance complexity with respect to viewing direction, and replacing that structure with a vanilla MLP reduces novel-view generalization.
- Shape-radiance ambiguity: An arbitrary incorrect shape can be paired with radiance fields that perfectly fit training images but generalize poorly to novel views.This degeneracy occurs because view-dependent radiance can compensate for incorrect geometry in the absence of regularization.
- Shape-radiance ambiguity: A unit-sphere reconstruction can reproduce input images by assigning each sphere intersection the corresponding training-pixel color, but requires an extremely complex view-dependent function for novel views.Accurate interpolation is unlikely unless training views are extremely dense.
- Implicit regularization: Incorrect geometry generally forces radiance to vary at higher frequencies with respect to viewing direction, making the solution harder for a limited-capacity MLP to represent.For the correct shape, the surface light field is generally smoother and is constant for Lambertian materials.
- Implicit regularization: NeRF’s specific MLP structure implicitly favors smooth surface reflectance by injecting viewing direction near the network output with fewer parameters and nonlinear activations.The architecture treats scene position and viewing direction asymmetrically.
- Implicit regularization: NeRF uses lower-frequency Fourier features for viewing direction than for position, limiting c(x, d)’s expressivity with respect to d.The cited comparison is γ4(·) for d versus γ10(·) for x.
- Experimental validation: Replacing NeRF’s radiance MLP with a vanilla MLP that treats position and direction symmetrically reduces novel-view test quality on DTU scenes.The experiment keeps σ’s structure unchanged and trains both fields from scratch.
4 INVERTED SPHERE PARAMETRIZATION
NeRF++ addresses the resolution problems of representing large-scale, unbounded 360° scenes by separating bounded foreground content from distant background content and parameterizing them differently.
- Motivation: NeRF’s Euclidean-depth sampling struggles when outdoor 360° scenes contain arbitrarily distant background elements.Large dynamic depth ranges cause severe resolution issues in the volumetric scene representation.
- Scene partitioning: NeRF++ partitions scene space into an inner unit sphere containing foreground and cameras, plus an outer volume containing the remaining environment.The two regions use separate scene representations.
- Rendering: The inner and outer volumes are modeled with separate NeRFs, raycast individually, and composited into the final color.The inner NeRF remains in Euclidean space, while the outer NeRF uses inverted sphere parameterization.
- Outer parameterization: Outer points are represented by bounded direction coordinates (x′, y′, z′) and inverse radius 1/r, improving numeric stability and allocating less resolution to farther objects.The reparameterized quadruple is (x′, y′, z′, 1/r), with all coordinates bounded.
- Rendering integral: For each ray, the unit sphere divides rendering into inner and outer segments, with the outer segment integrated using inverse radius as the variable.The method evaluates outer opacity and radiance at coordinates derived for each sampled 1/r.
- Interpretation: The inverted sphere has a virtual-camera interpretation: the unit sphere acts as an image plane, while 1/r represents inverse depth or disparity.This spherical projection differs from the virtual pinhole-camera interpretation of NDC for forward-facing capture.
5 EXPERIMENTS
NeRF++ is evaluated against NeRF on real-world 360° captures from Tanks and Temples and Light Field datasets. It consistently outperforms NeRF across reported metrics and produces higher-fidelity synthesized images.
- Setup: The evaluation uses PSNR, SSIM, and LPIPS on real-world Tanks and Temples and Light Field captures.Both datasets contain hand-held camera imagery with recovered structure-from-motion poses.
- Datasets: The Tanks and Temples benchmark contains four large-scale 360° scenes: Truck, Train, M60, and Playground.The experiments use provided training/testing images and SfM poses.
- Results: NeRF++ consistently outperforms NeRF in all metrics across the four Tanks and Temples scenes and four Light Field scenes.The comparison covers PSNR, SSIM, and LPIPS.
- Datasets: The Light Field evaluation uses Africa, Torch, Ship, and Basket, with sparse surrounding captures created by temporally subsampling dense image sequences.Training and testing frames are separated by the subsampling pattern.
- Results: NeRF++ synthesized images have significantly higher fidelity than NeRF in the challenging unbounded-scene 360° capture setting.The paper reports this result alongside the quantitative comparison in Table 2.
6 OPEN CHALLENGES
The paper identifies substantial computational, calibration, and photometric challenges that remain for NeRF and NeRF++.
- Computation: Training and testing a single large-scale scene remain time-consuming and memory-intensive.Training NeRF++ on four RTX 2080 Ti GPUs takes approximately 24 hours.
- Computation: Rendering a 1280x720 image takes approximately 30 seconds on one RTX 2080 Ti GPU, remaining far from real-time.Prior work has accelerated inference, but the paper still describes rendering as non-real-time.
- Calibration: Small camera calibration errors may impede photorealistic synthesis.The paper suggests robust loss functions as a possible response.
- Photometry: Auto-exposure and vignetting are photometric effects that may need to be modeled to increase image fidelity.The paper relates this direction to prior work on lighting changes.