PaperlayerView exact revision on arXiv

Source-linked AI summary

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman

arXiv:2111.12077v3cs.CVcs.GR

TL;DR

Unbounded scenes can produce floaters and background collapse in NeRF reconstructions. mip-NeRF 360 combines a non-linear scene parameterization, online distillation, and distortion regularization, reducing mean-squared error by 57% relative to mip-NeRF while synthesizing realistic views and complex depth maps.

  • Problem

    Ill-posed unbounded-scene reconstruction can produce floaters and background collapse in trained NeRFs.

  • Method

    mip-NeRF 360 uses a Kalman-like scene parameterization, proposal-based coarse-to-fine online distillation, and a regularizer for mip-NeRF ray intervals.

  • Results

    57% reduction in mean-squared error relative to mip-NeRF, while producing realistic novel views and complex depth maps for challenging unbounded real-world scenes.

  • Takeaways & Limitations

    The method synthesizes realistic novel views and complex depth maps for unbounded real-world scenes with unconstrained camera orientations.

  • Takeaways & Limitations

    Thin structures and fine details may be missed, quality may degrade when the camera moves far from the scene center, and training requires several hours on an accelerator.

Abstract

from arXiv · show

Though neural radiance fields (NeRF) have demonstrated impressive view synthesis results on objects and small bounded regions of space, they struggle on "unbounded" scenes, where the camera may point in any direction and content may exist at any distance. In this setting, existing NeRF-like models often produce blurry or low-resolution renderings (due to the unbalanced detail and scale of nearby and distant objects), are slow to train, and may exhibit artifacts due to the inherent ambiguity of the task of reconstructing a large scene from a small set of images. We present an extension of mip-NeRF (a NeRF variant that addresses sampling and aliasing) that uses a non-linear scene parameterization, online distillation, and a novel distortion-based regularizer to overcome the challenges presented by unbounded scenes. Our model, which we dub "mip-NeRF 360" as we target scenes in which the camera rotates 360 degrees around a point, reduces mean-squared error by 57% compared to mip-NeRF, and is able to produce realistic synthesized views and detailed depth maps for highly intricate, unbounded real-world scenes.

1. Preliminaries: mip-NeRF

Mip-NeRF represents each ray with conical-frustum intervals, integrated positional encodings, and volumetric rendering. Coarse sampling is refined using weights concentrated around scene content, with reconstruction losses applied at both scales.

  • Ray representation: Each ray is divided into intervals, whose conical-frustum means and covariances are encoded before MLP prediction.The frustum radii depend on camera focal length and pixel size.
  • Volume rendering: The MLP predicts density and color, which are combined through numerical quadrature to render each pixel.Alpha-compositing weights sum to at most 1.
  • Coarse-to-fine sampling: Coarse weights define an inverse-transform sampling histogram for fine ray distances, concentrating samples near scene content.Training uses stochastic sampling, whereas evaluation uses evenly spaced coarse samples.
  • Optimization: Mip-NeRF optimizes its MLP with a weighted combination of coarse and fine image-reconstruction losses.Both coarse and fine renderings are supervised against the ground-truth ray color.

2. Scene and Ray Parameterization

Mip-NeRF 360 extends mip-NeRF to unbounded scenes by smoothly transforming Gaussian volumes into a bounded contracted space and sampling rays linearly in disparity. These choices allocate spatial detail more effectively across nearby and distant content.

  • Gaussian transformation: A smooth coordinate transformation is linearized with its Jacobian to transform Gaussian means and covariances.The transformed covariance uses the Jacobian evaluated at the Gaussian mean.
  • Gaussian transformation: The contraction is Kalman-like and maps unbounded coordinates into a bounded ball while preserving nearby points.The figure shows unaffected points inside radius 1 and contracted centers inside radius 2.
  • Feature encoding: Mip-NeRF 360 applies integrated positional features in the contracted coordinate space rather than Euclidean space.The contracted Gaussian parameters are passed to the feature encoding.
  • Ray sampling: Ray distances are sampled linearly in disparity because uniform distance sampling is unsuitable for scenes unbounded in every direction.This explicitly generalizes the inverse-depth spacing associated with NDC.
  • Ray sampling: Uniform samples in normalized ray distance produce t-distances distributed linearly in disparity when g(x) = 1/x.The invertible mapping connects normalized distances s in [0, 1] with the near and far ray bounds.

3. Coarse-to-Fine Online Distillation

Mip-NeRF 360 replaces mip-NeRF’s repeatedly supervised coarse-to-fine MLP with a small proposal MLP and a large NeRF MLP. Online distillation trains the proposal network to bound the NeRF network’s weights while the NeRF network remains reconstruction-supervised.

  • Architecture: The proposal MLP predicts density-derived weights for resampling, while the NeRF MLP predicts the final weights, colors, and rendered image.Only the final NeRF MLP produces color.
  • Architecture: A large NeRF MLP paired with a small proposal MLP increases capacity while keeping repeated proposal evaluations tractable.The architecture avoids repeatedly using the large model for coarse rendering.
  • Online distillation: Online distillation trains both networks jointly, with proposal histograms learning to envelope the NeRF histogram weights.The proposal network is not trained directly against the input image.
  • Histogram consistency: The proposal loss compares histograms with potentially different intervals by penalizing proposal mass that fails to bound NeRF weights.The consistency condition must hold for every NeRF interval.
  • Histogram consistency: The asymmetric loss permits proposal overestimation because proposal intervals are expected to be coarser than NeRF intervals.A stop-gradient makes the NeRF network lead while the proposal network follows.

4. Regularization for Interval-Based Models

Mip-NeRF 360 introduces a distortion regularizer for interval-based rays to address floaters and background collapse caused by reconstruction ambiguity. The loss compacts weighted ray intervals and empirically suppresses these artifacts more effectively than density-noise injection.

  • Motivation: Ill-posed reconstruction can produce floaters and background collapse, where dense content explains views as disconnected or camera-near clouds.Both artifacts are visible in depth maps and arise in trained NeRFs.
  • Distortion loss: The regularizer is defined over normalized ray distances and weights represented as a one-dimensional step function.Using normalized distances avoids disproportionately upweighting distant intervals.
  • Distortion loss: The distortion loss is minimized by setting weights to zero when possible or consolidating nonzero weights into the smallest region available.Its gradient also narrows intervals and pulls separated intervals together.
  • Effect: Compared with density-noise injection, the regularizer more effectively suppresses floaters and background collapse while preserving distant-tree depth detail.The noise-based approach only partially removes artifacts and reduces reconstruction quality.
  • Distortion loss: A tractable reformulation separates weighted distances between interval midpoints from weighted individual-interval sizes.Because the step function is constant within each interval, the reformulation is straightforward to compute.

5. Optimization

The model uses a proposal MLP to efficiently generate ray intervals for a larger NeRF MLP, with Charbonnier reconstruction loss and a distortion regularizer trained under a specified optimization schedule.

  • Optimization: The model uses a 4-layer proposal MLP and an 8-layer NeRF MLP, with two proposal resampling stages followed by one NeRF evaluation stage.The proposal MLP has 256 hidden units, while the NeRF MLP has 1024 hidden units; each proposal stage uses 64 samples.
  • Optimization: Proposal supervision uses histogram weights from the NeRF MLP, while reconstruction uses Charbonnier loss and the distortion term is weighted by λ = 0.01.The stop-gradient in the proposal loss keeps proposal and NeRF parameter optimization independent.
  • Optimization: Training runs for 250k Adam iterations with batch size 2^14 and a learning rate annealed from 2 × 10^-3 to 2 × 10^-5.The schedule includes a 512-iteration warm-up and gradient clipping to a norm of 10^-3.

6. Results

On a novel nine-scene dataset, mip-NeRF 360 is compared with NeRF-like and alternative view-synthesis methods, showing strong overall accuracy and speed-quality trade-offs while retaining specific limitations.

  • Dataset and comparisons: The evaluation dataset contains 9 scenes—5 outdoors and 4 indoors—with complex central content and detailed backgrounds.Capture settings were controlled to reduce photometric variation, lighting variation, and moving objects.
  • Dataset and comparisons: The comparison includes NeRF, mip-NeRF, NeRF++, a DONeRF-parameterized NeRF, Stable View Synthesis, Deep Blending, and Point-Based Neural Rendering.NeRF-like baselines are evaluated with additional positional encoding frequencies to bound the full scene.
  • Quantitative results: 57% lower mean squared error than mip-NeRF is achieved, with a 2.17× increase in training time.The model outperforms prior NeRF-like models by a significant margin; larger mip-NeRF and NeRF++ baselines remain approximately 3× slower while less accurate.
  • Ablation study: Removing proposal supervision, the distortion regularizer, or integrated positional encoding reduces performance or introduces geometric floaters.Using one shared MLP increases training time by approximately 3×, while a smaller NeRF MLP reduces quality.
  • Ablation study: DONeRF-style parameterization reduces accuracy, while the model improves SSIM and LPIPS but not PSNR on front-facing LLFF scenes.The LLFF evaluation also reports a significant speedup when the NeRF MLP is large.
  • Limitations: Thin structures and fine details can be missed, quality may degrade when the camera moves far from the scene center, and training requires several hours on an accelerator.Examples include bicycle tire spokes and stump-scene leaf veins.

7. Conclusion

The paper presents mip-NeRF 360 as an extension of mip-NeRF for real-world scenes with unconstrained camera orientations. Its parameterization, proposal-based distillation, and regularization support realistic views and complex depth maps with lower error than mip-NeRF.

  • 7. Conclusion: Mip-NeRF 360 combines a Kalman-like scene parameterization, proposal-based coarse-to-fine distillation, and a regularizer designed for mip-NeRF ray intervals.The method targets challenging unbounded real-world scenes with unconstrained camera orientations.
  • 7. Conclusion: 57% lower mean-squared error than mip-NeRF is reported alongside realistic novel views and complex depth maps.These results are reported for challenging unbounded real-world scenes.

A. Additional Model Details

Additional model details refine positional encoding, sampling, proposal-histogram processing, and background compositing for the unbounded-scene pipeline.

  • Off-Axis Positional Encoding: Off-axis integrated positional encoding captures anisotropic Gaussian shapes that axis-aligned features cannot distinguish.Ablating off-axis features lowers bicycle-scene SSIM from 0.687 to 0.664.
  • Off-Axis Positional Encoding: An optimized covariance computation makes off-axis features only modestly more expensive than mip-NeRF’s axis-aligned features.The alternative computes an equivalent expression using element-wise products and row summation.
  • Proposal processing: Proposal weights are annealed from a flat distribution at training start toward the learned proposal distribution at training end.This encourages exploration during training through a Schlick bias schedule.
  • Proposal processing: Proposal histograms are dilated before resampling to counteract rotational aliasing caused by supervision from input-pixel rays.The dilated density is converted back to a normalized histogram for sampling.
  • Proposal processing: The dilation factor is tied to the expected histogram-bin size through hyperparameters a = 0.5 and b = 0.0025.The factor is adjusted at each coarse-to-fine resampling level according to preceding sample counts.
  • Sampling: Midpoint-based resampling avoids the histogram erosion and asymmetric gap coverage produced when sampled distances directly define interval endpoints.The method replaces mip-NeRF’s sampled endpoints with endpoints formed from midpoints of sorted samples.
  • Background Colors: Random RGB backgrounds encourage opaque scene reconstruction and improve the usefulness of median or mean ray-termination distances for depth maps.Fixed black or white backgrounds can produce incorrectly semi-transparent backgrounds.

B. Implementation Details

The Kalman-like reparameterization can use automatic differentiation to compute the Jacobian, or a less expensive function-based matrix multiplication alternative.

  • Autodiff frameworks can straightforwardly compute the Jacobian J_f(µ) used by the Kalman-like reparameterization.
  • A cheaper alternative avoids explicitly constructing the Jacobian matrix by applying a function equivalent to multiplication with J_f(µ).

C. Proposal Supervision Visualization

The proposal-supervision loss uses histogram bounds to encourage consistency between differently spaced histograms, with Figure 10 illustrating when the bound is exact or requires correction.

  • The proposal MLP loss is motivated by bounds between histograms of one-dimensional data.
  • The bound is guaranteed when two histograms arise from the same underlying true distribution.
  • Figure 10 contrasts identical point sets, where the bound is an upper bound and the loss is zero, with differing point sets.
  • Minimizing excess histogram mass that violates the bound encourages consistency between histograms with differently spaced bin locations.

D. Additional Results

Additional experiments describe the capture protocol, compare performance across datasets and baselines, and expose evaluation complications from photometric variation and metric disagreement.

  • Dataset construction: The authors locked ISO, white balance, shutter speed, aperture, and focus after configuring a reference view to limit photometric variation during capture.
  • Dataset construction: The dataset contains 100–330 images per scene, with capture taking 1–20 minutes; COLMAP supplies camera poses and every eighth image is held out for testing.
  • Dataset construction: The authors transform and rescale COLMAP poses to better fit the scene and align the coordinate frame with approximately circular camera orbits.
  • Additional results: Per-scene PSNR, SSIM, and LPIPS performance is roughly consistent with each technique’s reported average performance.
  • Additional results: On Blender, the model matches mip-NeRF in accuracy while proposal networks make training significantly faster, with speedups increasing for larger NeRF MLPs.
  • Additional results: On LLFF, the model does not improve PSNR over mip-NeRF but improves SSIM and LPIPS and significantly accelerates training with large NeRF MLPs.
  • Evaluation caveats: Tanks and Temples has varying photometric properties and overexposed images, making view-synthesis evaluation ill-posed without deciding which photometric condition to replicate.
  • Evaluation caveats: Color correction makes the model and SVS roughly quantitatively comparable, with the model slightly ahead in PSNR but behind in SSIM and LPIPS.

E. Potential Negative Impact

The paper identifies potential negative impacts from neural rendering, including misuse for deep fakes, surveillance, and visual effects that may affect artists’ employment. Training also requires substantial computation and energy, which may raise climate concerns.

  • Neural rendering techniques may contribute to deep fakes that could mislead people.
  • Accurate scene reconstruction could be used for surveillance, with negative impacts if deployed negligently or maliciously.
  • Generating visual effects with the system may negatively affect job opportunities for artists.
  • Training NeRF requires multiple hours of accelerator optimization, consuming energy that may concern climate impacts depending on its production.
Loading 2111.12077v3…