PaperlayerView exact revision on arXiv

Source-linked AI summary

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan

arXiv:2103.13415v3cs.CVcs.GR

TL;DR

NeRF’s single-ray sampling can cause blur and aliasing across resolutions, while supersampling is impractical because each ray requires many MLP evaluations. Mip-NeRF learns a continuously scaled radiance field by rendering conical frustums with integrated positional encoding, reducing errors relative to NeRF by 17% on NeRF’s dataset and 60% on a challenging multiscale dataset. It also matches brute-force supersampled NeRF accuracy while running approximately 22× faster.

  • Problem

    NeRF can produce blur and aliasing across scene resolutions, while supersampling is impractical because rendering each ray requires hundreds of MLP evaluations.

  • Method

    Mip-NeRF learns a continuously scaled prefiltered radiance field by querying Gaussian-approximated conical frustums with integrated positional encoding.

  • Results

    Mip-NeRF reduces average error relative to NeRF by 60% on the presented multiscale dataset and 17% on NeRF’s single-scale dataset.

  • Takeaways & Limitations

    Mip-NeRF matches brute-force supersampled NeRF accuracy while being approximately 22× faster, and is approximately 7% faster than NeRF.

Abstract

from arXiv · show

The rendering procedure used by neural radiance fields (NeRF) samples a scene with a single ray per pixel and may therefore produce renderings that are excessively blurred or aliased when training or testing images observe scene content at different resolutions. The straightforward solution of supersampling by rendering with multiple rays per pixel is impractical for NeRF, because rendering each ray requires querying a multilayer perceptron hundreds of times. Our solution, which we call "mip-NeRF" (a la "mipmap"), extends NeRF to represent the scene at a continuously-valued scale. By efficiently rendering anti-aliased conical frustums instead of rays, mip-NeRF reduces objectionable aliasing artifacts and significantly improves NeRF's ability to represent fine details, while also being 7% faster than NeRF and half the size. Compared to NeRF, mip-NeRF reduces average error rates by 17% on the dataset presented with NeRF and by 60% on a challenging multiscale variant of that dataset that we present. Mip-NeRF is also able to match the accuracy of a brute-force supersampled NeRF on our multiscale dataset while being 22x faster.

1. Introduction

NeRF’s single-ray rendering can blur or alias scenes observed at multiple resolutions, while supersampling is too expensive. Mip-NeRF addresses this by learning a continuously scaled, prefiltered radiance field using conical frustums and integrated positional encoding.

  • Motivation: NeRF produces excessive blur in close-up views and aliasing in distant views when training images contain multiple resolutions.Supersampling each pixel with multiple rays is prohibitively expensive because each ray requires hundreds of MLP evaluations.
  • Approach: Mip-NeRF extends NeRF to represent a prefiltered radiance field across a continuous space of scales.It integrates the radiance field over 3D Gaussian regions that approximate pixel-aligned conical frustums.
  • Approach: Mip-NeRF replaces point samples along rays with conical frustums and encodes each region using integrated positional encoding.IPE generalizes NeRF’s positional encoding to compactly featurize a spatial region rather than a single point.
  • Results: 60% average error reduction is achieved relative to NeRF on the presented challenging multiresolution benchmark.The model also improves accuracy on scenes observed at different resolutions and uses one MLP instead of NeRF’s separate coarse and fine MLPs.
  • Results: Mip-NeRF is approximately 7% faster than NeRF and has half as many parameters.Its scale-aware structure permits merging NeRF’s coarse and fine MLPs into a single MLP.

2. Related Work

Prior rendering methods address aliasing through supersampling or prefiltering, but coordinate-based neural view synthesis has lacked an efficient learned multiscale solution. Mip-NeRF adapts cone-based prefiltering to image-based scene recovery while supporting continuous scales and learned geometry.

  • Anti-aliasing in Rendering: Supersampling reduces aliasing by casting multiple rays per pixel but increases runtime linearly with the supersampling rate.Prefiltering instead uses lowpass-filtered scene content and selects an appropriate scale at render time.
  • Mip-NeRF’s Position: Mip-NeRF learns the prefiltered scene during training because the recovered geometry is unknown beforehand.This distinguishes it from graphics pipelines that can precompute multiscale representations for predefined geometry.
  • Mip-NeRF’s Position: Mip-NeRF represents scale continuously with one neural scene model rather than multiple copies at discrete mipmap scales.The model can be queried at arbitrary scales.
  • Anti-aliasing in Rendering: Traditional multiscale prefiltering efficiently anti-aliases meshes and voxel grids, whereas coordinate-based neural representations generally require expensive supersampling.This mismatch is especially problematic because neural representations already render slowly.
  • NeRF Preliminaries: NeRF renders pixels by sampling positions along camera rays, applying positional encoding, querying an MLP for density and color, and compositing the results.Its rendering and training procedure uses numerical volume integration and separate coarse and fine MLPs for hierarchical sampling.

3. Method

Mip-NeRF replaces NeRF’s point samples along rays with scale-aware representations of conical frustums, enabling anti-aliased rendering across continuously varying scene scales. It approximates each frustum with a Gaussian and uses integrated positional encoding so a single MLP can model multiple scales.

  • Cone Tracing: Mip-NeRF casts cones and divides them into conical frustums, allowing the representation to encode each pixel’s viewed volume rather than only its centroid.The frustums are approximated with multivariate Gaussians for efficient processing.
  • Gaussian Approximation: The conical frustum is represented by a Gaussian whose mean and covariance summarize its position, longitudinal variance, and radial variance.The covariance is transformed into world coordinates before feature construction.
  • Integrated Positional Encoding: Integrated positional encoding computes the expected positional encoding of coordinates distributed according to the frustum Gaussian.The Gaussian is lifted into the Fourier-feature basis, after which sine and cosine expectations have closed-form expressions.
  • Integrated Positional Encoding: IPE softly suppresses frequencies that vary repeatedly across an interval while preserving frequencies that remain nearly constant, producing anti-aliased features that encode volume size and shape.This makes the frequency limit L effectively unnecessary to tune because it can be set extremely large.
  • Architecture: Cone casting and IPE encode scale in the input features, enabling mip-NeRF to use one multiscale MLP instead of NeRF’s separate coarse and fine MLPs.The single-MLP design cuts model size in half and improves rendering accuracy and sampling efficiency.

4. Results

Mip-NeRF is evaluated on multiscale and single-scale Blender datasets using error metrics, runtime, parameter counts, visualizations, and supersampled comparisons. It consistently outperforms NeRF, with the largest gains on multiscale scenes.

  • Evaluation: The evaluation reports PSNR, SSIM, LPIPS, an average error metric, runtimes, and network parameter counts.The average error metric summarizes the three error metrics, while runtime variability is reported with median absolute deviation.
  • Multiscale Blender Dataset: The multiscale Blender benchmark combines original images with versions downsampled by factors of 2, 4, and 8.The modified camera intrinsics correspond to the downsampling factors.
  • Multiscale Blender Dataset: 60% average error reduction lets mip-NeRF outperform NeRF by a large margin across all metrics and scales on the multiscale task.Pixel centering improves NeRF substantially but does not approach mip-NeRF.
  • Ablations: Cone-casting and IPE features are the primary factors driving mip-NeRF’s multiscale performance, while area loss also contributes substantially.Removing IPE reduces performance to that of centered NeRF.
  • Single-scale Blender Dataset: Mip-NeRF reduces average error by approximately 17% versus NeRF on the single-scale Blender dataset and remains faster.The improvement is most visible for small or thin structures such as LEGO truck holes and ship ropes.
  • Supersampling: Mip-NeRF nearly matches supersampled NeRF accuracy while rendering the test set 22× faster.The supersampled comparison uses seconds per megapixel for rendering time.

5. Conclusion

Mip-NeRF extends NeRF with explicit reasoning about sampling and scale to address aliasing. It reduces error relative to NeRF across both multiscale and single-scale datasets while improving rendering speed and matching supersampled accuracy much faster.

  • Conclusion: Mip-NeRF casts cones, encodes conical-frustum positions and sizes, and uses one neural network to model scenes at multiple scales.This contrasts with NeRF’s rays, point encodings, and separate networks at distinct scales.
  • Conclusion: 60% lower error on the multiscale dataset and 17% lower error on NeRF’s single-scale dataset summarize mip-NeRF’s accuracy gains.Both comparisons are relative to NeRF.
  • Conclusion: Mip-NeRF is 7% faster than NeRF and matches brute-force supersampled NeRF accuracy while being 22× faster.These results support the efficiency of its multiscale rendering approach.

A. Conical Frustum Integral Derivations

The derivation models points uniformly sampled within a conical frustum and uses their moments to construct a Gaussian approximation. The resulting formulas require numerical stabilization when frustum endpoints are close, and the approximation has a camera-dependent limitation.

  • Frustum parameterization: The conical-frustum parameterization maps (r, t, θ) to Cartesian coordinates and yields the volume element rt^2 dr d t dθ.The Jacobian determinant supplies the volume element used for integration over the frustum.
  • Moments: The derivation computes first and second moments of t, along with second moments of x and y, using symmetry to eliminate cross terms.The mean along the ray direction comes from the first moment of t, while x and y have zero first moments and equal second moments.
  • Gaussian approximation: The conical-frustum moments are converted into a mean and covariance for a random point within the frustum.The mean along the ray direction is the first moment of t, and radial variance follows from the second moment because the first moment of x is zero.
  • Numerical stability: Numerical evaluation of the raw moment formulas can produce 0 or NaN when t0 and t1 are near each other, so the equations are reparameterized by center and spread.Such endpoint configurations occur frequently during training and can cause training to fail.
  • Limitation: The Gaussian approximation becomes inaccurate when the base and top radii differ substantially, especially near a wide-FOV camera center of projection.The paper notes this is uncommon in most datasets but may matter for unusual settings such as macro photography with a fisheye lens.

B. The L Hyperparameter in PE and IPE

Mip-NeRF's integrated positional encoding makes feature frequencies depend on the represented Gaussian region rather than requiring a carefully tuned positional-encoding degree. Consequently, large L values remain effective in mip-NeRF while large values can overfit NeRF.

  • Feature representation: Integrated positional encoding generalizes positional encoding by representing a position together with its Gaussian covariance.Conventional positional encoding is recovered when the mean equals x and the covariance is zero.
  • Empirical comparison: In NeRF, PSNR peaks over a range of L values, while values that are too large or too small reduce performance.The reported sweep uses test-set PSNR for a single lego scene.
  • Empirical comparison: In mip-NeRF, performance is unaffected when L is set arbitrarily large, so the parameter need not be tuned; the experiments use L = 16.The paper states that L = 16 makes the last dimension of every training IPE feature less than numerical epsilon.

C. Hyperparameters

The experiments retain NeRF's hyperparameters to isolate mip-NeRF's cone-casting and integrated positional encoding contributions. Sampling uses 128 samples at each of two levels, with only selected parameters hand-tuned on the lego validation set.

  • Experimental controls: Mip-NeRF uses the same hyperparameter set as Mildenhall et al. to isolate the effects of cone-casting and integrated positional encoding.The relevant behavior is governed by the sample count, histogram padding, and a λ hyperparameter.
  • Sampling: N = 128 samples are drawn at each of the two levels, matching NeRF's total number of MLP evaluations.The sample count was not tuned and was selected to equalize evaluation cost.
  • Tuning: The α and λ hyperparameters were briefly tuned by hand on the lego validation set before the experiments.The paper explicitly states that N was not tuned.

D. Forward-Facing Scenes

The paper does not evaluate mip-NeRF on forward-facing LLFF scenes, where NeRF uses normalized device coordinates. A cylinder-casting variant in NDC roughly matches NeRF's accuracy, while mip-NeRF's main contributions target multiscale aliasing more broadly.

  • Evaluation scope: The paper does not evaluate on LLFF, a forward-facing handheld-cellphone dataset.This is an explicit evaluation-scope boundary.
  • Coordinate systems: LLFF scenes use normalized device coordinates, which warp frustum-shaped space into a unit cube and are restricted to forward-facing scenes.Scenes with substantial camera rotation instead use conventional 3D world coordinates.
  • NDC experiment: A cylinder-casting mip-NeRF variant in NDC achieves average PSNR 26.838 versus JaxNeRF's 26.843 on this task.The variant roughly matches NeRF's accuracy, with its main apparent benefit being removal of positional-encoding L tuning.
  • Core contributions: Cone tracing, integrated positional encoding, and a single unified multiscale model jointly target multiscale data and aliasing reduction.The unified model replaces NeRF's separate per-scale models.

E.3. Optimization

The experiments use a standardized JaxNeRF optimization setup, including learning-rate warm-up, while comparing mip-NeRF with brute-force supersampling.

  • Training procedure: All mip-NeRF and JaxNeRF experiments use 1 million Adam iterations, batch size 4096, and logarithmic annealing from η0 = 5 · 10−4 to ηn = 5 · 10−6.The learning rate is additionally warmed up for the first 2500 iterations, scaling it from 0.01 to 1.
  • Training procedure: The warm-up procedure does not improve mip-NeRF itself but stabilizes some ablations, so it is applied across mip-NeRF and JaxNeRF experiments for fair comparison.
  • Training procedure: The warm-up and annealing schedule is visualized in Figure 8.
  • Model details: Mip-NeRF handles viewing directions as NeRF does, normalizing and positionally encoding the ray direction before predicting color.The direction encoding uses L = 4 and is injected into the MLP after τ prediction.
  • Comparison baseline: Brute-force supersampling roughly matches mip-NeRF’s performance but is 22× slower and requires oracle scale information.

G. Alternative Gaussian Positional Encoding

An alternative Gaussian positional encoding performs comparably to IPE, but IPE is preferred for compactness, interpretability, and fewer hyperparameters; the comparison supports explicit conical-frustum modeling.

  • Results: Both the alternative encoding and IPE perform comparably on the multiscale and single-scale Blender datasets.
  • Design choice: IPE is preferred because it is more compact, easier to justify as an expectation over a conical frustum, and free of hyperparameters.The alternative is sensitive to its two L hyperparameters and covariance parameterization choices.
  • Interpretation: The lack of improvement from modeling covariance correlations may suggest that correlation is not a useful cue, contrary to the authors’ intuition.
  • Interpretation: The experiment reinforces the claim that mip-NeRF’s gains arise from explicitly modeling conical frustums rather than using point samples along rays.

H. Additional Results

Additional per-scene and visual results show mip-NeRF’s error reduction across the multiscale Blender scenes and compare renderings across scales and datasets.

  • Multiscale Blender Dataset: Mip-NeRF significantly reduces error compared with NeRF across every scene in the multiscale Blender dataset.
  • Multiscale Blender Dataset: Table 6 reports per-scene test-set metrics averaged over the four scales used in the multiscale dataset.
  • Visual comparisons: Figures 9 and 10 compare mip-NeRF, NeRF, and an improved NeRF version with ground truth across eight multiscale Blender scenes.Figure 9 shows cropped regions rendered at four resolutions as image pyramids and reports SSIM for each scale.
  • Blender Dataset: Table 23 reports per-scene test-set results for the single-scale Blender dataset from Mildenhall et al.
Loading 2103.13415v3…