Source-linked AI summary
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman
TL;DR
Grid-based NeRFs accelerate training but lack scale awareness, causing aliasing, while mip-NeRF 360’s anti-aliasing is not directly compatible with grids. Zip-NeRF combines mip-NeRF 360 with Instant NGP using multisampling and prefiltering, achieving 8%–77% lower error rates and 24× faster training than mip-NeRF 360.
Problem
Grid-based features queried at single coordinates cannot reason about scale or aliasing, while mip-NeRF 360’s scale-aware approach is incompatible with current grid-based techniques.
Method
Zip-NeRF integrates iNGP grids into mip-NeRF 360 using multisampling, feature downweighting, and prefiltered proposal supervision.
Results
8%–77% lower error rates and 24× faster training than mip-NeRF 360 are reported across the paper’s benchmarks.
Takeaways & Limitations
Zip-NeRF combines scale-aware anti-aliasing with fast grid-based NeRF training within a single model.
Takeaways & Limitations
The most severe missing-content z-aliasing artifact is difficult to measure in small still-image benchmarks because it appears only at certain distances.
Abstract
from arXiv · showhide
Neural Radiance Field training can be accelerated through the use of grid-based representations in NeRF's learned mapping from spatial coordinates to colors and volumetric density. However, these grid-based approaches lack an explicit understanding of scale and therefore often introduce aliasing, usually in the form of jaggies or missing scene content. Anti-aliasing has previously been addressed by mip-NeRF 360, which reasons about sub-volumes along a cone rather than points along a ray, but this approach is not natively compatible with current grid-based techniques. We show how ideas from rendering and signal processing can be used to construct a technique that combines mip-NeRF 360 and grid-based models such as Instant NGP to yield error rates that are 8% - 77% lower than either prior technique, and that trains 24x faster than mip-NeRF 360.
1. Preliminaries
Mip-NeRF 360 provides scale-aware anti-aliasing through conical frustums, while Instant NGP accelerates training with multiresolution grids. Zip-NeRF combines their frameworks but must address aliasing in grid features and proposal supervision.
- Mip-NeRF 360 and Instant NGP render pixels by casting rays, featurizing samples along them, and alpha-compositing neural-network outputs.
- Mip-NeRF 360 represents ray intervals as conical frustums and uses integrated positional encoding with a large MLP.
- Instant NGP interpolates a hierarchy of differently sized 3D grids and processes the resulting feature vectors with a small MLP.
- Zip-NeRF addresses the incompatibility by integrating iNGP’s grid featurization into mip-NeRF 360’s scale-aware framework.
- Grid features are aliased because they are queried at single coordinates rather than over sub-volumes, preventing the model from reasoning about scale.
2. Spatial Anti-Aliasing
Zip-NeRF converts each conical frustum into multisampled isotropic Gaussians and averages scale-dependent, downweighted grid features. This approximates scale-aware prefiltered features while retaining iNGP-style grid processing.
- Mip-NeRF’s integrated positional encoding represents sub-volume scale, whereas iNGP’s point interpolation cannot natively reason about scale or aliasing.
- Zip-NeRF approximates each conical frustum with points treated as isotropic Gaussians, then averages downweighted grid features to obtain scale-aware prefiltered features.
- Each frustum is sampled with a 6-point hexagonal pattern whose coordinates match the frustum’s moments and are randomly or deterministically rotated and flipped.
- For each grid scale, features are downweighted according to the fraction of a multisample’s Gaussian lying inside the interpolated grid cell.
- Naive trilinear interpolation remains aliased at high frequencies, motivating scale-dependent feature reweighting.
- The weighted multisample features are averaged, concatenated across grid levels, and supplied to an MLP as in iNGP.
3. Z-Aliasing and Proposal Supervision
Zip-NeRF addresses z-aliasing in proposal supervision by smoothing NeRF histograms before resampling them into proposal intervals. It also uses a flexible distance normalization for near and far scene content.
- Mip-NeRF 360’s proposal MLP can learn non-smooth densities, causing rays to skip scene content as sampling is repeatedly resampled along the ray.
- The interlevel loss causes z-aliasing because it assigns equal treatment to partially and completely overlapping histogram bins and penalizes only complete non-overlap.
- Zip-NeRF smooths the NeRF histogram with a rectangular-pulse convolution and resamples the result into the proposal histogram’s intervals.
- The anti-aliased loss compares the resampled NeRF weights with proposal weights in a shared coordinate space, making supervision smooth with respect to ray distance.
- The power transformation maps metric distance to normalized distance while interpolating between linear-, logarithmic-, inverse-, and inverse-square-like behavior.
- The chosen normalization remains roughly linear near the ray origin and becomes between inverse and inverse-square toward farther normalized distances.
4. Results
Zip-NeRF combines grid-based training with anti-aliasing and achieves strong accuracy, scale robustness, and training-speed improvements across the evaluated benchmarks.
- Implementation: Zip-NeRF reimplements iNGP’s grid-and-hash pyramid within the mip-NeRF 360 codebase, with anti-aliasing modifications and normalized feature-code weight decay.The architecture otherwise follows mip-NeRF 360, while separate proposal iNGPs and MLPs are used for proposal sampling.
- 360 Dataset: On the 360 benchmark, Zip-NeRF reduces RMSE, DSSIM, and LPIPS by 11%, 17%, and 19% versus mip-NeRF 360, respectively.It trains 24× faster than mip-NeRF 360, but is approximately 6× slower to train than iNGP.
- Multiscale 360 Dataset: The multiscale benchmark evaluates four bicubically downsampled image scales, increasing difficulty by requiring generalization across scales.The evaluated scale factors are [2] [4].
- Multiscale 360 Dataset: On the multiscale benchmark, Zip-NeRF reduces RMSE by 8% at the finest scale and 17% at the coarsest scale versus mip-NeRF 360.Against the mip-NeRF 360 + iNGP baseline, RMSE is 19% lower at the finest scale and 55% lower at the coarsest scale; coarsest-scale DSSIM and LPIPS are both 77% lower.
- Sample Efficiency: Zip-NeRF’s anti-aliased interlevel loss degrades gracefully as ray samples decrease, whereas mip-NeRF 360’s loss fails catastrophically below 16 samples.The comparison uses mean and interquartile range of test-set PSNR while reducing samples from 32 to 2.
- Limitations: The most severe z-aliasing artifact—missing scene content—is difficult to measure in small still-image benchmarks because disappearance occurs only at certain distances.The paper uses supplemental video to demonstrate this artifact and its mitigation.
5. Conclusion
The conclusion presents Zip-NeRF as a unified approach to scale-aware anti-aliasing and fast grid-based NeRF training. It reports lower error rates and faster training than prior techniques while highlighting broader analysis of spatial and z-aliasing.
- 5. Conclusion: Zip-NeRF integrates scale-aware anti-aliased NeRFs with fast grid-based NeRF training using multisampling and prefiltering.The paper frames these areas as previously divergent.
- 5. Conclusion: Zip-NeRF achieves error rates 8%–77% lower than prior techniques while training 24× faster than mip-NeRF 360.The conclusion identifies mip-NeRF 360 as the previous state of the art on the paper’s benchmarks.
- 5. Conclusion: The paper’s analysis addresses both spatial aliasing in learned coordinate-to-color-and-density mappings and z-aliasing in online-distillation losses.The authors present these tools as supporting further work on quality, speed, and sample efficiency.
A. Video Results
The supplement presents video results for benchmark scenes and newly captured, more challenging scenes to qualitatively demonstrate aliasing and Zip-NeRF’s mitigation.
- A. Video Results: Supplemental videos show scenes from the 360 benchmark alongside new, more challenging captured scenes.The new scenes are used for qualitative demonstrations rather than the paper’s quantitative experiments.
- A. Video Results: The video results demonstrate various kinds of aliasing and Zip-NeRF’s ability to ameliorate them.The supplement specifically includes video evidence for qualitative inspection.
- A. Video Results: For video results only, the authors use per-image GLO appearance embeddings to model view-dependent exposure and lighting variation.This approach does not improve quantitative metrics and is not used for the paper’s experiments.
B. Multisampling Pattern Derivation
The multisampling derivation constructs efficient patterns whose moments match the conical frustum, then warps them into the frustum geometry used by Zip-NeRF.
- Pattern Criteria: The multisampling pattern is designed for uniform coverage along the ray and around its angles, matching the conical frustum’s mean and covariance with few points.Samples are placed at distances proportional to the cone radius at their respective positions.
- Cylindrical Construction: The derivation first constructs an n-point cylindrical pattern with zero mean and identity covariance before forming the conical-frustum pattern.This simplifies the analysis of the multisampling geometry.
- Angle Choices: For small n under uniformly distributed angles, the zero-mean and identity-covariance property appears only for n = 6 and two specific angle choices.The paper uses one of these choices because the alternative has potentially undesirable higher-order correlations between adjacent angles.
- Frustum Warping: The cylindrical samples are warped, shifted, and scaled into a conical frustum so their along-ray and total perpendicular variances match the analytical frustum values.The mismatch between full covariances approaches zero as t ≫ r.
C. Scale Featurization
The model incorporates scale information into grid features by averaging and concatenating featurized scale values, with normalization based on learned feature statistics. An approximation to erf is used for speed without discernible quality loss.
- Scale values are featurized, averaged, and concatenated with grid features as inputs to the MLP.The scale features are shifted and scaled to [−1, 1] before normalization.
- Feature scaling uses the standard deviation of each grid’s values, padded by the initialization magnitude to prevent collapse toward zero.A stop-gradient operator is used in the scaling expression.
- An approximation to erf is used when computing the downweighting factor for speed.The approximation replaces erf(x) with a sign- and exponential-based expression.
- The erf approximation has no discernible impact on quality and only a very marginal impact on performance.
D. Spatial Contraction
The method contracts unbounded scene coordinates into a bounded domain whose resolution allocation follows perspective projection. Gaussian scales are contracted by linearizing the contraction and matching generalized variance with an isotropic approximation.
- Mip-NeRF 360 uses spatial contraction to parameterize unbounded scenes with bounded coordinates.
- The contraction maps [−∞, ∞]d to [−2, 2]d, allocating capacity near the origin and roughly disparity-proportional capacity to distant content.
- Gaussian means are contracted directly, while Gaussian scales are approximated by linearizing the contraction with its Jacobian.
- The contracted isotropic scale is computed from the geometric mean of the Jacobian’s eigenvalues, equivalently the dth root of the absolute determinant.Here d = 3 because the coordinates are three-dimensional.
- This isotropic approximation matches mip-NeRF 360’s generalized-variance treatment while requiring significantly less computation, with a further closed-form acceleration for the specific contraction.
F. Power Transformation Details
The power transformation curves metric distance into a normalized space for resampling and supervision, and it is also applied to mip-NeRF 360’s distortion loss. Its parameter controls a broad family of distance-response shapes.
- The transformation includes special cases at λ = −∞, 0, and +∞, alongside a general finite-λ expression.
- Its slope is 1 near the origin, while λ controls shapes ranging from exponential and squared to logarithmic and inverse families.Scaling the input controls the range over which the transformation remains approximately linear.
- The transformation is related to the robust loss ρ(x, λ, 1) = P(x^2/2, λ/2) and resembles a modified Yeo–Johnson transformation.The modification preserves a straight-line behavior near the origin.
- The power transformation curves metric distance into normalized space for effective resampling and interlevel supervision.
- Applied to distortion loss, a steep near-origin gradient that tapers toward log-distance more aggressively penalizes floaters.
G. Model Details
The model uses a multi-round proposal-sampling pipeline with grid hierarchies, distinct NGPs, and a larger view-dependent MLP. Training and ablation settings include fixed iteration schedules and memory-adjusted supersampling experiments.
- Training setup: Training uses 25k iterations with batch size 216, Adam optimization, logarithmic learning-rate decay from 10^-2 to 10^-3, and a 5k-iteration warm-up.
- Grid hierarchy: The iNGP hierarchy contains 10 grid scales from 16 to 8192 with 4 channels per level, using 128^3 hashing for larger grids.
- Sampling pipeline: The pipeline performs two proposal-sampling rounds followed by a final NeRF rendering round, using a distinct NGP and MLP for each round.
- Distortion loss: A power-transformed distortion loss can heavily penalize distant content under linear metric distance while ignoring near-camera floaters.
- Sampling pipeline: Proposal NGPs truncate grid hierarchies at maximum sizes of 512 and 2048 and use one feature channel because they predict density only.
- View dependence: A larger view-dependent MLP addresses iNGP’s bottleneck, which limits complicated view-dependent effects and is associated with shiny-surface degradation and floaters.
- Ablations: Naive and jittered 6× supersampling ablations used half-sized batches and twice as many iterations after exhausting memory.
H. Results
The paper reports per-scene and average evaluations across the mip-NeRF 360 and Blender datasets, including both single-scale and multiscale settings.
- Tables 3 and 4 report per-scene performance on the mip-NeRF 360 dataset for single-scale and multiscale settings, respectively.
- The reported results cover indoor and outdoor scenes in the mip-NeRF 360 dataset.
- Table 5 reports per-scene and average performance on the Blender dataset.