Source-linked AI summary
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs
Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, Noha Radwan
TL;DR
Sparse input views cause NeRF to produce incorrect geometry and unstable early optimization, reducing novel-view quality. RegNeRF regularizes geometry and appearance on unseen-view patches while annealing the sampling space, and it outperforms comparable single-scene methods and often pre-trained conditional models. Its predictions may remain blurry in unobserved areas containing fine geometric structures.
Problem
NeRF’s novel-view quality drops significantly with sparse inputs because sparse-view optimization is not biased toward 3D-consistent solutions.
Method
RegNeRF regularizes geometry and appearance on patches rendered from unseen viewpoints and anneals scene sampling bounds to stabilize optimization.
Results
RegNeRF outperforms other single-scene methods and, in many cases, conditional models pre-trained on large multi-view datasets.
Takeaways & Limitations
Regularizing unseen views and annealing sampling produces 3D-consistent representations that support high-quality sparse-view novel synthesis.
Takeaways & Limitations
Predictions may be blurry in unobserved areas containing fine geometric structures.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) have emerged as a powerful representation for the task of novel view synthesis due to their simplicity and state-of-the-art performance. Though NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available, its performance drops significantly when this number is reduced. We observe that the majority of artifacts in sparse input scenarios are caused by errors in the estimated scene geometry, and by divergent behavior at the start of training. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints, and annealing the ray sampling space during training. We additionally use a normalizing flow model to regularize the color of unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but in many cases also conditional models that are extensively pre-trained on large multi-view datasets.
1. Introduction
NeRF performs well with dense views but degrades sharply with sparse inputs because optimization can produce incorrect geometry and divergent early behavior. RegNeRF addresses this by regularizing unseen-view patches and annealing ray sampling to improve sparse-view novel synthesis.
- NeRF’s novel-view quality drops significantly when only a few input views are available.
- RegNeRF samples unobserved viewpoints and regularizes geometry and appearance on rendered patches.
- A normalizing flow model regularizes predicted colors by maximizing the log-likelihood of rendered patches.
- Sparse-view artifacts are linked primarily to incorrectly optimized scene geometry and divergent behavior at the start of training.
- Scene-space annealing starts ray sampling within a small range and expands to the full scene bounds to prevent early divergence.
2. Related Work
Prior work uses coordinate-based neural representations and conditional models to address novel-view synthesis, but sparse-input methods face supervision, resolution, pre-training, and domain-generalization constraints. RegNeRF instead regularizes scene geometry and appearance during per-scene optimization without depth input or expensive pre-training.
- Neural Representations: Coordinate-based neural representations encode 3D geometry and color in neural-network weights, producing compact scene representations.
- Neural Representations: mip-NeRF improves robustness across camera distances, but neither mip-NeRF nor NeRF produces realistic novel views in the paper’s much sparser setting.
- Sparse Input Novel-View Synthesis: Conditional radiance-field models require multi-view datasets containing many scenes for pre-training, which may not be available.
- Sparse Input Novel-View Synthesis: Conditional models may degrade when test-time data come from a different domain than the training data.
- Sparse Input Novel-View Synthesis: RegNeRF avoids expensive pre-training, uses only RGB images rather than depth input, and regularizes unseen-view geometry and appearance.
3. Method
RegNeRF addresses sparse-view NeRF failures by regularizing geometry and appearance in patches from unobserved viewpoints, while annealing ray sampling to stabilize early training. It builds on mip-NeRF and evaluates object-focused reconstruction to reduce DTU background bias.
- Sparse-view NeRF can produce degenerate novel views because sparse reconstruction supervision does not enforce a 3D-consistent solution.
- RegNeRF samples small patches from relevant unobserved camera poses and regularizes them for smooth geometry and high-likelihood colors.
- Geometry Regularization: Depth smoothness regularization incorporates a piece-wise smooth geometry prior into rendered patches from unobserved viewpoints.
- Color Regularization: Color regularization maximizes the likelihood of rendered patches using a RealNVP flow trained on patches from the JFT-300M dataset.
- Sample Space Annealing: Sample-space annealing begins with a restricted region and expands to the full scene bounds, avoiding early divergence and degenerate solutions.
- Evaluation: On DTU, object-focused PSNR changes from 8.79 to 20.08 for RegNeRF, whereas PixelNeRF drops from 17.37 to 16.89 under the same comparison.
4. Experiments
Experiments show that RegNeRF improves sparse-input view synthesis across DTU and LLFF, with smoother geometry, sharper renderings, and strong data efficiency. Ablations indicate that scene-space annealing and geometry regularization are especially important.
- View Synthesis from Sparse Inputs: On DTU, RegNeRF is comparable to pre-trained conditional models with 3 input views and achieves the best overall quantitative results with 6 or 9 views.It also strongly outperforms mip-NeRF and DietNeRF, which do not use pre-training.
- View Synthesis from Sparse Inputs: On LLFF, RegNeRF achieves the best results for all input-view scenarios, including the out-of-distribution setting for conditional models trained on DTU.Conditional models can overfit to DTU, while mip-NeRF and DietNeRF are weak with 3 views but improve with 6 or 9.
- Geometry Prediction: RegNeRF produces smoother scene geometry and higher-quality novel views, whereas mip-NeRF retains floating artifacts as input views increase from 3 to 6 or 9.The experiments directly correlate geometry quality with view synthesis quality.
- Data Efficiency: 55% fewer input views are sufficient for RegNeRF to match mip-NeRF’s mean PSNR on the DTU test set in sparse settings.The data-efficiency gap is larger with fewer input views.
- View Synthesis from Sparse Inputs: On DTU visualizations, RegNeRF produces sharp novel views, while baselines show blurriness or incorrect geometry; DietNeRF blends input images with 3 views.The reported failure is attributed to DietNeRF’s global semantic consistency loss.
- Ablation Studies: Ablations find scene-space annealing crucial for avoiding degenerate solutions, geometry regularization more impactful than appearance regularization, and the full combination best-performing.Among geometry priors, the proposed depth smoothness strategy achieves the best performance; alternative opacity, density, and normal smoothness priors do not produce accurate geometry.
5. Conclusion
RegNeRF addresses sparse-input NeRF by regularizing geometry and appearance in unseen-view patches and annealing the sampling space, enabling high-quality novel views. The method outperforms single-scene optimization methods and often extensively pre-trained conditional models, but can produce blurry predictions in unobserved areas with fine geometric structures.
- RegNeRF regularizes geometry and appearance from unseen-view patches and uses sample-space annealing to learn 3D-consistent representations for high-quality novel views.Its key insight is that sparse-input artifacts arise from incorrectly optimized geometry and divergent early optimization behavior.
- The model outperforms methods that optimize a single scene and, in many cases, conditional models extensively pre-trained on large-scale multi-view datasets.
- RegNeRF does not hallucinate geometric detail and may therefore produce blurry predictions in unobserved areas containing fine geometric structures.The authors identify uncertainty prediction and generative components as promising future work.