Source-linked AI summary
Dense Depth Priors for Neural Radiance Fields from Sparse Input Views
Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, Matthias Nießner
TL;DR
NeRF requires many RGB views for high-quality room-scale reconstruction, where sparse indoor observations create correspondence ambiguity. The paper completes free SfM depth into uncertain dense priors and uses them to guide NeRF optimization, enabling synthesis from as few as 18 images. The method reports higher novel-view image quality and more accurate depth than approaches using SfM or MVS depth.
Problem
NeRF requires many views for room-scale reconstruction, while sparse indoor observations introduce correspondence ambiguity from limited overlap, textureless regions, and inconsistent colors.
Method
The method completes sparse SfM depth into dense depth and uncertainty estimates, then uses them for NeRF depth constraints and guided scene sampling without additional scene depth sensors.
Results
18–36 images enable novel view synthesis for complete room-size scenes, with higher image quality and more accurate depth than approaches using SfM or MVS depth.
Takeaways & Limitations
Dense depth priors with uncertainty support data-efficient NeRF reconstruction of complete room-size scenes from very few RGB views.
Takeaways & Limitations
Long optimization times and slow rendering remain, and the depth prior network requires a larger training dataset despite needing no additional scene depth input.
Abstract
from arXiv · showhide
Neural radiance fields (NeRF) encode a scene into a neural representation that enables photo-realistic rendering of novel views. However, a successful reconstruction from RGB images requires a large number of input views taken under static conditions - typically up to a few hundred images for room-size scenes. Our method aims to synthesize novel views of whole rooms from an order of magnitude fewer images. To this end, we leverage dense depth priors in order to constrain the NeRF optimization. First, we take advantage of the sparse depth data that is freely available from the structure from motion (SfM) preprocessing step used to estimate camera poses. Second, we use depth completion to convert these sparse points into dense depth maps and uncertainty estimates, which are used to guide NeRF optimization. Our method enables data-efficient novel view synthesis on challenging indoor scenes, using as few as 18 images for an entire scene.
1. Introduction
The paper targets room-scale novel view synthesis from few RGB views, addressing NeRF’s difficulty with sparse, ambiguous indoor observations. It introduces dense depth priors with uncertainty to guide NeRF optimization without additional scene depth input.
- Room-scale novel view synthesis is pursued using few input views to reduce data, acquisition time, cost, and static-scene requirements.
- NeRF struggles with tens of images because RGB-only correspondence is ambiguous, especially under inside-out room views, textureless walls, and inconsistent colors.
- The method completes sparse SfM depth into dense depth and per-pixel uncertainty, then uses both to constrain optimization and guide scene sampling.
- 18–36 images enable NeRF-based novel view synthesis for complete room-size scenes using dense depth priors with uncertainty estimates.
- The contributions include data-efficient room-scale synthesis, enhancement of noisy SfM depth, and uncertainty-aware depth guidance.
2. Related Work
Related work spans geometry-based view synthesis and depth-supervised NeRF methods. The paper positions its approach as combining SfM depth completion, uncertainty modeling, and depth supervision for few-input scenes.
- Geometry-based view synthesis uses scene geometry to reproject observed images when views are sparse, with heuristic or learned blending.
- Sparse-view NeRF can produce floaters because its high-capacity geometry and appearance model is underconstrained.
- NerfingMVS uses MVS depth to overfit a scene-specific depth predictor that guides NeRF sampling, whereas this method completes SfM depth and adds depth supervision.
- Compared with direct sparse-depth supervision weighted by reprojection error, dense uncertain depth priors aim to provide more detailed views, accurate geometry, and robustness to SfM outliers.
3. Method
The method derives sparse depth from SfM, predicts dense depth with uncertainty, and incorporates these priors into NeRF through guided sampling and a depth constraint. A radiance-field MLP jointly models color and density while accounting for view-specific effects.
- Method: SfM preprocessing supplies camera poses, intrinsics, and sparse depth maps from the input RGB images.
- Depth Completion with Uncertainty: SfM depth can contain outliers and only 0.04% valid pixels on average for 18–20-image ScanNet scenes, motivating uncertainty prediction and sparse-input completion.
- Depth Completion with Uncertainty: The depth prior network predicts dense depth and pixelwise standard deviation from RGB images and sparse depth maps.
- Depth Completion with Uncertainty: CSPN refinement propagates information spatially and iteratively to make predictions sharper and handle extremely sparse input depth.
- Network Training: The network is trained on RGB-D data with subsampled and perturbed sensor depth designed to resemble sparse SfM input, using Gaussian negative log likelihood.
- Scene Representation: The NeRF MLP predicts color and volume density from position and viewing direction, while omitting viewing-direction positional encoding for angularly undersampled scenes.
- Scene Representation: Per-image latent codes let the radiance field compensate for view-specific lighting and lens-shading inconsistencies.
- Radiance Field with Dense Depth Priors: The radiance field combines color MSE with a depth GNLL, encouraging termination within one target standard deviation while retaining flexibility for color reconstruction.
4. Results
Experiments on ScanNet and Matterport3D evaluate sparse-depth construction, baseline comparisons, and component ablations for room-scale NeRF novel-view synthesis. The method outperforms baselines across metrics, reduces artifacts, and benefits from completion, uncertainty, GNLL, and latent codes, while retaining stated computational and data-scope limitations.
- Depth Priors: Matterport3D produces more accurate dense depth priors because its sparse depth is higher quality, while ScanNet interpolation avoids a relevant accuracy drop despite noisier SfM depth.Table 1 reports depth-prior accuracy on the three ScanNet and three Matterport3D scenes used for NeRF.
- Baseline Comparison: Our method outperforms NeRF, DS-NeRF, and NerfingMVS in all reported metrics.NeRF and the proposed method are evaluated on both datasets, while DS-NeRF and NerfingMVS are run on ScanNet.
- Baseline Comparison: Dense depth priors with uncertainty strongly reduce floaters, improve color and depth accuracy, and increase robustness to sparse-depth outliers.The method also better preserves object boundaries that are absent from sparse depth input, such as the chair back in the qualitative comparison.
- Ablation Study: Removing completion causes inaccurate, less sharp outputs; removing uncertainty causes wrong edges, duplication, and missing detail, especially with lower-quality ScanNet depth.The full method achieves the best image quality and depth estimates in the ablation experiments.
- Ablation Study: Replacing GNLL with MSE weakens density constraints behind surfaces, while removing latent codes causes inconsistent colors and visible training-image frustums.The GNLL issue is most visible for tangential novel views, such as views into the corridor.
5. Conclusion
The method enables NeRF-based novel view synthesis for complete rooms from only 18 to 36 input images by leveraging dense depth priors with uncertainty. These priors guide optimization and improve novel-view image quality and depth estimates without requiring scene depth sensors.
- 18 to 36 input images suffice for reconstructing a complete room with the proposed dense-depth-prior NeRF method.The depth prior network uses sparse SfM reconstruction rather than depth sensor input.
- Dense depth priors with uncertainty guide NeRF optimization when only a few input views are available.The method derives dense guidance from sparse SfM reconstruction through a learned depth prior.
- The method achieves significantly higher novel-view image quality and more accurate depth estimates than approaches using SfM or multi-view stereo output in NeRF.
A.1. ScanNet [6]
The ScanNet evaluation uses selected sharp video frames, calibrated downsampled images, and complete large-room scenes. It also measures unfiltered SfM depth error against sensor depth across three evaluation scenes.
- Motion Blur Detection: The sharpest frame is selected from each window of 10 or 20 consecutive video frames using Laplacian-variance sharpness.Frames are converted to grayscale, convolved with a discrete Laplacian kernel, and evaluated by variance.
- Image Resolution: 468×624 images are used after downsampling and cropping dark calibration borders.
- Test Scenes: Three complete, sufficiently large rooms—scene0710 00, scene0758 00, and scene0781 00—are used for evaluation.
- SfM Quality on Few Views: SfM depth error is measured as mean absolute error against sensor depth on 6291 points from the three ScanNet evaluation scenes.All COLMAP SfM points are projected into corresponding input views and passed to depth completion without filtering; the maximal error is 5.85m.
A.2. Matterport3D [2]
The Matterport3D evaluation uses controlled train/test overlap, calibrated 504×630 images, and selected bounded indoor scenes. The benchmark contains three named house regions and avoids unbounded open space because it is challenging for NeRF.
- Train/Test Image Selection: 25% of train pixels are unseen by other train views, 45% are seen by one other view, and 30% are seen by two or more.
- Evaluation Metric: Table 4 reports the impact of sparse depth density on Matterport3D using depth RMSE measured in meters.
- Train/Test Image Selection: Test views have 67% average overlap with their most overlapping train view.
- Image Resolution: 504×630 images are used after downsampling and cropping dark calibration borders.
- Test Scenes: The evaluation avoids unbounded open space because it is challenging for NeRF approaches.
- Test Scenes: The Matterport3D test scenes are Region 5, house VzqfbhrpDEA; Region 2, house Vvot9Ly1tCj; and Region 19, house Vvot9Ly1tCj.
B. Impact of Sparse Depth Density
Reducing sparse depth density lowers performance, but depth completion retains an advantage even when the input is extremely sparse. At 0.01% density, the method receives about 32 points per image on average.
- 0.1%, 0.05%, and 0.01% sparse depth densities are evaluated on Matterport3D.
- Reduced sparse depth lowers performance, while depth completion increases the value of very sparse depth input.
- 32 points per image on average at 0.01% density still outperform the version without depth completion.The authors note that view-consistent monocular depth estimation may be challenging at this sparsity.
C.1. Our Method
The method builds on NeRF with a depth-completion network that predicts depth and uncertainty, then uses these estimates to guide optimization and scene sampling. The section also describes adaptations needed for room-scale baseline evaluation and reports implementation settings.
- Radiance Fields: NeRF uses an eight-layer MLP with positional encoding to represent radiance and density fields.The eighth layer feeds separate density and feature pathways, with density produced using softplus activation.
- Baseline Settings: Table 5 reports depth loss weights λ used in the experiments.
- Depth Completion: The depth-completion network uses a ResNet-18 encoder and a second upsampling branch for uncertainty estimation.Both branches use skip connections and CSPN modules configured with 48 depth iterations and 24 standard-deviation iterations.
- Baseline Adaptation: The NerfingMVS error-map baseline required adaptation because its original computation was not robust for entire rooms.The section introduces the room-scale problem before describing the adapted baseline calculation.
- Baseline Adaptation: The original NerfingMVS error map averages the four smallest normalized depth-reprojection errors for each input view.Errors are computed by projecting 3D points from other views using the depth prior.
- Baseline Adaptation: Room-scale reprojections can place points behind the camera, producing negative error maps and near planes that yield invalid sampling ranges.In this setting, the far bound can lie in front of the near bound.
- Baseline Settings: For ScanNet scenes, a depth loss weight of 0.1 was suitable for the adapted baseline.The baseline also used the method’s positional-encoding frequencies, which improved performance.
- Baseline Settings: The DS-NeRF baseline likewise used the method’s positional-encoding frequencies, which improved its performance.