Source-linked AI summary
NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou
TL;DR
Indoor multi-view stereo is challenged by unreliable pixel correspondences and NeRF’s shape-radiance ambiguity. NerfingMVS adapts depth priors from sparse SfM+MVS reconstruction, uses them to guide NeRF volume sampling, and applies confidence-based filtering. The framework improves indoor depth estimation over prior methods while also improving NeRF rendering on seen and novel views.
Problem
Indoor scenes expose limitations of correspondence-based matching and NeRF geometry because poorly textured regions make accurate depth estimation difficult.
Method
The method adapts a monocular depth network using scene-specific SfM+MVS reconstruction, guides NeRF volume sampling with the adapted priors, and filters depths using rendered-image confidence.
Results
The framework significantly improves indoor multi-view depth estimation over prior works and improves NeRF rendering quality on seen and novel views.
Takeaways & Limitations
Direct NeRF optimization over implicit volumes can improve depth quality over adapted priors while retaining or improving neural radiance-field view synthesis.
Takeaways & Limitations
The method remains inefficient and difficult to scale to large datasets despite being 3x faster than NeRF.
Abstract
from arXiv · showhide
In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM+MVS reconstruction from COLMAP. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS.
1. Introduction
Indoor multi-view stereo remains difficult for correspondence-based methods because poorly textured regions, thin structures, and non-Lambertian surfaces undermine pixel matching. NerfingMVS instead guides NeRF optimization with scene-specific depth priors and improves indoor depth estimation and view synthesis.
- Conventional multi-view stereo often struggles with poorly textured regions, thin structures, and non-Lambertian surfaces in real-world indoor environments.
- Learning-based multi-view depth methods can produce inconsistent depth maps across views and violate photometric consistency because inference lacks constraints.
- NerfingMVS directly optimizes neural radiance-field volumes, adapting monocular depth priors from scene-specific SfM+MVS reconstruction to guide NeRF sampling.
- Scene-specific depth-prior adaptation improves depth quality, whereas existing correspondence-based optimization on those adapted priors surprisingly degrades performance.
- Guided optimization significantly improves indoor multi-view depth estimation and improves NeRF rendering quality on both seen and novel views.
2. Related Work
Prior work spans volumetric and per-view reconstruction, neural implicit representations, and view synthesis. NerfingMVS extends NeRF with guided optimization to support accurate multi-view depth estimation while retaining view-synthesis capability.
- Multi-view Reconstruction: Volumetric reconstruction performs global photometric optimization but suffers from large computational complexity, while per-view depth methods rely on accurately matched pixels.
- Neural Implicit Representation: Neural implicit representations provide memory-efficient geometric representations with effectively unlimited resolution across applications including reconstruction, view synthesis, and differentiable rendering.
- Method Overview: Figure 2’s pipeline combines conventional SfM/MVS reconstruction, scene-specific monocular depth adaptation, NeRF sampling guidance, and confidence-based depth filtering.
- Neural Implicit Representation: NerfingMVS enriches NeRF with guided optimization for accurate depth estimation that surpasses leading multi-view depth estimation approaches.
- View Synthesis: Neural radiance fields represent scenes as continuous implicit radiance fields and have demonstrated impressive view-synthesis results.
3. Approach
The method adapts scene-specific monocular depth priors from COLMAP reconstruction and uses them to guide direct NeRF volume optimization for indoor multi-view depth estimation. It addresses NeRF’s shape-radiance ambiguity through adaptive ray sampling based on cross-view errors, while retaining view-synthesis quality.
- 3.1. Overview: Direct optimization over implicit volumes avoids the accurate cross-view pixel matching required by correspondence-based neural optimization.The method performs test-time optimization on NeRF rather than relying on estimated correspondences and depth reprojection.
- 3.1. Overview: The system finetunes a monocular depth network on sparse, geometrically consistent COLMAP depths to obtain scene-specific priors.Sparse depths are produced by projecting fused COLMAP SfM+MVS point clouds and provide supervision for test-time adaptation.
- 3.2. Scene-specific Adaptation of the Depth Priors: Scene-specific adaptation improves depth quality, while existing correspondence-based optimization degrades adapted priors and guided NeRF optimization improves them.The adapted priors can also support novel-view synthesis by using the nearest seen view’s adapted depth prior, outperforming original NeRF in view-synthesis quality.
- 3.3. Guided Optimization of NeRF: Volume rendering estimates RGB and per-view depth as transmittance-weighted sums over sampled points along each ray.The rendered depth is computed as the expectation of sampled depths using the same volume-rendering weights used for RGB.
- 3.3. Guided Optimization of NeRF: NeRF can explain indoor training images with smoothed radiance fields whose geometry is inaccurate, especially in poorly textured regions.Blurred images and large camera motion further aggravate this shape-radiance ambiguity across tested indoor scenes.
- 3.3. Guided Optimization of NeRF: Cross-view geometric consistency errors from adapted priors define adaptive near and far sampling bounds for each camera ray.Low-error pixels receive concentrated sampling around the adapted depth, whereas high-error pixels retain ranges closer to the original NeRF formulation.
4. Experiments
Experiments on ScanNet evaluate depth estimation, optimization variants, component ablations, and view synthesis. The proposed guided NeRF optimization improves depth quality and NeRF rendering across the reported evaluations.
- Experimental Setup: The evaluation randomly selected 8 ScanNet scenes, using 40 images per scene and holding out 1/8 for novel-view synthesis.Images were resized to 484 × 648, and median groundtruth scaling was used for depth evaluation.
- Multi-view Depth Estimation: Our method outperforms state-of-the-art depth estimation methods on ScanNet across all reported metrics.The comparison includes methods trained with ScanNet groundtruth depth supervision, including DeepV2D, DELTAS, and Atlas.
- Multi-view Depth Estimation: The confidence-based filter further smooths per-pixel depth estimates after NeRF optimization.Qualitative ScanNet comparisons report that the unfiltered method already outperforms the compared methods in depth quality.
- Multi-view Depth Estimation: Existing correspondence-based optimization degrades adapted depth priors, whereas guided optimization over NeRF consistently improves them.The degradation is attributed mainly to incorrect flow correspondences, especially in poorly textured indoor regions.
- Ablation Studies: Each system component benefits final depth quality, while adaptive ranges and bounds balance sampling diversity and precision.The bounds αl and αh reduce over-concentrated or overly random sampling, according to the ablation studies.
- View Synthesis: Guided optimization improves NeRF view synthesis on all 8 scenes and achieves comparable or better quality than state-of-the-art novel-view synthesis methods.The method also improves rendering quality on seen views by helping NeRF focus on more informative regions.
5. Conclusion and Future Work
The framework integrates adapted depth priors into NeRF to improve indoor multi-view depth and view synthesis, while remaining difficult to scale to large datasets.
- Guided optimization regularizes NeRF sampling with adapted depth priors to address shape-radiance ambiguity in indoor depth estimation.
- The method significantly improves indoor multi-view depth estimation and reveals that correspondence-based optimization can degrade adapted depth priors because of wrongly estimated flow correspondences.
- 3x faster than NeRF, the optimization remains inefficient and difficult to scale to large datasets.
- Guided optimization improves NeRF view synthesis quality on both seen and novel views.
A. Implementation Details
The implementation follows NeRF with ray sampling, density regularization, positional encoding, and standard depth-evaluation metrics defined over image pixels.
- The system samples 64 points per ray and uses batches of 1024 rays during training.
- Without coarse-to-fine sampling, one NeRF-style network is used to optimize the neural radiance fields.
- Random Gaussian noise with zero mean and unit variance is added to density σ to regularize the network.
- The evaluation includes Abs Rel, Sq Rel, and δ < t, with predicted and ground-truth depths defined over all pixels in the depth image.
B. Baseline Method Details
The evaluation compares the method with conventional, learning-based, monocular, and video-based depth estimation approaches, including established indoor-scene baselines.
- Conventional MVS baselines include COLMAP and ACMP, with ACMP adding planar models for low-textured indoor regions.
- Learning-based MVS baselines include DELTAS and Atlas, both trained on ScanNet with ground-truth depth supervision.
- The monocular baseline is Mannequin Challenge, evaluated directly using its pretrained weights.
- Video-based baselines include CVD and DeepV2D, with scene images ordered according to their timeline.
C. Hyperparameter Analysis
Hyperparameter analysis on scene0521 examines the minimum-error count K and guided-sampling bounds α_l and α_h, finding that unsuitable K values reduce performance.
- The analysis varies the number of minimum errors K and the guided-sampling bounds α_l and α_h on scene0521.
- Both overly small and overly large K values degrade performance.
- Small K can increase the probability of passing the multi-view consistency check even when the estimated depths are incorrect.