Source-linked AI summary
Visibility-aware Multi-view Stereo Network
Jingyang Zhang, Yao Yao, Shiwei Li, Zixin Luo, Tian Fang
TL;DR
Learning-based MVS methods rarely model pixel-wise visibility explicitly, so occluded pixels can affect cost aggregation. Vis-MVSNet jointly estimates pair-wise depth and matching uncertainty, uses uncertainty to weight multi-view cost-volume fusion, and reports strong benchmark performance, especially under severe occlusion.
Problem
Learning-based MVS methods rarely explicitly handle pixel-wise visibility, while aggregating features from invisible views can deteriorate reconstruction.
Method
Vis-MVSNet jointly infers pair-wise depth and uncertainty, then uses uncertainty-guided weighting to fuse pair-wise latent volumes before final regularization and depth regression.
Results
Vis-MVSNet ranks first on the Tanks and Temples online benchmark, achieving a mean F-score of 60.03, with improvements especially evident under severe occlusion.
Takeaways & Limitations
Explicit uncertainty-guided fusion provides an effective visibility-aware depth inference framework across DTU, BlendedMVS, and Tanks and Temples evaluations.
Takeaways & Limitations
The method assumes that lower-entropy depth distributions indicate higher confidence because unimodality is treated as a confidence indicator.
Abstract
from arXiv · showhide
Learning-based multi-view stereo (MVS) methods have demonstrated promising results. However, very few existing networks explicitly take the pixel-wise visibility into consideration, resulting in erroneous cost aggregation from occluded pixels. In this paper, we explicitly infer and integrate the pixel-wise occlusion information in the MVS network via the matching uncertainty estimation. The pair-wise uncertainty map is jointly inferred with the pair-wise depth map, which is further used as weighting guidance during the multi-view cost volume fusion. As such, the adverse influence of occluded pixels is suppressed in the cost fusion. The proposed framework Vis-MVSNet significantly improves depth accuracies in the scenes with severe occlusion. Extensive experiments are performed on DTU, BlendedMVS, and Tanks and Temples datasets to justify the effectiveness of the proposed framework.
1 Introduction
Learning-based MVS methods often aggregate multi-view evidence without explicitly modeling pixel-wise visibility, allowing occluded or invisible pixels to distort reconstruction. Vis-MVSNet addresses this by jointly estimating pair-wise depth and uncertainty, then using uncertainty-guided fusion to improve reconstruction, particularly under severe occlusion.
- Pixel-wise visibility is critical in MVS, but it is unknown before dense 3D recovery, creating a chicken-and-egg problem.
- Many learning-based MVS methods aggregate features from all views with variance, averaging, or max pooling without explicit visibility handling.
- Vis-MVSNet jointly estimates pair-wise depth and uncertainty, then uses uncertainty as weighting guidance to fuse pair-wise latent volumes into a multi-view cost volume.
- The end-to-end network trains its uncertainty component without visibility labels, requiring only ground-truth depth maps from existing MVS datasets.
- Vis-MVSNet ranks first on the Tanks and Temples online benchmark and improves most when input images contain severe occlusion.
2 Related Work
Prior MVS research includes learned cost volumes, recurrent or point-based refinement, coarse-to-fine reconstruction, and classical visibility estimation. Vis-MVSNet connects visibility estimation with learned uncertainty so pair-wise volumes can be fused more selectively.
- Learning-based MVS: Learning-based MVS methods learn feature matching, cost-volume regularization, differentiable projection, recurrent inference, point refinement, or conditional random-field optimization.
- Learning-based MVS: Recent learning-based systems use coarse-to-fine strategies with feature pyramids and progressively refined depth hypotheses.
- Visibility Estimation: Classical visibility estimation uses heuristic cost thresholds or EM-based joint depth-visibility recovery, but probabilistic formulations are difficult to integrate directly with deep networks.
- Visibility Estimation: Existing deep methods often handle visibility implicitly through variance-based aggregation or max pooling, relying on network regularization or selecting a best latent representation.
- Framework: The framework jointly infers pair-wise depth and uncertainty, uncertainty-weighted fusion, and subsequent regularization for final depth regression.
- Uncertainty Estimation: Vis-MVSNet indicates visibility through pair-wise matching uncertainty and uses that uncertainty to guide fusion of pair-wise latent volumes.
3 Method
Vis-MVSNet estimates pair-wise depth and uncertainty, then uses uncertainty-aware fusion to reduce the influence of likely occluded pixels across a coarse-to-fine reconstruction pipeline.
- 3.4 Coarse-to-fine Architecture: The network extracts multi-scale features and performs coarse-to-fine depth estimation across three resolutions, using previous depth outputs to construct later cost volumes.The final depth map is produced at the third stage.
- 3.1 Overview: For each reference-source pair, groupwise correlations across depth hypotheses form a pair-wise cost volume with 8 correlation groups.The first stage uses predetermined hypotheses, while later stages dynamically determine them from preceding depth estimates.
- 3.2 Uncertainty Estimation: Pair-wise depth is regressed from the probability-volume expectation, while uncertainty is inferred from its entropy through a shallow 2D CNN.The model jointly learns depth and uncertainty under a Laplacian likelihood, with log uncertainty used in practice.
- 3.3 Volume Fusion: The latent pair-wise volumes are fused by a normalized weighted sum whose weights decrease with estimated uncertainty.This attenuates pixels with high uncertainty, which are more likely to lie in occluded regions, while avoiding an empirical hard threshold.
- 3.5 Training Loss: Training combines pair-wise L1 losses, pair-wise joint losses, and final-depth L1 losses across the three stages.The pair-wise L1 terms help prevent uncertainty learning from over-relaxing pair-wise depth estimation.
4 Experiment
Experiments on Tanks and Temples, DTU, and BlendedMVS evaluate Vis-MVSNet’s reconstruction quality and visibility-aware fusion. The method ranks first on Tanks and Temples, remains comparable on DTU, and improves ablation results over alternative fusion strategies.
- Benchmarking on Tanks and Temples Dataset: Vis-MVSNet achieves a mean F-score of 60.03 and ranks 1st among methods on the Tanks and Temples benchmark.The model is trained on BlendedMVS and applied directly without fine-tuning.
- Benchmarking on DTU Dataset: On DTU, Vis-MVSNet achieves an overall score of 0.365, comparable with other state-of-the-art methods.The model uses DTU training data for this benchmark.
- Ablation Study: The ablation study evaluates average L1 loss, sub-one-pixel accuracy, and related depth-map metrics on the BlendedMVS validation set.The comparison removes the coarse-to-fine architecture to isolate volume-fusion effects.
- Ablation Study: Variance fusion assumes all reference pixels are visible, so increasing the number of input images can worsen evaluation metrics.This baseline is widely used in MVSNet and related methods.
- Ablation Study: Uncertainty-guided weighted averaging explicitly reduces invisible pixels’ influence during pair-wise volume fusion.The proposed fusion improves over averaging and max pooling, while the full coarse-to-fine model achieves the strongest result.
5 Conclusion
The paper presents a visibility-aware depth-inference framework for multi-view stereo. It jointly estimates pair-wise depth and uncertainty, uses uncertainty for weighted volume fusion, and reports effectiveness across several datasets.
- Conclusion: Vis-MVSNet combines two-step cost-volume regularization, joint pair-wise depth and uncertainty inference, and uncertainty-weighted fusion.The framework is evaluated on several datasets to demonstrate visibility-aware depth inference.