Source-linked AI summary
Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation
Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, Ronggang Wang
TL;DR
Learning-based MVS depth estimation faces a trade-off: regression can predict sub-pixel depth but tends to overfit, whereas classification is robust but discrete. The paper proposes Unification and Unified Focal Loss, combining them in UniMVSNet, which ranks first on DTU and Tanks and Temples.
Problem
Regression methods tend to overfit from indirect cost-volume learning, while classification methods cannot directly infer exact depth because their predictions are discrete.
Method
The paper introduces Unification, a continuous Unity representation for the optimal depth hypothesis and its proximity to ground truth, plus Unified Focal Loss for imbalanced continuous labels.
Results
UniMVSNet achieves state-of-the-art performance on DTU and Tanks and Temples, ranking first on both benchmarks.
Takeaways & Limitations
Unification and Unified Focal Loss are plug-and-play modules that improve MVS without additional memory or computational costs.
Takeaways & Limitations
The paper identifies GFL as imperfect for scaling hard and easy continuous-label samples because it ignores ground-truth magnitude.
Abstract
from arXiv · showhide
Depth estimation is solved as a regression or classification problem in existing learning-based multi-view stereo methods. Although these two representations have recently demonstrated their excellent performance, they still have apparent shortcomings, e.g., regression methods tend to overfit due to the indirect learning cost volume, and classification methods cannot directly infer the exact depth due to its discrete prediction. In this paper, we propose a novel representation, termed Unification, to unify the advantages of regression and classification. It can directly constrain the cost volume like classification methods, but also realize the sub-pixel depth prediction like regression methods. To excavate the potential of unification, we design a new loss function named Unified Focal Loss, which is more uniform and reasonable to combat the challenge of sample imbalance. Combining these two unburdened modules, we present a coarse-to-fine framework, that we call UniMVSNet. The results of ranking first on both DTU and Tanks and Temples benchmarks verify that our model not only performs the best but also has the best generalization ability.
1. Introduction
Learning-based MVS traditionally estimates depth by regression or classification, but each representation sacrifices either robustness or exactness. The paper introduces Unification, Unified Focal Loss, and UniMVSNet to combine direct cost-volume constraints with sub-pixel prediction.
- Existing representations: Regression uses Soft-argmin weighting to achieve sub-pixel depth, but indirect cost-volume supervision and weight-combination ambiguity tend to cause overfitting.Different weight combinations can produce the same depth, complicating model convergence.
- Existing representations: Classification directly constrains the probability volume with cross-entropy and provides confidence, but its discrete prediction cannot infer exact depth directly.It selects the depth hypothesis with maximum probability.
- Unification: Unification represents the optimal hypothesis and its offset using at most one continuous Unity target based on proximity to ground truth.The loss acts directly on the regularized probability volume while retaining continuous depth information.
- Unified Focal Loss: Unified Focal Loss addresses category and hardness imbalance for continuous labels, while treating traditional Focal Loss as a special case and GFL as an imperfect expression.Its design responds to the limitation that GFL ignores ground-truth magnitude when distinguishing hard and easy samples.
- UniMVSNet: UniMVSNet combines Unification and Unified Focal Loss in a coarse-to-fine framework and achieves state-of-the-art performance on DTU and Tanks and Temples.The framework replaces traditional depth representations and adopts Unified Focal Loss for optimization.
2. Related Works
Multi-view stereo methods include volumetric, point-cloud, mesh, and depth-map representations, with depth maps offering a flexible intermediate representation. Figure 2 illustrates the proposed Unification around the optimal depth hypothesis.
- Traditional MVS methods: Traditional MVS methods are categorized as volumetric, point-cloud-based, mesh-based, or depth-map-based according to their output scene representation.Depth-map methods operate in 2D and can support later point-cloud, volumetric, or mesh reconstruction.
- Traditional MVS methods: Depth-map-based MVS is described as the most flexible classic representation because estimated maps can be merged into consistent 3D reconstructions.The cited reconstruction forms include point clouds, volumes, and meshes.
- Unification: Figure 2 identifies the (m −1)th depth hypothesis as optimal for the red point in the Unification illustration.
3. Methodology
UniMVSNet replaces conventional depth regression or classification with a unified representation that directly optimizes the cost volume while retaining sub-pixel depth estimation. It combines this representation with Unified Focal Loss in a coarse-to-fine pipeline.
- Learning-based MVS pipeline: Learning-based MVS constructs a cost volume from multi-view image features and estimates depth from sampled depth hypotheses.Features are warped into the reference camera frustum, aggregated across views, and regularized into a probability volume.
- Existing representations: Regression estimates depth by weighting hypotheses with Soft-argmin, enabling sub-pixel prediction but risking overfitting under indirect cost-volume supervision.The predicted depth is the weighted sum of discrete hypotheses, with weights expected to favor low-cost hypotheses.
- Existing representations: Classification predicts each hypothesis probability and selects the maximum, directly constraining the cost volume but producing discrete depth estimates.Its training uses cross-entropy between the predicted and one-hot probability volumes.
- Unified depth representation: Unification recasts depth estimation as multi-label classification while regressing a continuous proximity target for the optimal hypothesis containing the ground-truth depth.The target has at most one non-zero continuous value, representing the optimal hypothesis location and its offset from ground truth.
- Unified depth representation: Unity regression selects the hypothesis with maximum unity and fuses its estimated offset to produce the final depth.The representation uses sigmoid operators to estimate unity values across hypotheses, then applies the regression procedure.
- Unified Focal Loss: Unified Focal Loss addresses category and hardness imbalance by extending focal weighting to continuous targets and scaling errors relative to ground-truth magnitude.The method improves generalized focal loss because absolute error alone cannot distinguish samples with different target magnitudes.
- UniMVSNet: UniMVSNet applies Unification and Unified Focal Loss within a coarse-to-fine framework, replacing the traditional depth representation and optimization.The proposed modules are applied to the existing coarse-to-fine pipeline rather than changing its overall procedure.
4. Experiments
UniMVSNet is evaluated on DTU and Tanks and Temples, with comparisons, qualitative results, and ablations assessing performance, generalization, and component effectiveness.
- Results on DTU: UniMVSNet ranks first in DTU Completeness and Overall metrics, while Gipuma ranks first in Accuracy.The model also shows greater robustness in reflective and low-textured challenge regions.
- Datasets and implementation: The experiments evaluate UniMVSNet on DTU and Tanks and Temples, with finetuning on BlendedMVS for the latter benchmark.DTU uses indoor scenes, while Tanks and Temples contains intermediate and advanced realistic scenes.
- Results on DTU: UniMVSNet generates more complete point clouds with finer details in qualitative DTU comparisons.Figure 6 presents full point clouds and enlarged local regions for scan 15.
- Results on Tanks and Temples: UniMVSNet outperforms the previous best model by 2.68 points on the intermediate set and 3.24 points on the advanced set.The model achieves state-of-the-art F-score performance and first place in most scenes.
- Ablation Studies: Ablations show that replacing regression with Unification improves performance and remains more robust when the finer-stage hypothesis range misses ground truth.The study also evaluates adaptive aggregation and finer ground-truth strategies using five input views.
- Ablation Studies: The proposed unification remains effective with only 50% of the training data.The ablation results report that unification performs better and is more concise than other tested strategies.
5. Conclusion
The paper concludes that Unification and Unified Focal Loss improve multi-view stereo without adding memory or computational costs.
- 5. Conclusion: Unification recovers finer 3D scene benefits from direct cost-volume learning, while UFL rebalances samples and handles continuous labels.Both modules are presented as improving multi-view stereo effectiveness.
- 5. Conclusion: Each module is plug-and-play, integrates into existing MVS frameworks, and adds no memory or computational costs.UniMVSNet demonstrates significant performance improvements using the two modules.
A. More Explanation of Unified Focal Loss
The appendix explains scaling functions used in Unified Focal Loss, including an asymmetrical strategy for preserving positive learning signals.
- Scaling function: The scaling factor S_b(x) has range [0.5, 1) when q+ ranges over [0, +∞).This range follows from the sigmoid-like definition given for S_b(x).
- Asymmetrical scaling: An asymmetrical scaling strategy maps S+ to [0, 1) to protect positive learning signals.The appendix gives separate implementation expressions for the positive scaling terms.
- Asymmetrical scaling: The appendix provides explicit sigmoid-based expressions for the positive scaling functions used in UFL.The displayed forms use base-5 sigmoid transformations.
B. Finer DTU Ground-truth
The appendix describes using refined DTU ground truth produced by crossfiltering neighboring viewpoints, while retaining masks that indicate valid points.
- Ground-truth refinement: Each DTU ground-truth refinement is obtained by crossfiltering with neighboring viewpoints.The processed refinement results are adopted from prior work for convenience.
- Validity masks: The training procedure uses the validity mask associated with the refined ground truth.The mask is combined with masks from prior methods and an up-sampled mask.
C. More Ablation Studies on DTU Dataset
The DTU ablations examine Unified Focal Loss settings and the two ways Unification can regress depth proximity or offset. They show that stage-specific rebalancing and proximity-based regression are preferred design choices.
- The scaling factor in UFL: Large scaling factors can dominate Unified Focal Loss training, making convergence difficult and performance extremely poor.The dedicated function controls the scaling-factor range, while asymmetrical scaling preserves positive learning signals.
- The tunable parameter in UFL: Unified Focal Loss performs best when its tunable parameters are configured stage by stage rather than identically across stages.The three stages use 48, 32, and 8 depth hypotheses, with the greatest sample imbalance occurring in stage1.
- Proximity VS. Offset: Proximity-based regression is much more powerful than directly estimating offset in Unification.The authors associate this advantage with proximity magnitude reflecting the quality of the classified optimal depth hypothesis.
- Proximity VS. Offset: Unity can be generated using either proximity or offset, with proximity using the interval above the depth hypothesis and offset using the median-value area.These are the two supervised-signal construction strategies illustrated in Figure C.
D. More Comparisons between Unification, Classification and Regression
Regression and classification offer complementary strengths in learning-based MVS. Regression supports sub-pixel depth estimation but is harder to optimize, whereas classification directly constrains the cost volume but predicts depth discretely.
- Regression: Regression methods can achieve sub-pixel depth estimation and therefore better accuracy.They treat depth as the expectation of discrete depth hypotheses.
- Regression: Regression methods are harder to converge and have greater overfitting risk because they use an indirect learning strategy.They also tend to produce smooth depth at object boundaries.
- Classification: Classification methods cannot generate accurate depth directly because their predictions are discrete.The supplied comparison passage contrasts this limitation with regression’s sub-pixel estimation.
E. More Results on DTU Dataset
Additional DTU results show that UniMVSNet reconstructs point clouds with excellent accuracy and completeness. Figure D provides further qualitative point-cloud examples.
- More Results on DTU Dataset: UniMVSNet’s additional DTU point-cloud reconstructions have excellent accuracy and completeness.These results are presented as additional reconstruction examples in Figure D.
- More Results on DTU Dataset: The additional DTU evidence concerns reconstructed point clouds rather than only depth-map predictions.
- More Results on DTU Dataset: Figure D presents additional qualitative results on the DTU dataset.