Source-linked AI summary
Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume
Qingshan Xu, Wenbing Tao
TL;DR
Learned MVS remains constrained by memory-consuming cost volumes and depth-inference strategies that trade robustness against sub-pixel estimation. CIDER uses average group-wise correlation, cascade 3D U-Net filtering, and inverse depth regression, and reports promising reconstruction results across DTU and Tanks and Temples while targeting large-scale applicability.
Problem
Learned MVS faces scalability and depth-inference challenges because cost volumes consume substantial memory and existing depth formulations have limitations.
Method
CIDER builds a lightweight average group-wise correlation cost volume, applies cascade 3D U-Net filtering, and performs inverse depth regression.
Results
CIDER achieves promising reconstruction results on DTU and Tanks and Temples datasets.
Takeaways & Limitations
The proposed design targets scalable MVS on high-resolution and large-scale scenes while retaining sub-pixel depth estimation.
Abstract
from arXiv · showhide
Deep learning has shown to be effective for depth inference in multi-view stereo (MVS). However, the scalability and accuracy still remain an open problem in this domain. This can be attributed to the memory-consuming cost volume representation and inappropriate depth inference. Inspired by the group-wise correlation in stereo matching, we propose an average group-wise correlation similarity measure to construct a lightweight cost volume. This can not only reduce the memory consumption but also reduce the computational burden in the cost volume filtering. Based on our effective cost volume representation, we propose a cascade 3D U-Net module to regularize the cost volume to further boost the performance. Unlike the previous methods that treat multi-view depth inference as a depth regression problem or an inverse depth classification problem, we recast multi-view depth inference as an inverse depth regression task. This allows our network to achieve sub-pixel estimation and be applicable to large-scale scenes. Through extensive experiments on DTU dataset and Tanks and Temples dataset, we show that our proposed network with Correlation cost volume and Inverse DEpth Regression (CIDER), achieves state-of-the-art results, demonstrating its superior performance on scalability and accuracy.
Introduction
CIDER addresses MVS scalability and depth-inference robustness by combining a lightweight correlation cost volume with inverse depth regression. The method targets memory limits in high-resolution scenes while preserving sub-pixel estimation and improving applicability to large-scale scenarios.
- Memory-heavy cost volumes limit learned MVS methods in large-scale and high-resolution scenarios.
- Average group-wise correlation constructs a lightweight cost volume by aggregating compact similarities between warped source and reference features.
- Inverse depth regression samples hypotheses in inverse-depth space, regresses a sub-pixel ordinal, and converts it into the final depth value.
- CIDER reports promising reconstruction results on both DTU and Tanks and Temples datasets.
- The method combines correlation cost volume, cascade 3D U-Net filtering, and inverse depth regression to improve scalability, accuracy, and large-scale applicability.
Related Work
Prior learning-based MVS methods use learned similarity, geometric warping, cost volumes, and recurrent or variational refinement. CIDER extends this line with average group-wise correlation and a multi-stage architecture for cost-volume filtering and regression.
- Learning-based stereo and MVS methods use neural architectures to learn similarities, encode camera geometry, or fuse multi-view representations.
- CIDER extracts shared features, differentially warps source features, constructs an average group-wise correlation cost volume, and filters it before regression.
- MVSNet uses differentiable homography warping and a reference-camera-frustum cost volume, while R-MVSNet sequentially regularizes it with a GRU.
Method
CIDER builds a memory-efficient multi-view cost volume with average group-wise correlation, regularizes it using a cascade 3D U-Net, and performs inverse-depth regression for sub-pixel estimation.
- Correlation Cost Volume Construction: CIDER warps source-image features into the reference-camera coordinates at sampled depth hypotheses using camera parameters and plane-sweep geometry.The network receives a reference image, source images, and camera parameters, then constructs the cost volume from warped source features.
- Correlation Cost Volume Construction: Average group-wise correlation constructs a lightweight cost volume that reduces memory consumption and eases subsequent cost-volume filtering.Features are divided into groups, correlated after differential warping, and averaged across source images.
- Cost Volume Filtering: A cascade 3D U-Net filters the raw 8-channel cost volume to incorporate context and handle ambiguous depth regions.Two cascaded 3D U-Nets use repeated top-down and bottom-up processing, enabled by the lightweight representation.
- Inverse Depth Regression: Inverse-depth sampling makes projected points along neighboring epipolar lines more uniformly distributed than direct depth sampling in unrectified multi-view settings.The sampled hypotheses generate cost-volume slices for subsequent filtering.
- Inverse Depth Regression: The network regresses a sub-pixel ordinal from softmax probabilities over sampled depths and converts it into a continuous depth map.Three output branches produce filtered cost volumes, while training uses ground-truth depth with weighted mean absolute differences.
Experiments
Experiments evaluate CIDER on DTU and Tanks and Temples, using ablations to test its cost volume, regression, and filtering choices before benchmark comparisons.
- Ablation Studies: Inverse depth regression substantially improves generalization over the Base and AGC models on Tanks and Temples training datasets.The AGC-IDR model also estimates depth maps more accurately than depth-regression variants.
- Ablation Studies: Average group-wise correlation reduces total memory consumption by nearly half while retaining reconstruction results almost equivalent to the Base model.The cost volume shrinks from 32×H^4×W^4×D to 8×H^4×W^4×D.
- Ablation Studies: Cascade 3D U-Net filtering suppresses noise in ambiguous regions and improves 3D reconstruction over AGC-IDR with only a slight memory increase.Figure 3 provides qualitative depth-map comparisons, while Table 2 reports the reconstruction results.
- Ablation Studies: 49.60% F1 is achieved when depth sampling increases from 192 to 256, while memory consumption remains acceptable.The increased sampling is performed over the same depth range.
- Benchmarking: CIDER outperforms MVSNet on Tanks and Temples and remains applicable to Advanced scenes, where MVSNet cannot be applied.The comparison uses models trained on DTU without fine-tuning.
- Benchmarking: On DTU, CIDER obtains the best mean completeness and overall score among compared methods without post-processing.The quantitative comparison is reported in Table 3 and qualitative reconstructions are shown in Figure 4.
Conclusion
CIDER combines a lightweight correlation cost volume, inverse depth regression, and cascade 3D U-Net filtering for scalable and accurate multi-view stereo. Experiments on different datasets demonstrate its applicability to unseen large-scale scenarios and ambiguous areas.
- CIDER combines a lightweight correlation cost volume, inverse depth regression, and cascade 3D U-Net filtering for multi-view stereo.The correlation volume reduces memory consumption, inverse depth regression improves generation on unseen large-scale scenarios, and cascade filtering targets ambiguous areas.