Source-linked AI summary
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
Xiaodong Gu, Zhiwen Fan, Zuozhuo Dai, Siyu Zhu, Feitong Tan, Ping Tan
TL;DR
Existing 3D cost-volume methods face cubic time and memory growth when producing high-resolution depth or disparity. This paper introduces a feature-pyramid cascade that progressively narrows hypotheses and increases resolution, achieving strong MVS and stereo results with lower resource use.
Problem
3D cost-volume methods are limited for high-resolution outputs because cost-volume resolution causes memory and run-time costs to grow cubically.
Method
The method builds progressively finer cost volumes from a feature pyramid and adaptively narrows each stage's depth or disparity hypotheses using the previous estimate.
Results
The method achieves the best reported DTU performance with MVSNet, state-of-the-art learning-based performance on Tanks and Temples, and reduces GwcNet EPE and GPU memory by about 15.2% and 36.9%.
Takeaways & Limitations
Cascade cost volumes provide a memory- and computationally efficient complement to existing 3D cost-volume approaches for high-resolution MVS and stereo matching.
Takeaways & Limitations
Erroneous early disparity estimates can produce biased results when the next stage's hypothesis range excludes the ground-truth value; current ranges are identical for every pixel.
Abstract
from arXiv · showhide
The deep multi-view stereo (MVS) and stereo matching approaches generally construct 3D cost volumes to regularize and regress the output depth or disparity. These methods are limited when high-resolution outputs are needed since the memory and time costs grow cubically as the volume resolution increases. In this paper, we propose a both memory and time efficient cost volume formulation that is complementary to existing multi-view stereo and stereo matching approaches based on 3D cost volumes. First, the proposed cost volume is built upon a standard feature pyramid encoding geometry and context at gradually finer scales. Then, we can narrow the depth (or disparity) range of each stage by the depth (or disparity) map from the previous stage. With gradually higher cost volume resolution and adaptive adjustment of depth (or disparity) intervals, the output is recovered in a coarser to fine manner. We apply the cascade cost volume to the representative MVS-Net, and obtain a 23.1% improvement on DTU benchmark (1st place), with 50.6% and 74.2% reduction in GPU memory and run-time. It is also the state-of-the-art learning-based method on Tanks and Temples benchmark. The statistics of accuracy, run-time and GPU memory on other representative stereo CNNs also validate the effectiveness of our proposed method.
1. Introduction
Existing 3D cost-volume methods capture geometry well but are constrained by the time and GPU memory demands of high-resolution processing. The paper introduces a coarse-to-fine cascade that adaptively narrows depth or disparity ranges and improves benchmark performance across MVS and stereo matching.
- Motivation: 3D cost volumes capture geometry structures and perform photometric matching in 3D space, but their computational demands limit high-resolution outputs.Existing methods commonly downsample feature maps and then upsample or refine the resulting predictions.
- Method: The proposed method decomposes cost-volume construction into a coarse-to-fine cascade based on a multi-scale feature pyramid.Early stages use larger-scale semantic features and sparse hypotheses, while later stages use finer features.
- Method: Earlier depth or disparity estimates adaptively narrow the hypothesis range for subsequent cost volumes, allocating finer computation where needed.This adaptive sampling and feature-resolution adjustment targets both computation and memory efficiency.
- Results: The cascade achieves the best reported DTU performance with MVSNet and state-of-the-art learning-based performance on Tanks and Temples.For stereo matching, it reduces GwcNet's EPE and GPU memory consumption by about 15.2% and 36.9%, respectively.
2. Related Work
Prior work uses cost-volume, aggregation, refinement, and multi-view reconstruction strategies to improve stereo and MVS accuracy, while high-resolution methods reduce memory through alternative representations or refinement. The proposed cascade cost volume is presented as complementary to these existing approaches.
- Stereo Matching: Traditional stereo matching methods calculate, aggregate, select, and refine matching costs using local or global optimization strategies.Global approaches formulate an energy function, while local approaches typically aggregate neighboring costs and use winner-take-all selection.
- Stereo Matching: Deep stereo methods progressed from learned patch similarity to 3D cost volumes, spatial pooling, hourglass regularization, aggregation layers, and coarse-to-fine search pruning.These methods progressively improve learned cost computation, cost-volume regularization, and disparity search efficiency.
- High-Resolution Output: The cascade cost volume can be combined with existing 3D cost-volume methods to improve disparity accuracy and GPU memory efficiency.It is described as complementary rather than a replacement for those methods.
- Multi-View Stereo: Learning-based MVS methods build 3D cost volumes from warped multi-view features and apply 3D CNNs for cost regularization and depth regression.Their large GPU-memory requirements motivate alternative high-resolution formulations.
- High-Resolution Output: Point MVSNet reduces memory by combining a small coarse cost volume with point-based iterative refinement for full-resolution depth.The paper contrasts this strategy with standard MVSNet combined with its cascade cost volume.
3. Methodology
The cascade cost volume extends 3D cost-volume methods with multi-stage, coarse-to-fine estimation using progressively finer features, narrower hypothesis ranges, and finer plane intervals. It is designed to improve high-resolution reconstruction while controlling computational cost.
- Cascade architecture: The cascade formulation applies existing 3D cost-volume methods in a coarse-to-fine architecture using feature pyramids and progressively higher spatial resolutions.For MVSNet, three cost volumes use feature-pyramid resolutions of 1/16, 1/4, and 1 of the input image size.
- Cost volume formulation: 3D cost volumes warp 2D features onto hypothesized depth or disparity planes, then fuse feature volumes into a representation for matching and regression.Multi-view stereo uses homography-based warping, while rectified stereo matching uses x-axis translation.
- Hypothesis range: Each later stage narrows its depth or disparity hypothesis range around the previous stage’s prediction, with R_k+1 = R_k · w_k and w_k < 1.The first stage covers the full input-scene range; subsequent stages reduce the range while preserving the output range across stages.
- Hypothesis plane interval: Hypothesis plane intervals become progressively finer, following I_k+1 = I_k · p_k with p_k < 1, so early stages estimate coarsely and later stages recover detail.The initial interval is larger than in a single cost volume, while later stages use finer intervals.
- Hypothesis planes: At stage k, the number of planes is D_k = R_k/I_k; reducing ranges stage by stage can lower total plane count while maintaining coverage of the output range.For fixed spatial resolution, more planes can improve accuracy but increase GPU memory and run-time.
- Spatial resolution and supervision: Cost-volume spatial resolution doubles at every stage, with N = 3 stages for multi-view stereo and N = 2 for stereo matching.The cascade produces N − 1 intermediate outputs and one final prediction, with supervision applied to all outputs using weighted stage losses.
4. Experiments
Experiments evaluate the cascade cost volume on multi-view stereo and stereo matching benchmarks, showing improved accuracy alongside reduced memory and runtime. Ablations examine stage count, resolution, feature pyramids, and parameter sharing.
- Multi-view stereo: MVSNet+Ours ranks first on DTU, improving overall quality by 35.6% while reducing GPU memory by 50.6% and runtime by 59.3%.It produces more complete point clouds with finer details and achieves state-of-the-art learning-based performance on Tanks and Temples.
- Stereo matching: GwcNet+Ours rises from 29th to 17th on the KITTI 2015 test set, while PSMNet+Ours ranks 37th on Middlebury’s avgerr metric.The KITTI comparison is dated November 5, 2019, and the Middlebury ranking February 7, 2020.
- Ablation study: Increasing cascade stages improves overall quality before stabilization, with the three-stage configuration used for MVSNet+Ours.The implementation uses 48, 32, and 8 depth hypotheses across three stages.
- Ablation study: Higher spatial resolution improves quality but increases GPU memory and runtime, with MVSNet+Ours scoring 0.355 versus 0.453 for MVSNet-Cas3.The corresponding memory and runtime are 5345 versus 2373 MB and 0.492 versus 0.322 seconds.
- Ablation study: Feature pyramids reduce memory and runtime while improving quality from 0.379 to 0.355, and separate stage parameters outperform shared parameters.Feature-pyramid memory and runtime are 5345 MB and 0.492 seconds, compared with 6227 MB and 0.676 seconds.
5. Conclusion
The paper presents a GPU-memory- and computation-efficient cascade cost volume for high-resolution multi-view stereo and stereo matching. It narrows each stage’s depth or disparity range using previous predictions and uses higher-resolution volumes for finer outputs.
- The method decomposes one cost volume into multiple stages that progressively narrow depth or disparity hypotheses using preceding maps.It is designed as a complement to existing 3D cost-volume-based methods.
- Higher-spatial-resolution cost volumes generate outputs with finer details while the cascade formulation reduces the total hypothesis range.
6. Appendix
The appendix analyzes how cascade cost-volume settings affect hypothesis ranges, accuracy, memory, runtime, and qualitative reconstruction results across multi-view stereo and stereo matching.
- Hypothesis Range: 90%: the depth range narrows from approximately 500mm at stage one to about 50mm at stages two and three.This narrowing enables substantially smaller hypothesis ranges in later stages.
- Memory and Runtime: GPU memory decreases 50.6% and runtime becomes about 2 times faster for MVSNet+Ours than MVSNet.MVSNet+Ours uses 48 first-stage hypothesis planes versus MVSNet’s 192, while later stages increase spatial resolution.
- Cascade Stage Number: Accuracy increases as the number of cascade stages increases in the GwcNet experiments on Scene Flow.The experiment varies hypothesis-plane counts and depth intervals while comparing GwcNet with cascade-cost-volume variants.
- Spatial Resolution: 0.972 to 0.619: increasing stereo cost-volume resolution across three stages improves end-point-error.The corresponding GPU-memory increase is reported alongside this accuracy improvement.
- Limitations: Cascade cost volumes can fail when an erroneous earlier disparity places the ground-truth value outside the next stage’s hypothesis range.The paper describes this as a biased result and notes that the hypothesis range is currently identical for every pixel.