Source-linked AI summary
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching
Zhelun Shen, Yuchao Dai, Zhibo Rao
TL;DR
Deep stereo models often generalize poorly across datasets because of domain shifts and unbalanced disparity distributions. CFNet combines fused and cascade cost volumes with uncertainty-guided search refinement, and it reports state-of-the-art overall performance across three datasets plus first place in Robust Vision Challenge 2020.
Problem
Deep stereo matching methods are often limited to specific datasets because large domain shifts and unbalanced disparity distributions hinder generalization without adaptation.
Method
CFNet fuses multiple low-resolution dense cost volumes for robust initial estimation and uses cascade cost volumes with variance-based uncertainty to adaptively narrow later disparity searches.
Results
CFNet achieves state-of-the-art overall performance across KITTI2015, Middlebury, and ETH3D with fixed parameters and hyperparameters, and ranks first in Robust Vision Challenge 2020.
Takeaways & Limitations
CFNet performs well across multiple real-world datasets without adaptation and is reported to do so with high efficiency.
Abstract
from arXiv · showhide
Recently, the ever-increasing capacity of large-scale annotated datasets has led to profound progress in stereo matching. However, most of these successes are limited to a specific dataset and cannot generalize well to other datasets. The main difficulties lie in the large domain differences and unbalanced disparity distribution across a variety of datasets, which greatly limit the real-world applicability of current deep stereo matching models. In this paper, we propose CFNet, a Cascade and Fused cost volume based network to improve the robustness of the stereo matching network. First, we propose a fused cost volume representation to deal with the large domain difference. By fusing multiple low-resolution dense cost volumes to enlarge the receptive field, we can extract robust structural representations for initial disparity estimation. Second, we propose a cascade cost volume representation to alleviate the unbalanced disparity distribution. Specifically, we employ a variance-based uncertainty estimation to adaptively adjust the next stage disparity search space, in this way driving the network progressively prune out the space of unlikely correspondences. By iteratively narrowing down the disparity search space and improving the cost volume resolution, the disparity estimation is gradually refined in a coarse-to-fine manner. When trained on the same training images and evaluated on KITTI, ETH3D, and Middlebury datasets with the fixed model parameters and hyperparameters, our proposed method achieves the state-of-the-art overall performance and obtains the 1st place on the stereo task of Robust Vision Challenge 2020. The code will be available at https://github.com/gallenszl/CFNet.
1. Introduction
Stereo matching models often struggle to generalize across datasets because domain shifts and disparity-distribution differences require adaptation. CFNet addresses these challenges with fused and cascade cost volumes, achieving strong fixed-parameter performance across three datasets.
- Stereo matching supports applications including autonomous driving, robot navigation, and SLAM.
- Current state-of-the-art methods often specialize in one dataset and require substantial adaptation to perform comparably elsewhere.KITTI emphasizes urban driving, whereas Middlebury focuses on indoor high-resolution scenes.
- Domain differences such as indoor versus outdoor scenes, color versus gray imagery, and real versus synthetic data can distort learned features.
- Middlebury’s half-resolution disparity range exceeds ETH3D’s full-resolution range by more than six times, at 400 versus 64.
- CFNet fuses multiple low-resolution dense cost volumes to reduce domain differences and uses cascade cost volumes with variance-based uncertainty to balance disparity distributions.The fused representation enlarges receptive fields, while uncertainty adaptively adjusts later disparity search spaces.
- Across KITTI2015, Middlebury, and ETH3D, CFNet achieves state-of-the-art overall performance using fixed model parameters and hyperparameters without adaptation.The method also obtains first place in the stereo task of Robust Vision Challenge 2020.
2. Related Work
Deep stereo methods rely on cost volumes whose representation determines the trade-off between information retention and computational cost. Cascade representations progressively refine disparity estimates, while CFNet adds uncertainty-aware adaptation for cross-dataset robustness.
- 2.1. Cost Volume based Deep Stereo Matching: Cost-volume stereo methods use either efficient single-feature 3D volumes or more informative multi-feature 4D volumes with higher memory and computational costs.Concatenation and group-wise correlation commonly generate multi-feature 4D volumes.
- 2.1. Cost Volume based Deep Stereo Matching: Most cost-volume networks regularize 4D volumes, apply softmax over disparity, and use soft argmin to obtain the final disparity.Soft argmin can be affected by multimodal disparity probability distributions.
- 2.1. Cost Volume based Deep Stereo Matching: Multi-scale and cascade cost volumes first estimate disparity coarsely, then narrow the search space and progressively refine the result.
- 2.1. Cost Volume based Deep Stereo Matching: Prior cascade approaches use predefined parameters for the next-stage search range, whereas CFNet learns parameters to adapt better across datasets.
- 2.1. Cost Volume based Deep Stereo Matching: Robust stereo matching includes cross-domain generalization for unseen scenes and joint generalization across multiple datasets.
3. Our Approach
CFNet combines pyramid feature extraction, fused low-resolution cost volumes, and cascade cost volumes to improve robust stereo matching. It fuses dense volumes for initial disparity estimation, then uses variance-based uncertainty to adaptively narrow the search space and refine disparity coarse-to-fine.
- Overview of CFNet: CFNet comprises pyramid feature extraction, fused cost volume, and cascade cost volume components.
- Fused Cost Volume: Multiple low-resolution dense cost volumes are fused to enlarge the receptive field and generate an initial disparity map.The fused representation targets domain shifts by capturing global and structural representations.
- Fused Cost Volume: Feature concatenation and group-wise correlation are combined to construct multi-feature cost volumes across image scales.The combination volume uses extracted left and right features indexed by disparity.
- Fused Cost Volume: Dense low-resolution volumes sample the whole disparity range efficiently, while later stages use uncertainty to narrow the search space.The fused volumes have dense disparity sampling with unit hypothesis-plane intervals.
- Cascade Cost Volume: Variance-based uncertainty estimates pixel-level confidence from disparity probability distributions and expands the next search range when uncertainty is higher.Multimodal distributions are associated with higher prediction error, especially in ill-posed, textureless, and occluded regions.
- Cascade Cost Volume: The cascade progressively narrows disparity ranges and increases cost-volume resolution to refine disparity estimates coarse-to-fine.Learned normalization factors determine adaptive search boundaries before uniform sampling creates the next stage’s hypothesis planes.
4. Experiments
Experiments evaluate CFNet under fixed settings across multiple real-world datasets, including ablations, joint and cross-domain generalization, and KITTI-specific finetuning. The results support robust performance across datasets, adaptive uncertainty estimation, and efficient KITTI accuracy.
- Experimental Setup: CFNet is trained on common settings and evaluated on KITTI2015, Middlebury, and ETH3D, with ablations using validation splits from each dataset.The network uses three stages, a fixed disparity search range of 256, and end-to-end Adam training; Table 1 evaluates components on the three validation sets.
- Ablation Study: Uncertainty estimation achieves the best cascade-search performance with tiny added complexity and avoids dataset-specific hyperparameter adjustment.On the KITTI 2015 validation set, the D1 all error rate decreases from 1.55% to 1.09% when using the uncertainty criterion U >= 2.5.
- Ablation Study: 29.68% lower D1 all error rate reduces the KITTI 2015 validation result from 1.55% to 1.09% with uncertainty estimation.The passage attributes comparable behavior to the other two datasets and motivates pixel-level confidence estimation for disparity prediction.
- Ablation Study: Later cascade stages benefit from larger loss weights, while the proposed finetuning strategy alleviates domination of small datasets by larger ones.Simply extending iterations does not improve accuracy on small datasets, whereas the proposed strategy does.
- Robustness Evaluation: Without adaptation, CFNet performs well across all three real datasets and achieves the best overall joint-generalization performance.It ranks 2nd on KITTI 2015, 1st on ETH3D, and 2nd on Middlebury; the evaluation defines robustness as joint generalization.
- Robustness Evaluation: Training only on Scene Flow lets CFNet outperform domain-specific methods on all four evaluated datasets and surpass DSMnet on three datasets.This cross-domain evaluation tests synthetic-to-real generalization without the domain-specific finetuning used in the KITTI benchmark comparison.
- Results on KITTI Benchmark: 1.88% D1 all error rate is achieved on the KITTI benchmark, with 3 times faster speed than the base casstereo model.CFNet has similar performance to GANet-deep and ACFNet and outperforms all published methods faster than 200ms on both KITTI datasets.
5. Conclusion
The paper concludes that cascade and fused cost volumes address domain shifts and differing disparity distributions in robust stereo matching. CFNet uses uncertainty-guided search-space narrowing and achieves efficient performance across datasets, with future work extending the representation to semi- or self-supervised learning.
- Conclusion: CFNet combines fused cost volumes for domain shifts with cascade cost volumes for differing disparity distributions.The fused representation supports initial disparity estimation, while the cascade representation progressively narrows each pixel’s disparity search space using variance-based uncertainty estimation.
- Conclusion: Experimental results show strong performance across varied datasets with high efficiency.The conclusion states this outcome without restricting it to a single benchmark.
- Conclusion: Future work will extend the cost-volume representation to semi-supervised or self-supervised settings.This direction is explicitly proposed for subsequent research.