Source-linked AI summary
Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching
Youmin Zhang, Yimin Chen, Xiao Bai, Suihanjin Yu, Kun Yu, Zhiwei Li, Kuiyuan Yang
TL;DR
Deep stereo systems often learn underconstrained cost volumes indirectly through disparity regression, allowing multiple distributions to yield the same disparity. AcfNet directly supervises cost volumes with adaptive unimodal distributions whose variances model pixelwise matching uncertainty, achieving leading results across three benchmarks.
Problem
Indirect disparity-regression supervision leaves cost volumes underconstrained, although only distributions peaked at true disparities are considered reasonable.
Method
AcfNet directly supervises cost volumes with true-disparity unimodal distributions and adaptively estimates per-pixel confidence and variance to model matching uncertainty.
Results
AcfNet was evaluated on Scene Flow, KITTI 2012, and KITTI 2015, ranking 1st on KITTI 2012 and 4th on KITTI 2015 as recorded on 2019.8.20.
Takeaways & Limitations
Adaptive unimodal cost-volume supervision improves testing performance on the same dataset and yields superior cross-dataset performance.
Abstract
from arXiv · showhide
State-of-the-art deep learning based stereo matching approaches treat disparity estimation as a regression problem, where loss function is directly defined on true disparities and their estimated ones. However, disparity is just a byproduct of a matching process modeled by cost volume, while indirectly learning cost volume driven by disparity regression is prone to overfitting since the cost volume is under constrained. In this paper, we propose to directly add constraints to the cost volume by filtering cost volume with unimodal distribution peaked at true disparities. In addition, variances of the unimodal distributions for each pixel are estimated to explicitly model matching uncertainty under different contexts. The proposed architecture achieves state-of-the-art performance on Scene Flow and two KITTI stereo benchmarks. In particular, our method ranked the $1^{st}$ place of KITTI 2012 evaluation and the $4^{th}$ place of KITTI 2015 evaluation (recorded on 2019.8.20). The codes of AcfNet are available at: https://github.com/DeepMotionAIResearch/DenseMatchingBenchmark.
Introduction
Stereo matching recovers 3D structure by estimating pixel disparities, but conventional deep models indirectly supervise cost volumes through disparity regression. AcfNet directly constrains cost distributions with adaptive unimodal targets and achieves strong benchmark rankings.
- Stereo matching estimates each pixel’s horizontal disparity to recover 3D structure from rectified image pairs.
- Cost volume computation is a crucial stereo-matching step, representing matching costs across image height, width, and disparity.
- Soft argmin enables differentiable sub-pixel disparity estimation by weighting disparity indices according to their costs.
- Indirect cost-volume supervision is underconstrained because infinitely many cost distributions can produce the same disparity, enabling unreasonable distributions and overfitting.
- AcfNet directly supervises cost volumes with unimodal distributions peaked at true disparities and estimates per-pixel confidence to control their sharpness.
- AcfNet was evaluated on Scene Flow, KITTI 2012, and KITTI 2015, ranking 1st on KITTI 2012 and 4th on KITTI 2015 as recorded on 2019.8.20.
Related Work
Deep stereo methods progressively learned features, cost functions, and aggregation, but typically left cost volumes without direct supervision. AcfNet adds dense, adaptive unimodal supervision using only stereo images.
- DispNetC introduced an end-to-end trainable stereo network that builds a correlation cost volume and regresses disparity with convolutional refinement.
- Later methods improved stereo matching through refinement subnetworks, edge and semantic information, and group-wise correlation with multiple cost volumes.
- GC-Net learned cost functions with 3D convolutions over concatenated feature volumes and estimated disparity using soft argmin.
- AcfNet addresses the lack of direct cost-volume supervision with dense unimodal distributions whose sharpness adapts to matching confidence, using images alone at training and testing.
AcfNet
AcfNet directly supervises stereo cost volumes with unimodal distributions peaked at true disparities, while adapting their sharpness to per-pixel matching confidence. It combines cost-volume supervision with disparity regression and confidence regularization to address underdetermined cost learning.
- AcfNet applies an adaptive unimodal cost-volume filtering module and adds a loss that directly supervises the cost volume toward the desired property.The method uses PSMNet as its basic cost-volume network.
- Indirect disparity supervision leaves infinitely many cost distributions able to produce similar disparities, making improperly learned cost volumes prone to overfitting.
- The target cost distribution is unimodal and peaked at the true disparity, with σ controlling peak sharpness around that disparity.The true match has the lowest cost, while costs increase with distance from the true matched pixel.
- A confidence network predicts σp per pixel from local matching states in the estimated cost volume, producing sharper or flatter target distributions for different contexts.Texture-less and occluded pixels are likely to have large σp because they have ambiguous or absent correct matches.
- Stereo focal loss addresses disparity-sample imbalance by emphasizing positive disparities and suppressing easy negative disparities.For α = 0 it reduces to cross entropy; α > 0 weights positive disparities according to Pp(d).
- The final objective combines stereo focal loss for cost-volume supervision, smooth L1 disparity regression, and confidence regularization.λregression and λconfidence control the trade-offs among the three components.
Experiments and Analysis
AcfNet is evaluated on Scene Flow and KITTI benchmarks through ablations, variance analysis, qualitative inspection, and comparisons with established stereo-matching systems. Results support adaptive variance estimation, direct cost-volume supervision, and improved performance in challenging regions.
- Evaluation settings: Scene Flow provides 35,454 training pairs and 4,370 testing pairs, while KITTI 2012 and 2015 provide small real-world driving datasets with server-held test ground truth.Scene Flow is primarily used for ablation studies; KITTI evaluation uses sparse training ground truth and held-out test benchmarks.
- Evaluation settings: EPE is used for hyperparameter analysis because most disparities have sub-pixel errors that 3PE does not reveal.The Scene Flow ablations set α to 5.0 in stereo focal loss and use EPE to measure performance variation.
- Ablation studies: σ = 1.2 performs best when variance is fixed, and stereo focal loss significantly outperforms cross entropy under this setting.The fixed-variance study uses s = 0 and sets the adaptive-study lower bound ϵ to 1.0.
- Ablation studies: s = 1 performs best, with stable performance for s from 0.5 to 3.0; most pixels favor small variances while a long tail requires larger variances.For s = 1, σ lies in [1.0, 2.0], indicating predominantly sharp distributions with context-dependent flattening.
- Variance analysis: Removing 6.9% of pixels with the highest predicted variances halves average EPE on the Scene Flow test set.The sparsification analysis compares predicted-variance ranking with random removal and an EPE-ranked oracle.
- Variance analysis: Adaptive AcfNet improves over uniform AcfNet and PSMNet, especially in occluded regions, while high variances concentrate in difficult regions and help avoid overfitting.Qualitative analyses identify occlusions, repeated patterns, thin structures, textureless pixels, and image boundaries as challenging contexts.
Conclusions
AcfNet addresses under-constrained cost volumes by supervising them with unimodal distributions centered on true disparities and adapting per-pixel variances. It reports improved same-dataset and cross-dataset testing performance.
- AcfNet supervises cost volumes with unimodal distributions peaked at true disparities.
- Per-pixel variances are adaptively estimated to modulate learning according to pixel informativeness.
- AcfNet shows better testing performance on the same dataset and superior performance on cross-dataset evaluation.
A. Effectiveness on different backbones
The adaptive unimodal cost volume filtering scheme improves performance across the evaluated GC-Net and stack-hourglass PSMNet backbones.
- AcfNet delivers better performance across GC-Net and stack-hourglass PSMNet backbones.The evaluation equips different stereo matching models with the adaptive unimodal cost volume filtering scheme.
B. Architecture details
The experimental AcfNet architecture is based on PSMNet's stacked hourglass design and adds a confidence estimation network to each produced cost volume.
- AcfNet is based on PSMNet with a stacked hourglass architecture.The architecture produces three cost volumes.
- A Confidence Estimation network is added to each cost volume.