Source-linked AI summary
Attention Concatenation Volume for Accurate and Efficient Stereo Matching
Gangwei Xu, Junda Cheng, Peng Guo, Xin Yang
TL;DR
Stereo matching requires informative cost volumes that support high accuracy without excessive aggregation cost. The paper introduces correlation-guided attention concatenation volumes, using multi-level adaptive patch matching to generate reliable similarity cues, and reports improved accuracy with lighter aggregation across stereo networks and benchmarks.
Problem
Stereo matching needs cost-volume representations that preserve useful content and similarity information while supporting both high accuracy and efficient inference.
Method
The paper generates attention weights from correlation clues, uses multi-level adaptive patch matching for reliable similarity measures, and filters concatenation volumes to form ACV.
Results
ACV improves accuracy while reducing aggregation requirements, and ACVNet achieves top-three rankings across KITTI 2012, KITTI 2015, Scene Flow, and ETH3D.
Takeaways & Limitations
ACV is presented as a general cost-volume representation that can be integrated into various 3D CNN stereo models for improved performance.
Abstract
from arXiv · showhide
Stereo matching is a fundamental building block for many vision and robotics applications. An informative and concise cost volume representation is vital for stereo matching of high accuracy and efficiency. In this paper, we present a novel cost volume construction method which generates attention weights from correlation clues to suppress redundant information and enhance matching-related information in the concatenation volume. To generate reliable attention weights, we propose multi-level adaptive patch matching to improve the distinctiveness of the matching cost at different disparities even for textureless regions. The proposed cost volume is named attention concatenation volume (ACV) which can be seamlessly embedded into most stereo matching networks, the resulting networks can use a more lightweight aggregation network and meanwhile achieve higher accuracy, e.g. using only 1/25 parameters of the aggregation network can achieve higher accuracy for GwcNet. Furthermore, we design a highly accurate network (ACVNet) based on our ACV, which achieves state-of-the-art performance on several benchmarks.
1. Introduction
Stereo matching needs cost volumes that preserve matching and content information while supporting both high accuracy and efficient inference. The paper proposes ACV and ACVNet to address this trade-off through correlation-guided filtering and adaptive patch matching.
- Stereo matching establishes dense correspondences for applications including robotics, augmented reality, and autonomous driving, but achieving high accuracy and efficiency together remains challenging.
- Correlation volumes measure similarity efficiently but lose content, whereas concatenation volumes preserve content yet ignore similarity and require extensive 3D aggregation.
- GwcNet combines correlation and concatenation volumes but still requires twenty eight 3D convolutions for cost aggregation.
- ACV uses correlation-derived attention weights to filter concatenation volumes, suppress redundant information, and retain matching-related information.Multi-level adaptive patch matching produces more reliable similarity measures, including for textureless regions.
- Four 3D convolutions with ACV achieve better accuracy than GwcNet using twenty eight 3D convolutions, while PSMNet and GwcNet gain 42% and 39% additional accuracy improvement, respectively.
- ACVNet ranks second on KITTI 2012, KITTI 2015, and Scene Flow, third on ETH3D, and is the fastest among the top 10 KITTI methods.ACVNet is reported as the only method ranking in the top three concurrently across all four benchmarks; ACVNet-Fast outperforms state-of-the-art real-time methods.
2. Related work
Prior stereo methods trade off information richness, memory and computation, or accuracy during cost-volume construction and aggregation. The paper positions ACV as a similarity-regularized concatenation representation that enables lightweight aggregation while preserving informative matching cues.
- Cost volume construction and aggregation: CNN stereo models commonly improve two tightly coupled stages: cost volume construction and cost aggregation, which jointly determine accuracy and efficiency.
- Cost volume construction: Correlation volumes use low memory and computation but encode limited content information, while concatenation volumes preserve content at higher aggregation cost.
- Cost volume construction: Cascade cost volumes reduce construction complexity through coarse-to-fine disparity refinement but can accumulate errors across stages.
- Cost aggregation: Deep 3D CNN aggregation is computationally and memory intensive, while lighter alternatives can sacrifice nontrivial accuracy.
- Proposed direction: ACV uses similarity information from correlation volumes to regularize concatenation volumes so lightweight aggregation can achieve high efficiency and accuracy.The ACVNet architecture includes unary feature extraction, ACV construction, cost aggregation, and disparity prediction.
3. Method
ACV constructs attention weights from multi-level adaptive patch matching and applies them to a concatenation volume, suppressing redundant information before cost aggregation. ACVNet and ACVNet-Fast integrate this representation with lightweight aggregation and disparity prediction modules.
- Attention concatenation volume: ACV construction comprises initial concatenation volume construction, attention-weight generation, and attention filtering.The attention weights are applied to the initial volume to produce the attention concatenation volume.
- Attention concatenation volume: The initial concatenation volume concatenates left and right unary features at each disparity, producing a 2Nc×D/4×H/4×W/4 volume.The feature maps have Nc=32 channels and are extracted at one-quarter image resolution.
- Attention-weight generation: Multi-level adaptive patch matching uses atrous patches with learned weights to obtain more reliable disparity similarities, especially where pixel-to-pixel correlation is unreliable.Three feature levels are grouped without interference, with 320 channels divided into 40 groups.
- Attention-weight generation: Two 3D convolutions and a 3D hourglass regularize the patch-matching volume, while supervised soft-argmin disparity estimates guide attention-weight learning.The resulting attention weights have shape 1×D/4×H/4×W/4.
- ACVNet architecture: ACVNet combines unary feature extraction, ACV construction, cost aggregation, and disparity prediction in an end-to-end stereo network.Its disparity prediction uses probability volumes over disparity and soft argmin.
- ACVNet-Fast: ACVNet-Fast narrows the disparity search using six hypotheses near the predicted disparity and uses a sparse ACV with a lightweight aggregation network.It uses one 3D hourglass and two 3D convolutions for aggregation, balancing accuracy and speed.
4. Experiment
Experiments evaluate ACV through ablations, complexity analysis, integration into multiple stereo models, and benchmark comparisons. ACV improves accuracy while reducing aggregation requirements, and ACVNet performs competitively across Scene Flow, KITTI, and ETH3D.
- Experimental setup: ACV experiments cover ablations, computational complexity, universality across models, cascaded-volume comparisons, and evaluations on Scene Flow, KITTI, and ETH3D.The study also evaluates the real-time ACVNet-Fast variant on Scene Flow and KITTI.
- Ablation study: Multi-level adaptive patch matching substantially improves GwcNet, whereas ordinary fixed-size patch matching yields only a slight improvement over the original model.Ordinary patches use a fixed 3×3 size and equal pixel weights; multi-level adaptive patches use different rates and adaptive weights across feature levels.
- Ablation study: 7.9% and 8.7% improvements in D1 and EPE result when an hourglass aggregates the multi-level patch matching volume before attention filtering.The comparison is between GwcNet-mp-att-hg and GwcNet-mp-att on Scene Flow.
- Computational complexity analysis: 42.8% and 39.5% improvements in D1 and EPE are obtained by Gwc-acv-3 over GwcNet, while Gwc-acv-0 still outperforms GwcNet without an hourglass aggregation network.Reducing the number of hourglass networks slightly increases prediction errors, but ACV substantially reduces aggregation requirements.
- Universality and superiority of ACV: 39.5%, 42.2%, and 14.4% EPE reductions are achieved for GwcNet, PSMNet, and CFNet after replacing their original volumes with ACV.These results support integrating ACV into multiple existing stereo architectures.
5. Conclusion
The paper introduces ACV, which uses similarity-based attention to filter concatenation volumes, and ACVNet, which performs strongly across four public stereo benchmarks.
- ACV generates attention weights from similarity measures to filter the concatenation volume.This cost volume is designed to preserve matching-related information while suppressing redundancy.
- Multi-level adaptive patch matching produces accurate similarity measures even in textureless regions.
- ACVNet shows excellent performance on KITTI 2012, KITTI 2015, Scene Flow, and ETH3D.