PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate and Efficient Stereo Matching via Attention Concatenation Volume

Gangwei Xu, Yun Wang, Junda Cheng, Jinhui Tang, Xin Yang

arXiv:2209.12699v3cs.CV

TL;DR

Stereo matching needs cost-volume representations that support both high accuracy and efficiency. The paper proposes ACV and Fast-ACV, which use correlation clues to filter concatenation volumes, and reports strong benchmark results for ACVNet and Fast-ACVNet.

  • Problem

    Stereo matching remains challenging with repetitive structures, textureless or transparent objects, and occlusions, while achieving both high accuracy and efficiency is important for practical applications.

    Source: §1 · p. 1

  • Method

    ACV uses correlation-based attention to filter concatenation volumes, while Fast-ACV adds volume attention propagation and high-likelihood disparity sampling from low-resolution correlation clues.

    Source: §1 · p. 2

  • Results

    ACVNet ranks 2nd on KITTI 2015 and Scene Flow and 3rd on KITTI 2012 and ETH3D, while Fast-ACVNet outperforms almost all state-of-the-art real-time methods on Scene Flow, KITTI 2012, and KITTI 2015.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ACV and Fast-ACV are presented as general cost-volume representations that can be integrated into stereo models to reduce computational cost and improve accuracy.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

Stereo matching is a fundamental building block for many vision and robotics applications. An informative and concise cost volume representation is vital for stereo matching of high accuracy and efficiency. In this paper, we present a novel cost volume construction method, named attention concatenation volume (ACV), which generates attention weights from correlation clues to suppress redundant information and enhance matching-related information in the concatenation volume. The ACV can be seamlessly embedded into most stereo matching networks, the resulting networks can use a more lightweight aggregation network and meanwhile achieve higher accuracy. We further design a fast version of ACV to enable real-time performance, named Fast-ACV, which generates high likelihood disparity hypotheses and the corresponding attention weights from low-resolution correlation clues to significantly reduce computational and memory cost and meanwhile maintain a satisfactory accuracy. The core idea of our Fast-ACV is volume attention propagation (VAP) which can automatically select accurate correlation values from an upsampled correlation volume and propagate these accurate values to the surroundings pixels with ambiguous correlation clues. Furthermore, we design a highly accurate network ACVNet and a real-time network Fast-ACVNet based on our ACV and Fast-ACV respectively, which achieve the state-of-the-art performance on several benchmarks (i.e., our ACVNet ranks the 2nd on KITTI 2015 and Scene Flow, and the 3rd on KITTI 2012 and ETH3D among all the published methods; our Fast-ACVNet outperforms almost all state-of-the-art real-time methods on Scene Flow, KITTI 2012 and 2015 and meanwhile has better generalization ability)

1 INTRODUCTION

The introduction frames stereo matching as a balance between accuracy and efficiency, with cost-volume representations central to both. It presents ACV and Fast-ACV as ways to filter concatenation-volume information and reports strong benchmark performance for their networks.

1 INTRODUCTION

  • Stereo matching estimates disparity from rectified image pairs, but repetitive structures, textureless or transparent objects, occlusions, and the accuracy-efficiency trade-off remain challenges.Prior approaches include correlation volumes that lose content information, concatenation volumes that require extensive aggregation, and efficiency-oriented low-resolution methods that typically reduce accuracy.Source: §1 · pp. 1–2
  • The paper targets a cost volume that is both efficient and effective, using correlation’s rough geometric structure to filter redundant information from concatenation volumes.It presents ACV as a general representation that can be integrated into existing stereo models.Source: §1 · p. 2
  • Replacing GwcNet’s combined volume with ACV lets four 3D cost-aggregation convolutions achieve better accuracy than its twenty-eight; applying ACV improves PSMNet and GwcNet accuracy by 28% and 39%, respectively.ACV uses a lightweight correlation volume to generate attention weights that filter a concatenation volume.Source: §1 · p. 2
  • Fast-ACV combines volume attention propagation and Fine-to-Important sampling to restore high-resolution correlation clues and retain likely disparity hypotheses, reducing computational and memory costs.The approach constructs a low-resolution correlation volume; the paper describes Fast-ACV as targeting inference below 50 ms.Source: §1 · p. 2
  • ACVNet ranks 2nd on KITTI 2015 and Scene Flow, and 3rd on KITTI 2012 and ETH3D; Fast-ACVNet outperforms almost all state-of-the-art real-time methods on three listed benchmarks.The paper also reports ACVNet as the only method in the top three across all four datasets.Source: §1 · p. 2

2 RELATED WORK

Prior cost-volume designs trade information content against computational demands, while real-time methods reduce cost through low-resolution processing. The paper presents ACV and Fast-ACV as alternatives that use correlation information to guide concatenation-volume processing.

2.1 Cost Volume based Deep Stereo Matching

  • Correlation volumes are compact but lose content information, concatenation volumes preserve it but need deep 3D aggregation, and GwcNet combines both without fully using their complementary strengths.Cascaded methods also repeatedly reconstruct and aggregate volumes, and can miss true disparities when distributions have multiple peaks.Source: §2.1 · p. 3
  • Aggregation methods trade memory, runtime, and accuracy: GRU regularization increases runtime, AANet sacrifices accuracy, and GANet's guided layers increase aggregation time.The paper frames cost-volume construction and aggregation as coupled determinants of stereo matching accuracy and efficiency.Source: §2.1 · p. 3
  • ACV uses correlation-derived similarity information to regularize concatenation volumes, with the stated aim of enabling lightweight aggregation while retaining informative costs.The paper describes ACV as a general cost-volume representation.Source: §2.1 · p. 3

2.2 Real-time Stereo Matching

  • Real-time methods commonly use low-resolution 4D volumes to reduce computation, while StereoNet, DeepPruner, BGNet, and HITNet use different refinement or volume-restoration strategies.The paper contrasts Fast-ACV's stated generalization ability and plugin use with HITNet's extra training losses and potential generalization difficulty.Source: §2.2 · p. 3
  • Fast-ACV preserves multiple high-likelihood disparity hypotheses and adjusts their attention weights, unlike coarse-to-fine approaches that may accumulate errors by narrowing around one regressed disparity.The ACVNet figure depicts attention-weight generation, initial concatenation-volume construction, and attention filtering.Source: §2.2 · p. 4, Figure 1 · p. 4

3 METHOD

The method uses correlation-derived attention to filter concatenation volumes, then extends this design with low-resolution correlation processing and hypothesis selection for Fast-ACV. It describes VAP, F2I sampling, the two network architectures, and their training losses.

3.1 Attention Concatenation Volume

  • ACV derives attention from multi-level, adaptively weighted patch correlations, regularizes the matching volume, and uses the resulting weights to filter a concatenation volume.The method's MAPM patches use learned weights, and the matching costs are combined across feature levels.Source: §3.1 · pp. 4–5
  • The initial concatenation volume pairs left features at each location with right features shifted by each disparity, and attention weights are applied across its channels.The paper specifies the volume's size as 2Nc × D/4 × H/4 × W/4.Source: §3.1 · p. 5

3.2 Fast Attention Concatenation Volume

  • Fast-ACV performs most high-likelihood disparity-hypothesis and attention-weight calculations at low resolution, then builds and filters a compact concatenation volume from selected hypotheses.This reduces time and memory costs; the method describes F2I as reducing computation with little accuracy degradation.Source: §3.2 · p. 5, §3.2.2 · p. 6

3.2.1 Volume Attention Propagation

  • VAP propagates correlation values across a cross-shaped neighborhood using matching and confidence scores, favoring disparities that are both well matched and reliable.Confidence is based on probability-distribution variance; the method uses one propagation round and samples neighboring disparities in a cross shape.Source: §3.2.1 · pp. 5–6

3.2.2 Compact Concatenation Volume Construction via F2I and Filtering.

  • F2I selects the top K probability values per pixel as attention weights and retains their corresponding disparities as high-likelihood hypotheses, including when distributions are multimodal.The selected hypotheses define the compact concatenation volume that the attention weights filter.Source: §3.2.2 · pp. 6–7, Figure 5 · p. 7

3.3 ACVNet Architecture

  • ACVNet extracts multi-scale features, constructs and aggregates ACV, and predicts disparity using softmax probabilities and soft argmin.Its aggregation begins with four 3D convolutions and uses two stacked 3D hourglass networks.Source: §3.3 · p. 7

3.4 Fast-ACVNet Architecture

  • Fast-ACVNet uses multi-scale features, constructs Fast-ACV from low-resolution attention and selected disparities, then aggregates the compact volume and upsamples its disparity prediction.Its aggregation uses a guided hourglass network, with left-image features as guidance.Source: §3.4 · pp. 7–8

3.5 Loss Function

  • Both networks train with SmoothL1 disparity losses, with the ACVNet loss supervising attention-derived disparity as well as predicted disparities.For Fast-ACVNet, the text identifies compact attention weights and the final output as loss predictions.Source: §3.5 · p. 8

4.1 Datasets and Evaluation Metrics

The evaluation spans synthetic Scene Flow, real-world KITTI driving scenes, ETH3D grayscale scenes, and Middlebury indoor scenes, using dataset-specific disparity metrics.

4.1 Datasets and Evaluation Metrics

  • Scene Flow provides 35,454 training and 4,370 testing pairs at 960×540 resolution with dense ground truth; evaluation uses EPE and D1 outliers.D1 outliers are pixels whose disparity errors exceed max(3px, 0.05d*).Source: §4.1 · p. 8
  • KITTI 2012 and KITTI 2015 are real-world driving datasets with 194/195 and 200/200 training/testing pairs, respectively, and sparse LIDAR ground truth.KITTI 2012 reports x-noc, x-all, EPE-noc, and EPE-all; KITTI 2015 reports D1-bg, D1-fg, and D1-all.Source: §4.1 · p. 8
  • ETH3D contains 27 training and 20 testing grayscale pairs with sparse labels, while Middlebury has 15 training and 15 testing indoor pairs at full, half, and quarter resolutions.ETH3D reports bad 2.0 and bad 1.0; Middlebury reports bad 2.0.Source: §4.1 · p. 8

4.2 Implementation Details

The implementation uses PyTorch and RTX 3090 GPUs, Adam optimization, staged Scene Flow training, and KITTI fine-tuning of pretrained models.

4.2 Implementation Details

  • Experiments use PyTorch, NVIDIA RTX 3090 GPUs, and Adam with β1 = 0.9 and β2 = 0.999.The stated loss coefficients differ between ACVNet and Fast-ACVNet.Source: §4.2 · p. 8
  • On Scene Flow, ACVNet trains attention generation, the remaining network, and the complete network for 64 epochs each, while Fast-ACVNet trains attention generation and the complete network for 24 epochs each.Both schedules begin with a 0.001 learning rate and decay it at specified epochs.Source: §4.2 · p. 8
  • For KITTI, the pretrained Scene Flow model is fine-tuned on mixed KITTI 2012 and KITTI 2015 training sets for 500 epochs.The learning rate starts at 0.001 and drops to 0.0001 at epoch 300.Source: §4.2 · p. 8

4.3 Ablation Study

The ablations test ACV construction and aggregation choices, then examine VAP, F2I sampling, and lightweight components for Fast-ACV. They report accuracy gains from ACV and trade-offs between accuracy and runtime in the fast model.

4.3.1 Attention Concatenation Volume.

  • In the reported Scene Flow ablation, Gwc-att-hg improves D1 by 9.8% and EPE by 8.7%, respectively, compared with Gwc-att.Gwc-att-hg-s uses softmax and soft argmin functions for regression to predict disparity, supervised using ground truth.Source: §4.3.1 · p. 9, Table 1 · p. 9
  • 42.8% and 39.5% improvements in D1 and EPE over GwcNet establish the ACV result; the final ACVNet uses two hourglass aggregation networks.The selected model balances accuracy and aggregation-network size.Source: §4.3.1 · p. 9, Table 4 · p. 9

4.3.2 Fast Attention Concatenation Volume.

  • VAP reduced EPE from 0.66 to 0.62, while F2I sampling of 24 seeds gave the best accuracy-runtime balance.Increasing F2I sampling from 24 to 32 or 48 brought little accuracy improvement and substantially increased runtime.Source: Table 2 · p. 9, §4.3.2 · p. 10
  • The lightweight backbone and aggregation network reduced ACVNet runtime from 200ms to 62ms; adding VAP cut runtime by 19.4%, and VAP with F2I (24) accelerated it by 1.28×.The latter two changes incurred small accuracy degradation.Source: Table 3 · p. 9, §4.3.2 · p. 10

4.4 Performance of ACV and Fast-ACV

The section examines ACVNet's aggregation-network trade-off and Fast-ACV's accuracy against cascade cost volumes. It also identifies the scopes of the KITTI, runtime, and generalization evaluations.

4.4 Performance of ACV and Fast-ACV

  • 42.8% and 39.5% improvements in D1 and EPE over GwcNet are reported for Gwc-acv-3; reducing aggregation hourglasses slightly increased prediction errors, and ACVNet uses two.The paper selects two hourglasses to balance accuracy and efficiency.Source: §4.4 · p. 10
  • The evaluation materials cover KITTI 2012 and 2015 results, KITTI 2015 runtime for 1242×375 images, and generalization from Scene Flow training to KITTI and Middlebury training images.The supplied passages specify these evaluation scopes but not the corresponding table values.Source: Table 10 · p. 11, Table 11 · p. 11, Table 12 · p. 11
  • With one hourglass, Fast-ACV's attention filter improved EPE from 0.83 to 0.64 and outperformed the three-hourglass cascade cost volume without the filter.The paper attributes the reduced aggregation requirement to Fast-ACV's greater representation ability; cascade stages reconstruct and re-aggregate volumes.Source: §4.4 · p. 11

4.5 Universality and Superiority of Our Methods

The authors test whether ACV and Fast-ACV transfer to PSMNet and GwcNet, and whether VAP can improve networks using linear up-sampling cost volumes.

4.5 Universality and Superiority of Our Methods

  • ACV and Fast-ACV improved accuracy over the original PSMNet and GwcNet models and outperformed the corresponding cascaded cost-volume approaches.Fast-ACV also reduced FLOPs, parameter count, and runtime relative to the original models and cascaded approach.Source: §4.5 · p. 11
  • VAP integrated into PSMNet and GwcNet achieved significant performance improvements with little extra computation.The resulting models are named PSMNet-VAP and GwcNet-VAP.Source: §4.5 · p. 11

4.6 Comparisons with State-of-the-art

The reported comparisons place ACVNet among leading accuracy-oriented methods and Fast-ACVNet+ among leading real-time methods. The section also reports qualitative Middlebury 2014 results.

4.6 Comparisons with State-of-the-art

  • 38.4% EPE accuracy improvement on Scene Flow over LEAStereo is reported for ACVNet, which also achieves state-of-the-art results on ETH3D.The paper reports ACVNet as comparable in accuracy to LEAStereo on KITTI but faster, at 200ms versus 300ms.Source: §4.6 · pp. 11–12
  • Figure 9 presents qualitative generalization results on Middlebury 2014; the supplied passages do not state a quantitative outcome for that figure.The figure caption identifies the dataset and evaluation as generalization performance.Source: Figure 9 · p. 12
  • Fast-ACVNet+ achieved EPE 0.59 on Scene Flow, outperforming the other real-time methods reported at the time of writing.On KITTI 2012 and 2015, it outperformed all published real-time methods and was comparable in accuracy to HITNet at 45ms versus 54ms.Source: §4.6 · p. 12

4.7 Generalization Performance

The generalization evaluation trains comparison methods only on Scene Flow and tests Fast-ACVNet on Middlebury 2014 and KITTI 2012/2015. Fast-ACVNet outperforms other real-time methods in this evaluation.

4.7 Generalization Performance

  • The evaluation uses the half-resolution Middlebury 2014 training set and the full-resolution KITTI 2012 and 2015 training sets.Source: §4.7 · p. 12
  • Fast-ACVNet achieves state-of-the-art performance and outperforms other real-time methods in this generalization evaluation.The comparison methods are trained only on Scene Flow.Source: §4.7 · p. 12
  • All comparison methods in this evaluation are trained only on Scene Flow.Source: §4.7 · p. 12

5 CONCLUSION

ACV and Fast-ACV use similarity-based attention weights to filter concatenation volumes, and their networks achieve strong benchmark performance. The paper also presents these cost-volume representations as integrable into existing models and the Fast-ACV idea as applicable to cascade methods.

5 CONCLUSION

  • ACVNet ranks 2nd on KITTI 2015 and Scene Flow, and 3rd on KITTI 2012 and ETH3D.Source: §5 · p. 12
  • Fast-ACVNet outperforms almost all state-of-the-art real-time methods on Scene Flow, KITTI 2012, and KITTI 2015, with the best generalization ability among real-time methods.Source: §5 · p. 12
  • ACV and Fast-ACV filter concatenation volumes using attention weights generated from similarity measures.Source: §5 · p. 12
  • ACV and Fast-ACV can be integrated into existing 4D cost-volume stereo models for performance improvement.The Fast-ACV idea can also be applied to cascade cost-volume methods for accuracy gain and reduced computation and parameter count.Source: §5 · p. 12

Related papers

Loading 2209.12699v3…