PaperlayerView exact revision on arXiv

Source-linked AI summary

AANet: Adaptive Aggregation Network for Efficient Stereo Matching

Haofei Xu, Juyong Zhang

arXiv:2004.09548v1cs.CV

TL;DR

Leading stereo models often rely on costly 3D convolutions, limiting deployment despite strong accuracy. AANet replaces them with sparse adaptive intra-scale aggregation and neural cross-scale aggregation, achieving competitive Scene Flow and KITTI performance with substantially faster inference.

  • Problem

    Costly 3D convolutions in state-of-the-art stereo models make inference expensive in computation, memory, and deployment time.

  • Method

    AANet replaces 3D convolutions with complementary sparse adaptive intra-scale aggregation and neural cross-scale aggregation modules.

  • Results

    41× faster than GC-Net, 4× faster than PSMNet, and 38× faster than GA-Net, with competitive Scene Flow and KITTI results at 62ms.

  • Takeaways & Limitations

    The lightweight adaptive aggregation architecture provides a versatile, efficient alternative to commonly used 3D convolutions and can improve fast stereo models.

Abstract

from arXiv · show

Despite the remarkable progress made by learning based stereo matching algorithms, one key challenge remains unsolved. Current state-of-the-art stereo models are mostly based on costly 3D convolutions, the cubic computational complexity and high memory consumption make it quite expensive to deploy in real-world applications. In this paper, we aim at completely replacing the commonly used 3D convolutions to achieve fast inference speed while maintaining comparable accuracy. To this end, we first propose a sparse points based intra-scale cost aggregation method to alleviate the well-known edge-fattening issue at disparity discontinuities. Further, we approximate traditional cross-scale cost aggregation algorithm with neural network layers to handle large textureless regions. Both modules are simple, lightweight, and complementary, leading to an effective and efficient architecture for cost aggregation. With these two modules, we can not only significantly speed up existing top-performing models (e.g., $41\times$ than GC-Net, $4\times$ than PSMNet and $38\times$ than GA-Net), but also improve the performance of fast stereo models (e.g., StereoNet). We also achieve competitive results on Scene Flow and KITTI datasets while running at 62ms, demonstrating the versatility and high efficiency of the proposed method. Our full framework is available at https://github.com/haofeixu/aanet .

1. Introduction

Learning-based stereo matching performs well but commonly relies on costly 3D convolutions, motivating AANet’s replacement of them with lightweight adaptive aggregation modules. The proposed approach targets both disparity discontinuities and textureless regions while preserving competitive accuracy and substantially improving speed.

  • Motivation: 3D convolutions make leading stereo models expensive in computational cost, memory consumption, and practical inference time.PSMNet is reported to require about 4G memory and 410ms for a KITTI stereo pair even on high-end GPUs.
  • Motivation: AANet asks whether state-of-the-art stereo results can be achieved without any 3D convolutions while being significantly faster.The paper uses simple feature correlation rather than feature concatenation and evaluates on Scene Flow and KITTI.
  • Approach: Adaptive sparse-point intra-scale aggregation samples regions with similar disparities to alleviate edge-fattening at disparity discontinuities.The representation can also sample a large context more efficiently than a large window.
  • Approach: Neural cross-scale aggregation uses parallel multi-scale cost volumes and adaptive multi-scale interactions to produce accurate predictions in low-texture or textureless regions.This module approximates a traditional cross-scale cost aggregation algorithm with neural network layers.
  • Results: 41× faster than GC-Net, 4× faster than PSMNet, and 38× faster than GA-Net, AANet also improves fast stereo models such as StereoNet.The paper additionally reports competitive KITTI performance at 62ms.

2. Related Work

Prior stereo methods aggregate costs locally or across scales, but regular convolutions use spatially shared weights and fixed sampling, while large windows increase computation. AANet introduces adaptive sparse sampling, content-adaptive weighting, and simultaneous multi-scale aggregation to improve flexibility and efficiency.

  • Local Cost Aggregation: Local cost aggregation assumes neighboring pixels have similar disparities, an assumption that fails at discontinuities and causes edge-fattening in object boundaries and thin structures.Large windows can improve quality but increase computational cost.
  • Local Cost Aggregation: Regular convolutions use spatially shared, content-agnostic weights, whereas AANet learns content-adaptive weights for position-specific aggregation.This distinction addresses the weighting limitations of regular convolution-based aggregation.
  • Local Cost Aggregation: AANet’s sparse points are completely unconstrained and adaptively sampled, providing more flexibility than regular sparse sampling inside a matching window.The approach is designed for efficient and flexible cost aggregation.
  • Cross-Scale Cost Aggregation: Traditional cross-scale aggregation enforces multi-scale consistency, and AANet approximates the resulting adaptive cost-volume combination with end-to-end neural network layers.The approach is based on combining aggregation results from different scales.
  • Cross-Scale Cost Aggregation: Unlike hierarchical fusion from the lowest scale upward, AANet aggregates all scale cost volumes simultaneously.Both approaches construct multi-scale cost volumes, but differ in fusion strategy.
  • Stereo Matching Networks: AANet’s adaptive aggregation targets sampling complexity and flexibility, making it complementary to methods that improve cost-volume construction.The paper also relates its adaptive sampling formulation to deformable convolution while applying it to stereo cost aggregation.

3. Method

AANet replaces costly 3D convolutions with adaptive intra-scale and cross-scale cost aggregation modules, then refines multi-scale disparity predictions. Its pipeline uses correlated feature pyramids, stacked aggregation modules, end-to-end supervision, and soft-argmin regression.

  • Network Overview: AANet extracts shared downsampled feature pyramids and constructs multi-scale cost volumes by correlating left and right features at corresponding scales.The highest scale is indexed by s = 1, and each cost volume stores matching costs for disparity candidates.
  • Adaptive Intra-Scale Aggregation: Sparse adaptive sampling learns offsets and content-adaptive weights for intra-scale aggregation, targeting disparity discontinuities, object boundaries, and thin structures.The representation uses K = 3 sampling points and is implemented with deformable convolution.
  • Adaptive Intra-Scale Aggregation: The ISA module stacks 1 × 1, deformable 3 × 3, and 1 × 1 convolutions with a residual connection while preserving the disparity-candidate channels.This keeps the module reasoning about disparity candidates rather than converting the cost volume into generic feature channels.
  • Adaptive Cross-Scale Aggregation: CSA approximates traditional cross-scale aggregation by adaptively combining intra-scale cost volumes from multiple resolutions through neural network transformations.Stride-2 convolutions downsample finer volumes, while bilinear upsampling and 1 × 1 convolution align coarser volumes; lower-scale disparity channels are halved for efficiency.
  • Full Architecture: ISA and CSA are integrated into stacked AAModules, while the feature extractor uses deformable convolutions and a three-level feature pyramid before hierarchical disparity refinement.The pyramid uses 1/3, 1/6, and 1/12 resolutions, and the 1/3 prediction is upsampled through two refinement stages.
  • Disparity Regression and Training: Soft argmin converts aggregated matching costs into sub-pixel disparity predictions, and training combines losses across predictions using ground-truth and, for KITTI, pseudo-ground-truth supervision.The pseudo-ground truth comes from predictions of a pre-trained stereo model because KITTI’s disparity ground truth is sparse.

4. Experiments

Experiments show that AANet's complementary aggregation modules improve disparity predictions while replacing costly 3D convolutions with a faster, lighter architecture. The method performs competitively across Scene Flow, KITTI, and Middlebury evaluations, including improved results for fast stereo models and challenging regions.

  • Ablation Study: Ablating either ISA or CSA causes a clear performance drop, while integrating both modules gives the best performance.Visual comparisons also show better predictions in thin structures and textureless regions.
  • Sampling Points Visualization: Adaptive sampling concentrates on similar-disparity regions near object boundaries and samples a larger context in textureless regions.This demonstrates dynamic sampling that addresses local ambiguities without relying on a large fixed window.
  • Pseudo Ground Truth Supervision: D1-all decreases from 2.29 to 2.15 with pseudo ground truth supervision, while EPE increases from 0.68 to 0.69.The authors associate the EPE change with possible instability from the small validation set, while qualitative results show fewer artifacts where ground truth is unavailable.
  • Comparison with 3D Convolutions: AANet directly aggregates a 1/4-resolution cost volume with less computation while remaining more accurate and faster than StereoNet.Against GC-Net, PSMNet, and GA-Net, it uses fewer resources and is 41×, 4×, and 38× faster, respectively; GA-Net has lower EPE.
  • Comparison with 3D Convolutions: Deformable convolution has less than 1/130 the computational complexity of a 3D convolution under the stated assumptions.The comparison uses O(K^3C^2DHW) for 3D convolution and O(K^2D^2HW + 3K^4DHW + 3K^2DHW) for deformable convolution.
  • Benchmark Results: AANet achieves state-of-the-art Scene Flow results and remains substantially faster than existing top-performing methods, while AANet+ uses 8.4M parameters versus AANet's 3.9M.On Middlebury, a KITTI fine-tuned model produces sharper object boundaries and better preserves overall structures than PSMNet.

5. Conclusion

The paper presents an efficient cost-aggregation architecture with competitive performance on Scene Flow and KITTI. It also reports generic applicability and identifies extensions to other cost-volume tasks and downstream stereo-based detection.

  • AANet demonstrates high efficiency and competitive performance on both Scene Flow and KITTI datasets.
  • Extensive experiments validate the generic applicability of the proposed cost-aggregation method.
  • Future directions include high-resolution stereo matching, multi-view stereo, optical flow estimation, and stereo-based 3D object detection.

Appendix

The appendix describes traditional local and cross-scale cost aggregation through unified optimization formulations. AANet approximates the resulting adaptive multi-scale combination with neural network layers.

  • Traditional local cost aggregation can be reformulated within a unified optimization framework using neighborhood weighting.
  • The multi-scale formulation aggregates costs independently at each scale before imposing inter-scale consistency.
  • The optimization solution combines cost-aggregation results from different scales adaptively through a matrix-based formulation.
  • AANet approximates this adaptive cross-scale combination with neural network layers defined as functions f_k.
Loading 2004.09548v1…