Source-linked AI summary
GA-Net: Guided Aggregation Net for End-to-end Stereo Matching
Feihu Zhang, Victor Prisacariu, Ruigang Yang, Philip H. S. Torr
TL;DR
Stereo matching needs effective cost aggregation because ambiguous matches impair disparity estimation in occluded, textureless, reflective, and other challenging regions. The paper proposes differentiable semi-global and local guided aggregation layers to replace 3D convolutions, and reports state-of-the-art accuracy on Scene Flow and KITTI while reducing computational and memory costs.
Problem
Cost aggregation is needed to resolve ambiguous matching costs and obtain accurate disparity estimates in challenging regions.
Method
The paper introduces differentiable semi-global and local guided aggregation layers that replace 3D convolutions for end-to-end stereo reconstruction.
Results
GA-Net achieves state-of-the-art accuracy on both the Scene Flow dataset and KITTI benchmarks.
Takeaways & Limitations
Guided aggregation can improve disparity accuracy in challenging regions while replacing computationally costly 3D convolutions.
Takeaways & Limitations
Traditional SGM uses user-defined parameters and fixed penalties, while its hard-minimum selection can produce fronto-parallel surfaces.
Abstract
from arXiv · showhide
In the stereo matching task, matching cost aggregation is crucial in both traditional methods and deep neural network models in order to accurately estimate disparities. We propose two novel neural net layers, aimed at capturing local and the whole-image cost dependencies respectively. The first is a semi-global aggregation layer which is a differentiable approximation of the semi-global matching, the second is the local guided aggregation layer which follows a traditional cost filtering strategy to refine thin structures. These two layers can be used to replace the widely used 3D convolutional layer which is computationally costly and memory-consuming as it has cubic computational/memory complexity. In the experiments, we show that nets with a two-layer guided aggregation block easily outperform the state-of-the-art GC-Net which has nineteen 3D convolutional layers. We also train a deep guided aggregation network (GA-Net) which gets better accuracies than state-of-the-art methods on both Scene Flow dataset and KITTI benchmarks.
1. Introduction
Stereo matching is difficult in challenging regions, making cost aggregation a key step. GA-Net introduces guided aggregation layers that replace costly 3D convolutions while improving accuracy and efficiency.
- Stereo reconstruction estimates 3D geometry from pixel disparities but faces occlusions, textureless and reflective regions, thin structures, and repetitive textures.
- Cost aggregation is essential because ambiguous feature matches can assign lower costs to incorrect disparities in challenging regions.
- Semi-global aggregation approximates semi-global matching differentiably and aggregates costs across multiple directions over the whole image.
- Local guided aggregation targets thin structures and object edges, recovering details lost through down-sampling and up-sampling.
- Two guided aggregation layers with two 3D convolutional layers outperform GC-Net, which uses nineteen 3D convolutional layers.
- GA-Net achieves state-of-the-art accuracy on both the Scene Flow dataset and KITTI benchmarks.
2. Related Work
Stereo matching methods use feature extraction, cost aggregation, and disparity prediction, with aggregation needed to resolve ambiguous matches in difficult regions. Prior work spans local filtering, semi-global matching, and end-to-end neural networks, motivating differentiable guided aggregation.
- Deep stereo matching: End-to-end models incorporated feature extraction, cost aggregation, and disparity estimation into unified neural networks, including GC-Net's use of 3D convolutions.
- Local cost aggregation: Local cost filtering independently aggregates each cost-volume slice over neighboring pixels, offering fast or real-time operation.
- Semi-global matching: Semi-global matching aggregates costs along image-wide paths while enforcing penalties for small and large disparity changes.
- GA-Net replaces low-efficient 3D convolutions with backpropagatable semi-global and local guided aggregation layers.
3. Guided Aggregation Net
GA-Net introduces differentiable semi-global and local guided aggregation layers that replace costly 3D convolutions while addressing whole-image dependencies and thin structures. The network combines these layers with shared feature extraction, a guidance subnet, cost-volume processing, disparity regression, and smooth L1 training.
- GA-Net replaces low-efficient 3D convolutions with backpropagatable semi-global and local guided aggregation layers.
- Semi-Global Aggregation: Learnable penalty weights and weighted aggregation make SGA adaptive across locations and conditions while supporting repeated use in deep networks.
- Semi-Global Aggregation: SGA aggregates each disparity slice along four whole-image directions and selects the maximum directional result to preserve the strongest message.
- Local Aggregation: LGA uses three K × K filters per pixel to aggregate neighboring costs for disparities d, d −1, and d +1, refining thin structures and object edges.
- Network Architecture: The architecture combines shared stacked-hourglass feature extraction, a 4D cost volume, guided aggregation weights, and disparity regression trained with smooth L1 loss.
4. Experiments
Experiments show that guided aggregation improves stereo accuracy across ablations and benchmarks while reducing reliance on costly 3D convolutions. SGA suppresses ambiguity in challenging regions, and LGA further sharpens disparity estimates around details and edges.
- Ablation Study: 0.84 pixel average EPE and 2.71% 3-pixel threshold error were achieved by the best GA-Net setting on Scene Flow and KITTI 2015, respectively.The same setting also reached a 9.9% 1-pixel threshold error on Scene Flow.
- Effects of Guided Aggregations: 0.29 pixel lower average EPE was achieved by GA-Net-2 than GC-Net using two GA layers and two 3D convolutions.GA-Net-7 also outperformed PSMNet while using seven rather than twenty-five 3D convolutional layers.
- Effects of Guided Aggregations: 0.5-1.0 pixels of average EPE improvement resulted from adding GA layers across the evaluated architectures.GA-Net-2 reached EPE 1.51 versus 1.54 for GA-Net∗-11, indicating better performance with fewer 3D convolutions in this comparison.
- Comparisons with SGMs and 3D Convolutions: SGA suppressed probability noise in large textureless regions, while LGA further concentrated the probability peak near the ground-truth disparity.The analysis considered sky, reflective car-window, and object-edge regions.
- Comparisons with SGMs and 3D Convolutions: SGA produced better results than original SGM with handcrafted features and MC-CNN with CNN-based features.The paper attributes this to learned end-to-end parameters and fully guided aggregation controlled by learned weight matrices.
- Complexity and Real-time Models: 15∼20 fps was obtained by a real-time GA-Net-1 on 300×1000 images using a TESLA P40 GPU, while outperforming existing real-time stereo models.The model used one 3D convolutional layer, no LGA layers, and simplified cost-volume sampling.
- Scene Flow Dataset: 0.25 pixel EPE and 2.2% error-rate improvements over PSMNet were reported on the Scene Flow test set.The evaluation used average EPE and 1-pixel threshold error rate.
- KITTI 2012 and 2015 Datasets: GA-Net surpassed PSMNet on all reported KITTI evaluation metrics while using fewer low-efficient 3D convolutions.The reported examples also show accurate estimates in large textureless or reflective regions and preservation of object structures.
5. Conclusion
The paper develops guided aggregation strategies for end-to-end stereo matching, combining semi-global and local aggregation to improve accuracy in challenging regions while replacing costly 3D convolutions.
- The proposed GA strategies comprise semi-global aggregation (SGA) and local guided aggregation (LGA) layers for end-to-end stereo matching.
- The GA layers significantly improve disparity-estimation accuracy in occlusions, large textureless or reflective regions, and thin structures.
- GA layers can replace computationally costly 3D convolutions while achieving better accuracy.
A. Backpropagation of SGA
The SGA backpropagation is computed inversely, using gradients from the next max-selection layer and an iteratively calculated temporary gradient variable.
- SGA backpropagation computes gradients for w and C(p,d) inversely from the gradient supplied by the next max-selection layer.
- The backward procedure introduces a temporary gradient variable ∂E ∂Cbr and calculates it iteratively when d ≠ imax.
- When d = imax, the backward computation uses a separate case involving imax, the index of the maximum selected during forward propagation.
B. Details of the Architecture
GA-Net-15 uses guided aggregation together with 3D convolutions for cost aggregation, combining three SGA layers, two LGA layers, and fifteen 3D convolutional layers.
- GA-Net-15 contains three SGA layers for cost aggregation.
- GA-Net-15 contains two LGA layers for cost aggregation.
- GA-Net-15 contains fifteen 3D convolutional layers for cost aggregation.