Source-linked AI summary
End-to-End Learning of Geometry and Context for Deep Stereo Regression
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, Adam Bry
TL;DR
Stereo disparity estimation requires robust correspondence under difficult visual conditions. GC-Net combines a geometry-aware cost volume with learned 3-D contextual regularization and differentiable soft argmin regression. It achieves state-of-the-art KITTI results, improves on prior state of the art by 9% and 22% on KITTI 2015 and 2012, and avoids additional post-processing or regularization.
Problem
Stereo correspondence is difficult in textureless areas, reflective surfaces, thin structures, and repetitive patterns, while hand-engineered regularization can trade smoothing against detail.
Method
GC-Net forms a fully differentiable geometry-aware cost volume, learns contextual regularization with 3-D convolutions, and regresses sub-pixel disparity using soft argmin.
Results
GC-Net achieves state-of-the-art results on KITTI 2012 and 2015, improving on state of the art by 22% and 9%, respectively.
Takeaways & Limitations
The model learns disparity regression end-to-end without additional post-processing or regularization while incorporating wider contextual information.
Takeaways & Limitations
Soft argmin is susceptible to multi-modal disparity distributions because it estimates a weighted average of all modes.
Abstract
from arXiv · showhide
We propose a novel deep learning architecture for regressing disparity from a rectified pair of stereo images. We leverage knowledge of the problem's geometry to form a cost volume using deep feature representations. We learn to incorporate contextual information using 3-D convolutions over this volume. Disparity values are regressed from the cost volume using a proposed differentiable soft argmin operation, which allows us to train our method end-to-end to sub-pixel accuracy without any additional post-processing or regularization. We evaluate our method on the Scene Flow and KITTI datasets and on KITTI we set a new state-of-the-art benchmark, while being significantly faster than competing approaches.
1. Introduction
Stereo disparity estimation is difficult in challenging visual conditions, motivating an end-to-end model that combines geometric cost volumes with learned contextual reasoning. GC-Net uses 3-D convolutions and differentiable soft argmin to regress sub-pixel disparities, and is evaluated on Scene Flow and KITTI.
- Motivation: Stereo correspondence is challenging in textureless areas, reflective surfaces, thin structures, and repetitive patterns.Existing pooling or gradient-based regularization can trade surface smoothing against detailed-structure detection.
- Motivation: Deep stereo models could benefit from global semantic context rather than relying solely on local geometry.The paper uses reflective vehicle windshields as an example where semantic context can help infer local geometry.
- Approach: GC-Net formulates stereo disparity estimation end-to-end using a geometry-aware cost volume and a deep convolutional architecture for semantic reasoning.The model targets per-pixel disparity from a single rectified image pair.
- Approach: 3-D convolutions learn to regularize the cost volume across height, width, and disparity dimensions.This incorporates contextual information directly from the data.
- Approach: A fully differentiable soft argmin regresses sub-pixel disparity values from the cost volume.The operation supports end-to-end training without an additional post-processing stage.
- Evaluation: GC-Net is evaluated on Scene Flow and achieves state-of-the-art results on the KITTI 2012 and 2015 benchmarks.The introduction reports evidence that the model can learn semantic and contextual information.
2. Related Work
Prior stereo systems used hand-engineered matching, aggregation, optimization, and refinement components, while deep methods often learned only local matching costs. GC-Net instead forms a differentiable cost volume and learns contextual regularization and sub-pixel disparity regression end-to-end.
- Traditional stereo: Stereo algorithms traditionally decompose into matching-cost computation, cost aggregation, disparity computation and optimization, and disparity refinement.Middlebury and KITTI provide benchmark datasets and evaluation infrastructure for these components.
- Traditional stereo: Global stereo optimization combines local data terms with pairwise smoothness terms, with SGM approximating optimization through pathwise dynamic programming.SGM optimizes paths in multiple directions.
- Deep stereo: Deep networks have been used for patch matching and local matching-cost estimation, but these approaches commonly retain non-learned aggregation or regularization.Examples include patch-matching networks and Siamese multi-label disparity classification.
- Deep stereo: Mayer et al. improved disparity estimation with a large synthetic dataset and a 1-D correlation along the disparity line.Their volume was concatenated with single-image convolutional features and processed by further convolutions.
- GC-Net: GC-Net forms a fully differentiable cost volume, learns contextual regularization with 3-D convolutions, and directly regresses sub-pixel disparity without post-processing.The method does not collapse the feature dimension when computing the cost volume.
3. Learning End-to-end Disparity Regression
GC-Net learns end-to-end disparity regression while preserving stereo geometry in a cost volume and incorporating context with 3-D convolutions. A differentiable soft argmin converts the regularized volume into smooth disparity estimates, while addressing the limitations of discrete argmin.
- Architecture: The architecture uses learned end-to-end mappings rather than hand-designed stereo steps, with its layer structure summarized in Table 1.The unary representation is learned from image features before cost-volume construction and contextual regularization.
- Cost Volume: GC-Net forms a geometry-aware cost volume by concatenating corresponding left and right unary features across disparity levels while retaining feature dimensions.The volume has dimensions height×width×(max disparity + 1)×feature size, enabling learned processing over feature unaries.
- Learning Context: 3-D convolutions regularize the cost volume across height, width, and disparity to incorporate contextual information.A four-level encoder-decoder reduces the volume through sub-sampling and restores the original resolution with transposed convolutions.
- Differentiable ArgMin: Soft argmin replaces discrete argmin by applying softmax to negative costs and computing the probability-weighted sum of disparity indices.This operation is fully differentiable and produces smooth disparity estimates suitable for end-to-end training.
- Differentiable ArgMin: Soft argmin can fail on multi-modal cost distributions because it averages all modes rather than selecting the most likely one.The network is expected to regularize distributions toward a predominantly unimodal form and can pre-scale costs to make probabilities more extreme.
4. Experimental Evaluation
Experiments analyze GC-Net’s design on Scene Flow and evaluate it on KITTI benchmarks, showing benefits from learned 3-D cost-volume regularization and regression, alongside strong benchmark performance and contextual reasoning.
- Experimental setup: The experiments evaluate model variants on Scene Flow, then assess KITTI benchmark performance and the model’s ability to learn contextual information.Scene Flow contains 35,454 training and 4,370 testing images; KITTI provides challenging road-scene imagery but only 200 training images.
- Model Design Analysis: 3-D convolutional filters perform significantly better than unary-only features by regularizing and smoothing disparity outputs while retaining sharpness and accuracy.The hierarchical 3-D model aggregates substantially more context than vanilla 3-D convolution without significantly increasing computational demand.
- Model Design Analysis: Regression loss outperforms hard and soft disparity classification, especially for disparities within one pixel of the truth, because it supports sub-pixel predictions.Classification converges faster during training, but regression performs best overall.
- KITTI Benchmark: GC-Net achieves state-of-the-art results on KITTI 2012 and 2015, improving over the prior state of the art by 22% and 9%, respectively.The method is also reported to be notably faster than most competing approaches, which often require expensive post-processing.
- KITTI Benchmark: Compared with DispNetC, GC-Net improves performance by using a full geometry-aware cost volume, 3-D convolutions, and a soft argmin layer rather than a 1-D correlation approximation.The comparison identifies the explicit geometric formulation as the basis for the reported improvement.
- Model Saliency: Saliency results show that disparity predictions depend on a wide contextual field of view, including the vehicle and road surface rather than only local 9×9 patches.The evidence supports learned contextual reasoning beyond local geometry and appearance.
5. Conclusions
The paper presents an end-to-end stereo architecture that learns contextual regularization in a disparity cost volume and regresses sub-pixel disparities without additional post-processing. It demonstrates efficacy on KITTI, while identifying explicit semantic representation and uncertainty reasoning as future directions.
- 5. Conclusions: The method estimates disparity end-to-end without additional post-processing or regularization.
- 5. Conclusions: On the KITTI dataset, the method sets a new state-of-the-art benchmark.
- 5. Conclusions: The architecture learns context in the disparity cost volume using 3-D convolutions and regresses disparity with a differentiable soft argmin.These components enable regression rather than classification and support sub-pixel accuracy.
- 5. Conclusions: The model learns to incorporate wider contextual information into disparity estimation.
- 5. Conclusions: Future work includes more explicit semantic representations and Bayesian convolutional neural networks for reasoning under uncertainty.