Source-linked AI summary
Computing the Stereo Matching Cost with a Convolutional Neural Network
Jure Žbontar, Yann LeCun
TL;DR
Estimating depth from a rectified image pair requires reliable disparity and matching-cost computation. The paper trains a convolutional neural network to score patch matches, refines those costs through aggregation and semiglobal matching, and reports a 2.61% error rate on KITTI.
Problem
The problem is to compute pixelwise disparity from a rectified stereo pair so that depth can be recovered.
Method
The method trains a convolutional neural network to evaluate matching between left-right image patches, then applies aggregation, semiglobal matching, consistency checking, and filtering.
Results
2.61% error rate on the KITTI stereo test set ranks the method first on the online leaderboard.
Takeaways & Limitations
The result suggests convolutional neural networks are a good fit for computing stereo matching costs, while larger training datasets may reduce error further.
Abstract
from arXiv · showhide
We present a method for extracting depth information from a rectified image pair. We train a convolutional neural network to predict how well two image patches match and use it to compute the stereo matching cost. The cost is refined by cross-based cost aggregation and semiglobal matching, followed by a left-right consistency check to eliminate errors in the occluded regions. Our stereo method achieves an error rate of 2.61 % on the KITTI stereo dataset and is currently (August 2014) the top performing method on this dataset.
1. Introduction
The paper frames stereo reconstruction as estimating disparity from rectified image pairs and proposes a CNN-based matching-cost computation within a standard stereo pipeline. It reports a 2.61% KITTI error rate, improving on the previous best result.
- Problem: Stereo reconstruction estimates each pixel's disparity from images captured at different horizontal camera positions.Disparity is the horizontal location difference between corresponding objects, and it determines depth together with camera focal length and baseline.
- Pipeline: A typical stereo algorithm comprises matching-cost computation, cost aggregation, optimization, and disparity refinement.The paper groups the first two steps as computing the matching cost and the latter two as the stereo method.
- Method: The method trains a convolutional neural network on image-patch pairs with known true disparities to initialize matching costs.The network is trained using examples whose ground-truth disparity can be obtained from sources such as LIDAR.
- Method: Cross-based aggregation, semiglobal matching, consistency checking, and filtering refine the costs and produce the final disparity map.Consistency checking detects and eliminates errors in occluded regions, followed by subpixel enhancement, median filtering, and bilateral filtering.
- Result: 2.61% error rate on the KITTI stereo dataset improves on the previous best result of 2.83%.This is the paper's reported stereo-method result on KITTI.
2. Related work
Earlier stereo work used ground-truth data to learn matching-cost adjustments, confidence measures, or graphical-model parameters. The reviewed approaches include probabilistic classifiers, random forests, and learned graphical-model objectives.
- Supervised stereo: Before large stereo datasets, relatively few stereo algorithms used ground-truth information to learn model parameters.The section reviews stereo methods that incorporated supervised information before the recent dataset scale-up.
- Learned matching costs: Kong and Tao used class-probability predictions to adjust an initial matching cost based on whether disparities were correct or incorrect.Their later work combined predictions from normalized cross-correlation computed over different window sizes and centers.
- Graphical models: Zhang and Seitz estimated Markov random field hyperparameters by alternative optimization.Their work illustrates learning parameters of graphical models from ground-truth data.
- Graphical models: Scharstein and Pal learned conditional random field parameters from a newly constructed dataset of 30 stereo pairs.Li and Huttenlocher instead used a non-parametric cost function with a structured support vector machine to learn conditional random field parameters.
- Confidence estimation: Recent methods used random forests to estimate matching-cost confidence and incorporate those predictions into stereo inference.Haeusler et al. combined confidence measures, while Spyropoulos et al. used confidence predictions as soft constraints in a Markov random field.
3. Computing the matching cost
The method learns a patch-matching cost with a convolutional neural network, then makes its computation practical through shared feature processing and convolutional evaluation across image locations.
- Matching-cost formulation: A matching cost C(p, d) is computed at each image position p for every candidate disparity d.The cost measures how well a left-image patch matches a right-image patch shifted according to disparity.
- Training examples: Training examples pair left and right 9 × 9 patches, using known disparities to construct positive and corrupted negative matches.Negative offsets are sampled away from the true disparity, while positive offsets include near matches to support later aggregation.
- Network architecture: The network processes two grayscale 9 × 9 patches through a convolutional layer, fully connected layers, tied early weights, and a two-class softmax.The architecture has nearly 600 thousand parameters and distinguishes good from bad matches.
- CNN cost: The CNN matching cost CCNN(p, d) is the network’s negative-class output for patches centered at p and the disparity-shifted position.The negative-class probability supplies the cost used for each candidate disparity.
- Efficient evaluation: Runtime is reduced by computing early-layer features once per location and replacing later fully connected layers with convolutional filters.The method still requires a forward pass for each disparity under consideration.
4. Stereo method
The stereo method refines learned matching costs through adaptive aggregation and semiglobal smoothness optimization, then selects and refines disparities while handling occlusions and image-boundary effects.
- 4.1. Cross-based cost aggregation: Cross-based aggregation averages matching costs over support regions formed from intensity- and distance-constrained cross arms.The combined support region uses support regions from both stereo images, and averaging is repeated four times.
- 4.2. Semiglobal matching: Semiglobal matching minimizes a disparity energy along four image directions and averages the resulting costs.The energy penalizes high matching costs and disparity changes, with penalties adjusted according to image gradients.
- 4.3. Computing the disparity image: The disparity image is obtained by choosing the disparity d that minimizes the matching cost C(p, d).This uses a winner-take-all strategy after cost processing.
- 4.3.1 Interpolation: A left-right consistency check labels pixels as correct, mismatch, or occlusion to detect errors in occluded regions.Occlusion disparities are interpolated from the background, while mismatch disparities use medians from nearby correct pixels in 16 directions.
- 4.3.3 Refinement: Subpixel enhancement fits a quadratic through neighboring costs, followed by border enlargement, median filtering, and bilateral filtering.The bilateral-filtered disparity map DBF is the final output of the stereo method.
5. Experimental results
The method is evaluated on KITTI using a supervised CNN and a stereo pipeline that achieves a 2.61% test error. Runtime and training-set-size experiments examine practical performance and scaling.
- 5.3. Results: The method is evaluated on KITTI because its large training set supports learning the convolutional-network weights.The dataset provides ground-truth depth from a rotating laser scanner mounted behind the left camera.
- 5.3. Results: 2.61% error is achieved on the KITTI stereo test set, with the method ranked first on the online leaderboard.The KITTI dataset contains 194 training and 195 test rectified image pairs at 1240 × 376 resolution.
- 5.4. Runtime: Training takes 5 hours, while predicting one image pair takes 100 seconds on an Nvidia GeForce GTX Titan GPU.Most prediction time is spent in the convolutional neural network’s forward pass.
- 5.5. Training set size: Test error has an almost linear relationship with the number of stereo pairs used for training.The authors report that this relationship implies further improvement as larger datasets become available.
6. Conclusion
The results suggest that convolutional neural networks are well suited to computing stereo matching costs, while the method remains unsuitable for real-time applications.
- The KITTI result suggests that convolutional neural networks are a good fit for computing stereo matching costs.
- The method is not yet suitable for real-time applications such as robot navigation.Future work focuses on improving the network’s runtime performance.