Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

• Stereo matching estimates each left-image pixel's horizontal disparity relative to the right image, which can be used with camera focal length and baseline to compute depth.The figure illustrates a dense disparity map, with nearer objects having larger disparities.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The work focuses on computing a good matching cost, the first of four typical stereo-algorithm steps; cost aggregation, optimization, and disparity refinement follow.Source: §1 · p. 2
• The authors describe two CNN architectures for stereo matching costs and train on small image-patch pairs with known disparity, then apply aggregation, optimization, and refinement steps.Post-processing includes cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, and median and bilateral filtering.Source: §1 · p. 2
• The method is reported to achieve the lowest error rate on KITTI 2012, KITTI 2015, and Middlebury, and is accompanied by source code.Source: §1 · p. 2
• Experiments analyze training-set size, error rates versus other methods, and the accuracy-runtime trade-off across hyperparameter settings; this extends prior work with a new architecture and two new datasets.The extension also reports lower error rates and more thorough experiments.Source: §1 · pp. 2–3

2. Related Work

• Kong and Tao (2004, 2006) and Peris et al. (2012) learned corrections or mappings from initial matching costs, using error-class probabilities, multiscale correlations, or AD-Census.Kong and Tao predicted probabilities for disparity-error classes and later combined normalized cross-correlation across window sizes and centers; Peris et al. mapped AD-Census costs to final disparity using multiclass linear discriminant analysis.Source: §2 · p. 3
• Ground-truth supervision also learned graphical-model parameters: Zhang and Seitz (2007) optimized MRF hyperparameters, while Scharstein and Pal (2007) and Li and Huttenlocher (2008) learned CRF parameters.Scharstein and Pal constructed a 30-pair stereo dataset, and Li and Huttenlocher used a structured support vector machine with a non-parametric cost function.Source: §2 · p. 3
• Haeusler et al. (2013) and Spyropoulos et al. (2014) estimated matching-cost confidence with random forests; Spyropoulos et al. used predictions as soft constraints in an MRF to reduce stereo error.Haeusler et al. combined several confidence measures, while Spyropoulos et al. predicted matching-cost confidence.Source: §2 · p. 3
• Local-descriptor learning shares patch-similarity measurement with stereo matching costs and spans boosting, convex optimization, hierarchical similarity, convolutional kernels, and CNNs.Zagoruyko and Komodakis (2015) and Han et al. (2015) most closely resemble this work, differing mainly in network architecture, including pooling and subsampling to handle larger patches and viewpoint variation.Source: §2 · p. 3

3. Matching Cost

• Matching cost compares left and right patches at candidate disparities, aiming for low cost when they depict the same 3D point and high cost otherwise.A typical stereo algorithm computes a matching cost at each image position for every disparity under consideration.Source: §3 · p. 4

3.1 Constructing the Data Set

• Ground-truth disparities yield balanced binary training data: each known pixel provides a positive same-point patch pair and a negative pair depicting different points.Random offsets for positive pairs include near matches because cross-based cost aggregation benefits when they also receive low matching costs; the positive offset was at most one pixel.Source: §3.1 · pp. 4–5

3.2 Network Architectures

• Both architectures learn patch features and compare their feature vectors; the fast version is quicker, while the accurate version is slightly more accurate.The fast architecture uses a fixed feature-vector similarity measure, whereas the accurate architecture learns that measure.Source: §3.2 · p. 5

3.2.1 Fast Architecture

• The fast siamese network compares shared convolutional patch features by cosine similarity and trains paired positive and negative examples with a hinge loss using margin 0.2.The loss is zero when the positive example’s similarity exceeds the negative example’s similarity by at least the margin.Source: §3.2.1 · pp. 5–6

3.2.2 Accurate Architecture

• The accurate architecture concatenates convolutional features and uses fully connected layers with a sigmoid similarity score; it is slower but reduces error, and binary cross-entropy performed better than hinge loss.A final sigmoid transforms the output to a similarity score between 0 and 1.Source: §3.2.2 · pp. 6–7, Figure 3 · p. 6

3.3 Computing the Matching Cost

• Negative network similarity initializes matching cost; feature extractors run once per image, while the accurate architecture’s fully connected layers run once per disparity and the fast version repeats only the dot product.Full-image passes reuse intermediate computations instead of running separate patch-level passes for every location.Source: §3.3 · pp. 7–8

4.1 Cross-based Cost Aggregation

• Cross-based aggregation selects intensity-similar, distance-limited neighborhoods, combines support regions from both stereo images, and averages matching costs over that support.Source: §4.1 · p. 8

4.2 Semiglobal Matching

• Semiglobal matching optimizes a smoothness-regularized cost along four horizontal and vertical directions, averages directional costs, and adjusts disparity penalties using image gradients, with lower penalties vertically.Source: §4.2 · pp. 9–10

4.3 Computing the Disparity Image

• The disparity image is computed by choosing, at each pixel, the disparity that minimizes its matching cost.Source: §4.3 · p. 10

4.3.1 Interpolation

• A left-right consistency check labels disparities as correct, mismatched, or occluded; occlusions use a leftward background value, while mismatches use the median from 16 directions.Source: §4.3.1 · p. 11

4.3.2 Subpixel Enhancement

• Subpixel enhancement fits a quadratic through the costs at the selected disparity and its two neighboring disparities to refine the disparity estimate.Source: §4.3.2 · p. 11

4.3.3 Refinement

• Refinement applies a 5 × 5 median filter followed by a bilateral filter, which smooths the disparity map without blurring edges.Source: §4.3.3 · p. 11

5. Experiments

• Test error rates for KITTI 2012, KITTI 2015, and Middlebury were obtained by submitting disparity maps to online evaluation servers; other error rates used a train/validation split.Table error is the percentage of misclassified pixels, and runtime is seconds per image pair; the setting column indicates optical flow, multiple temporal images, or epipolar geometry for flow.Source: §5 · pp. 11–12

5.1 KITTI Stereo Data Set

• KITTI consists of rectified driving-image pairs recorded at 1240 × 376 resolution, with laser-scanner ground truth labeling around 30% of pixels.The cameras were mounted about 54 centimeters apart on a car driven around Karlsruhe in daytime, sunny or cloudy weather.Source: §5.1 · p. 12
• KITTI error is the percentage of pixels whose predicted disparity differs from ground truth by more than three pixels; test ground truth is withheld and evaluated through an online leaderboard.The three-pixel tolerance corresponds, for example, to 3 centimeters at 2 meters and 80 centimeters at 10 meters.Source: §5.1 · p. 12
• KITTI 2012 has 194 training and 195 test images, while KITTI 2015 has 200 of each; 2015 densely labels moving vehicles and evaluates car glass, emphasizing reflective surfaces.The datasets are nearly identical for stereo, though 2015 improves some aspects of optical flow.Source: §5.1 · p. 13
• On KITTI 2012, the accurate architecture ranked first with 2.43% error, while the previous work ranked third with 2.61%.The reported changes reducing error from 2.61% to 2.43% were data augmentation and doubling the convolution layers while reducing kernel size from 5 × 5 to 3 × 3.Source: Table 1 · p. 12, §5.1 · p. 13
• The fast architecture had 2.82% KITTI 2012 test error and took 0.8 seconds per image pair, versus 67 seconds for the accurate architecture.The fast architecture would have ranked fifth if leaderboard entry had been permitted; Figure 5 says both architectures outperform census transform on the illustrated examples, with fewer average mistakes for the accurate architecture.Source: §5.1 · p. 13, Figure 5 · p. 14
• On KITTI 2015, the accurate and fast architectures achieved error rates of 3.89% and 4.46%, placing first and second on the leaderboard.Because only one submission per paper was allowed, only the accurate architecture appears on the public leaderboard.Source: §5.1 · pp. 13, 15, Table 2 · p. 13

5.2 Middlebury Stereo Data Set

• The benchmark concatenates five Middlebury releases and uses indoor scenes with structured-light disparities measured at higher density and precision than KITTI.The 2005, 2006, and 2014 releases include multiple lighting conditions and exposures, typically 28 images per scene.Source: §5.2 · p. 15
• The test set contains 15 images from the 2005 and 2014 releases, and the public leaderboard permits only one test-set submission per participant.This is stricter than KITTI, where submissions are allowed every three days.Source: §5.2 · p. 15
• The fast architecture has a 2.79% error, while the accurate architecture has a 2.36% error.Source: §5.2 · p. 16
• 8.29% error ranks the method first on the Middlebury leaderboard, ahead of second-placed MeshStereo at 13.4%.Table 4 reports the leaderboard snapshot as of October 2015.Source: Table 4 · p. 17, §5.2 · p. 17
• The network is trained on imperfectly rectified pairs, although only two of the 15 test images are perfectly rectified.Standard calibration can leave vertical disparity errors of up to nine pixels in Middlebury 2014.Source: §5.2 · pp. 15, 17
• Error is the percentage of non-occluded pixels differing from ground truth by more than two pixels, averaged across 15 test images with dataset-author weights.Half- or quarter-resolution disparity maps are upsampled before evaluation; the method runs on half-resolution images because of graphics-card memory limits.Source: §5.2 · pp. 15, 17

5.3 Details of Learning

• The training sets contained 25 million KITTI 2012, 17 million KITTI 2015, and 38 million Middlebury examples.Source: §5.3 · p. 17
• Training used batches of 128 patch pairs, 14 epochs, momentum 0.9, and initial learning rates of 0.003 for the accurate and 0.002 for the fast architecture.The learning rate was reduced by a factor of 10 on epoch 11; training used mini-batch gradient descent.Source: §5.3 · pp. 17, 19
• On a difficult Middlebury pair with a nearly textureless white wall, the accurate architecture classified most of the wall correctly; the fast architecture performed worse but better than census.Source: Figure 7 · p. 18
• Each stereo image was normalized separately by its pixel-intensity mean and standard deviation, and color images were converted to grayscale.Initial experiments suggested that color did not improve disparity-map quality.Source: §5.3 · p. 19
• Post-processing ran in CUDA, network training used Torch with cuDNN convolution routines, and OpenCV handled affine transformations in data augmentation.The cited tools are CUDA, Torch, cuDNN (Chetlur et al., 2014), and OpenCV.Source: §5.3 · p. 19

5.4 Data Set Augmentation

• Training patches were randomly rotated, scaled, sheared, and adjusted for brightness and contrast; applying transforms after extraction preserves ground truth disparity and rectification without affecting runtime.Source: §5.4 · p. 20
• Transformation parameters were randomized for each patch pair and resampled each epoch, with slightly different transformations applied to the left and right patches.Source: §5.4 · p. 20
• Middlebury training used images with varied lighting and exposures, included mismatched lighting or exposure 20 % of the time, and added small vertical disparity; KITTI used the same augmentation parameters across both sets.Source: §5.4 · p. 20
• Validation error fell from 2.73 % to 2.61 % on KITTI 2012 and from 8.75 % to 7.91 % on Middlebury with data augmentation.Source: §5.4 · p. 21

5.5 Runtime

• Training time depended on data-set size and architecture but never exceeded two days.Source: §5.5 · p. 22

5.6 Matching Cost

• The study replaced the CNN with standard matching costs to test whether its low error rate came from the network rather than a superior stereo method.Figures 5, 6, and 7 visually compare the method with the census transform.Source: §5.6 · pp. 22, 24
• The sum of absolute differences matching cost sums intensity differences at corresponding locations in 9 × 9 image patches.Source: §5.6 · p. 22
• The census transform compares each pixel in a centered 9 × 9 patch with its center, encodes the comparisons in an 81-bit vector, and matches vectors by Hamming distance.The method is attributed to Zabih and Woodfill (1994).Source: §5.6 · p. 22
• Normalized cross-correlation computes cosine similarity between left and right patch vectors over an 11 × 11 neighborhood, matching the fast architecture’s final normalization and dot-product layers.Source: §5.6 · p. 22

5.5 Runtime

• On KITTI, Middlebury, and Tiny, the fast architecture’s matching-cost computation was up to 90 times faster than the accurate architecture, taking 0.78, 2.03, and 0.06 seconds, respectively.Table 7’s matching-cost runtimes include computation with each image as reference and exclude post-processing; its last row gives whole-method runtime including post-processing.Source: §5.5 · p. 22, Table 7 · p. 23

5.6 Matching Cost

• Across KITTI 2012, KITTI 2015, and Middlebury, accurate architecture ranked ahead of fast architecture and census, with errors of 2.61%, 3.02%, and 4.90%; 3.25%, 3.99%, and 5.03%; and 7.91%, 9.87%, and 16.72%, respectively.The sum of absolute differences and normalized cross-correlation produced larger errors.Source: §5.6 · p. 24

5.7 Stereo Method

• Removing all post-processing steps raised accurate-architecture validation error from 2.61% to 13.49% on KITTI 2012 and from 3.25% to 13.38% on KITTI 2015.The ablation excluded cross-based cost aggregation, semiglobal matching, interpolation, subpixel enhancement, a median filter, and a bilateral filter.Source: §5.7 · p. 24
• Removing all post-processing steps raised accurate-architecture validation error from 7.91% to 28.33% on Middlebury.The ablation excluded cross-based cost aggregation, semiglobal matching, interpolation, subpixel enhancement, a median filter, and a bilateral filter.Source: §5.7 · p. 24
• Semiglobal matching had the strongest effect among the tested post-processing steps, raising validation error from 2.61% to 4.26% on KITTI 2012 and from 3.25% to 4.51% on KITTI 2015 when removed.Source: §5.7 · p. 24
• Removing semiglobal matching raised validation error from 7.91% to 11.99% on Middlebury.Source: §5.7 · p. 24
• On Middlebury, using the left-right consistency check raised accurate-architecture error from 7.91% to 8.22%, so the method omitted it despite its use for eliminating errors in occluded regions.Source: §5.7 · p. 24

5.8 Data Set Size

• The data-set-size experiment retrained the supervised patch-similarity networks on smaller training sets formed by randomly selecting examples; validation error decreased as training examples increased.Source: §5.8 · p. 24
• The data-set-size experiments suggest collecting a larger data set as a simple strategy for improving the stereo method’s results.Source: §5.8 · p. 24

5.9 Transfer Learning

• The transfer experiments train on one stereo data set and compute validation error on another, approximating settings where ground truth is unavailable for specialized training.Source: §5.9 · pp. 24, 26
• KITTI 2012 validation error is lower with Middlebury training than with KITTI 2015 training, and lower with the fast than the accurate architecture when trained on KITTI 2015.Source: §5.9 · p. 26
• Networks trained on Middlebury transfer well to KITTI, with validation errors similar to those of networks trained on KITTI.For example, the fast architecture trained on Middlebury has a 3.16% validation error when tested on KITTI 2012.Source: §5.9 · p. 26

5.10 Hyperparameters

• The hyperparameter search varies one setting at a time while holding the others at default values, addressing an exponentially growing search space without gradient guidance.Source: §5.10 · p. 26
• Increasing network size improves generalization only up to a point, after which performance begins to decline, presumably because of limited data.Source: §5.10 · pp. 26, 28, Table 11 · p. 27
• The number of convolutional layers also controls patch size: one layer with 3 × 3 kernels compares 3×3 patches, while five layers compare 11 × 11 patches.Source: §5.10 · p. 28

6. Conclusion

• The authors present two convolutional neural network architectures for learning image-patch similarity and applying it to stereo matching.Source: §6 · p. 28
• The implementation repository provides procedures for computing disparity maps, training the network, and running stereo-method post-processing.Source: §6 · p. 28
• The accurate architecture produces lower-error disparity maps than any previously published method on KITTI 2012, KITTI 2015, and Middlebury.The authors describe this result as a demonstration of the power of modern machine-learning approaches on stereo matching.Source: §6 · p. 28
• The fast architecture computes disparity maps up to 90 times faster than the accurate architecture, with only a small increase in error.Source: §6 · p. 28