PaperlayerView exact revision on arXiv

Source-linked AI summary

Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

Jure Žbontar, Yann LeCun

arXiv:1510.05970v2cs.CVcs.LGcs.NE

TL;DR

Stereo matching requires a useful matching cost for estimating disparity from image pairs. This paper learns patch similarity with supervised CNNs, using separate fast and accurate architectures, and reports lower error rates than previously published methods on KITTI 2012, KITTI 2015, and Middlebury. The fast architecture is also substantially faster, with only a small error increase.

  • Problem

    Stereo algorithms need an effective matching cost, a central component of stereo matching used in applications including autonomous driving, robotics, view generation, and 3D reconstruction.

  • Method

    The method trains CNNs on balanced positive and negative patch pairs from known disparities, using two architectures to produce similarity-based matching costs.

  • Results

    The accurate architecture achieves lower error rates than previously published methods on KITTI 2012, KITTI 2015, and Middlebury, while the fast architecture computes disparity maps up to 90 times faster with a small error increase.

  • Takeaways & Limitations

    The results suggest CNNs are well suited to stereo matching costs, including applications requiring real-time performance.

  • Takeaways & Limitations

    On Middlebury, the left-right consistency check was omitted because it increased the accurate architecture’s error rate from 7.91 % to 8.22 %.

Abstract

from arXiv · show

We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.

1. Introduction

The paper frames stereo matching as estimating disparity and depth from rectified image pairs, focusing on learning the matching cost with convolutional neural networks. It contributes two architectures and reports the lowest error rates on three stereo benchmarks.

  • Stereo matching estimates each pixel’s disparity from two horizontally displaced camera images, enabling depth computation.Disparity is the horizontal location difference for corresponding points; closer objects have larger disparities.
  • The work focuses on matching cost computation, the first stage of stereo algorithms, while later stages aggregate, optimize, and refine disparities.The matching cost should distinguish patches depicting the same 3D point from incorrect matches.
  • The authors propose two convolutional neural-network architectures for computing stereo matching costs.The contribution specifically describes architectures based on convolutional neural networks.
  • The method is accompanied by source code and achieves the lowest error rate on KITTI 2012, KITTI 2015, and Middlebury.
  • Experiments examine training-set size, error rates relative to other methods, and the accuracy–runtime trade-off across hyperparameter settings.

2. Related Work

Related work covers learned stereo costs, confidence estimation, graphical-model parameter learning, and local image-descriptor methods that measure patch similarity.

  • The reviewed literature includes methods using ground-truth data to learn model parameters before large stereo datasets became common.
  • Earlier methods learned stereo matching costs or adjusted initial costs using classifiers, graphical models, and structured prediction.
  • Some approaches estimated matching-cost confidence and used those predictions as soft constraints in graphical stereo models.
  • Local image descriptors address a related patch-similarity problem shared with stereo matching cost computation.

3. Matching Cost

The method learns patch similarity from balanced positive and negative examples, then uses two CNN architectures to initialize stereo matching costs. The fast design uses cosine similarity, while the accurate design learns the comparison through fully connected layers.

  • A conventional matching cost compares left-image patches at p with right-image patches at p−d, seeking low cost for corresponding 3D points.
  • 3.1 Constructing the Data Set: Ground-truth disparities create a balanced binary dataset with positive same-point patch pairs and negative mismatched pairs.One positive and one negative example are extracted at each position with known disparity.
  • 3.2.1 Fast Architecture: The fast network uses a siamese shared-weight structure, and its cosine computation is split into normalization and dot product for faster execution.
  • 3.1 Constructing the Data Set: Positive examples include small random disparity offsets because cross-based aggregation benefits from low costs for good and near matches.The offset hyperparameter was never larger than one pixel in the experiments.
  • 3.2 Network Architectures: Both architectures encode each patch as a feature vector, but the fast architecture compares vectors with cosine similarity while the accurate architecture learns comparison layers.
  • 3.2 Network Architectures: Training uses hinge loss for the fast architecture and binary cross-entropy for the accurate architecture based on empirical performance and architectural compatibility.
  • 3.2.2 Accurate Architecture: The accurate architecture replaces cosine similarity with fully connected layers and sigmoid output, increasing runtime while decreasing error rate.
  • 3.3 Computing the Matching Cost: The network output initializes the matching-cost tensor by negating similarity, requiring forward passes across image positions and candidate disparities.

4. Stereo Method

The stereo method improves raw CNN matching costs through adaptive aggregation, semiglobal optimization, consistency checks, interpolation, subpixel refinement, and edge-preserving filtering.

  • 4.1 Cross-based Cost Aggregation: Cross-based cost aggregation adaptively collects support from pixels with similar intensities and bounded distances, avoiding fixed-window assumptions near depth discontinuities.Support regions combine corresponding neighborhoods from both images and are averaged iteratively; the fast architecture skips this step because it is expensive and not crucial for low error.
  • 4.2 Semiglobal Matching: Semiglobal matching enforces disparity smoothness by minimizing the energy along multiple directions and averaging the resulting costs.The implementation uses two horizontal and two vertical directions; diagonal directions did not improve accuracy.
  • 4.2 Semiglobal Matching: Penalty parameters are reduced at strong image gradients so disparity jumps can coincide with image edges.The base penalties are reduced when one or both neighboring image pairs have strong gradients, with an additional vertical-direction adjustment.
  • 4.3 Computing the Disparity Image: The disparity image is obtained by choosing, at each pixel, the disparity with minimum aggregated matching cost.This winner-takes-all estimate follows the final cost aggregation across the four optimized directions.
  • 4.4 Interpolation: Left-right consistency labels pixels as correct, mismatch, or occlusion, then interpolation fills unreliable disparities using neighboring correct pixels.Occlusions use a leftward correct value, while mismatches use the median disparity from nearest correct pixels in 16 directions.
  • 4.5 Subpixel Enhancement and Filtering: Subpixel enhancement fits a quadratic through neighboring costs, followed by 5 × 5 median and bilateral filtering to smooth disparities while preserving edges.The bilateral-filtered disparity map is the final output of the stereo method.

5. Experiments

Experiments evaluate the method on KITTI 2012 and KITTI 2015 using leaderboard error rates, runtimes, and predicted disparity examples. The accurate architecture ranks first on both datasets, while the fast architecture is substantially quicker on KITTI 2012.

  • KITTI Stereo Data Set: KITTI 2012 uses rectified 1240 × 376 image pairs, with laser-scanner ground truth labeling around 30% of pixels.The dataset contains 194 training and 195 testing images.
  • Evaluation: KITTI error is the percentage of pixels whose predicted and true disparities differ by more than three pixels.The corresponding distance tolerance is 3 centimeters at 2 meters and 80 centimeters at 10 meters.
  • KITTI 2012: 2.43% error places the accurate architecture first on KITTI 2012, while the fast architecture achieves 2.82%.The accurate and fast architectures process one image pair in 67 seconds and 0.8 seconds, respectively.
  • KITTI 2015: 3.89% and 4.46% error place the accurate and fast architectures first and second on the KITTI 2015 leaderboard.KITTI 2015 densely labels moving vehicles and evaluates car glass, emphasizing reflective surfaces.
  • Qualitative Results: The accurate architecture makes fewer mistakes than the fast architecture in difficult regions such as textureless walls and asphalt.The KITTI 2012 examples are shown as predicted disparity maps.

5.2 Middlebury Stereo Data Set

Middlebury combines five indoor stereo datasets with dense, precise structured-light disparities and varied lighting conditions. The method ranks first on the reported leaderboard despite imperfect rectification and half-resolution processing.

  • Dataset: Middlebury concatenates five indoor stereo datasets acquired from 2001 through 2014 under controlled lighting, with structured light providing dense, precise disparities.Scenes from 2005, 2006, and 2014 include multiple lighting and exposure settings.
  • Evaluation: The test set contains 15 images, and the online evaluation reports a weighted average error over those images.Only two of the fifteen test images are perfectly rectified.
  • Evaluation: The method runs on half-resolution images because of limited graphics-card memory, then evaluation errors are computed at full resolution after upsampling.The evaluation uses non-occluded pixels by default.
  • Results: 8.29% error ranks the method first on Middlebury, ahead of MeshStereo at 13.4%.The comparison is based on the weighted leaderboard error after the specified evaluation procedure.

5.3 Details of Learning

Learning uses balanced positive and negative patch pairs from ground-truth disparities, trained in mini-batches with architecture-specific optimization settings. Qualitatively, the accurate model handles textureless regions better than the fast model and census.

  • Training Data: The binary classification datasets contain 25 million KITTI 2012, 17 million KITTI 2015, and 38 million Middlebury examples.Examples are constructed from available training image pairs.
  • Training Procedure: Each training batch contains 128 patch pairs, while testing uses the entire left and right image.Patch training permits controlled batch sizes and mixing examples from multiple images.
  • Optimization: Training uses mini-batch gradient descent with momentum 0.9 for 14 epochs, with learning rates of 0.003 for accurate and 0.002 for fast architectures.The learning rate is reduced by a factor of 10 on epoch 11.
  • Qualitative Results: The accurate architecture classifies most of a textureless white wall correctly, whereas the fast architecture performs worse but still better than census.The architecture hyperparameters are listed in Table 5.
  • Implementation: Post-processing runs in CUDA, training uses Torch with cuDNN convolution routines, and OpenCV handles affine augmentation.These implementation choices support the reported training and inference pipeline.

5.4 Data Set Augmentation

Data augmentation applies randomized geometric and photometric transformations to extracted patch pairs without changing disparity labels or rectification. It reduces validation error on both KITTI 2012 and Middlebury.

  • Transformations: Augmentation randomly rotates, scales, shears, and changes brightness and contrast of training patches.The left and right patches receive slightly different transformation parameters.
  • Transformations: Applying transformations after patch extraction preserves the ground-truth disparity map and rectification.The transformations therefore alter training examples without changing their labels.
  • Dataset-Specific Augmentation: On Middlebury, training uses images across lighting conditions and shutter exposures, including mismatched left-right illumination 20% of the time.This targets cases such as an underexposed right image or differing light conditions.
  • Dataset-Specific Augmentation: A small vertical disparity is included during training to address imperfect Middlebury rectification.Standard calibration can produce vertical disparity errors of up to nine pixels.
  • Results: Data augmentation reduces validation error from 2.73% to 2.61% on KITTI 2012 and from 8.75% to 7.91% on Middlebury.The validation effects of individual augmentation steps are reported in Table 6.

5.5 Runtime

The section compares runtime and matching-cost accuracy across architectures and standard baselines. The fast architecture substantially reduces runtime, while the accurate architecture achieves the lowest reported errors.

  • The runtime measurements cover KITTI, Middlebury at half resolution, and the Tiny data set across specified image and disparity dimensions.The measured image sizes are 1242 × 350 for KITTI, 1500 × 1000 for Middlebury, and 320 × 240 for Tiny.
  • The fast architecture is up to 90 times faster than the accurate architecture.Its runtime is 0.78 seconds on KITTI, 2.03 seconds on Middlebury, and 0.06 seconds on Tiny.
  • The accurate architecture performs best on all three data sets, followed by the fast architecture and census transform.Their error rates are 2.61%, 3.02%, and 4.90% on KITTI 2012; 3.25%, 3.99%, and 5.03% on KITTI 2015; and 7.91%, 9.87%, and 16.72% on Middlebury.
  • Sum of absolute differences and normalized cross-correlation produce disparity maps with larger errors than the leading methods.

5.7 Stereo Method

The section evaluates stereo-method post-processing by removing individual steps and measuring validation error. Post-processing is crucial, with semiglobal matching producing the strongest effect among the tested steps.

  • The tested stereo method includes cross-based cost aggregation, semiglobal matching, interpolation, subpixel enhancement, median filtering, and bilateral filtering.
  • Removing all post-processing increases accurate-architecture error from 2.61% to 13.49% on KITTI 2012, from 3.25% to 13.38% on KITTI 2015, and from 7.91% to 28.33% on Middlebury.
  • Semiglobal matching affects validation error most strongly among the post-processing steps.

5.9 Transfer Learning

The section examines transfer across training and validation data sets and explores how network choices affect generalization. Middlebury-trained networks transfer well to KITTI, while larger networks improve performance only up to a point.

  • 5.9 Transfer Learning: A Middlebury-trained fast architecture achieves 3.16% validation error when tested on KITTI 2012.
  • 5.9 Transfer Learning: Some transfer results are unexpected: KITTI 2012 error is lower with Middlebury training than KITTI 2015 training, and lower with the fast architecture in one comparison.
  • 5.9 Transfer Learning: Middlebury-trained matching-cost networks transfer well to KITTI, with validation errors similar to those from KITTI-trained networks.
  • 5.10 Hyperparameters: Increasing network size improves generalization performance only up to a point, after which performance starts to decrease.
  • 5.10 Hyperparameters: The number of convolutional layers also controls the compared image-patch size, ranging from 3×3 with one layer to 11×11 with five layers.

6. Conclusion

The paper presents two CNN architectures that learn image-patch similarity for stereo matching and provides implementation code. The accurate architecture achieves the lowest reported errors, while the fast architecture offers much faster computation with a small accuracy cost.

  • The paper presents two convolutional neural-network architectures for learning a similarity measure on image patches and applying it to stereo matching.
  • The implementation source code includes procedures for computing disparity maps, training the network, and performing stereo-method post-processing.
  • The accurate architecture produces lower error rates than previously published methods on KITTI 2012, KITTI 2015, and Middlebury.
  • The fast architecture computes disparity maps up to 90 times faster than the accurate architecture with only a small increase in error.
  • The results suggest that CNNs are suited to stereo matching-cost computation in applications requiring real-time performance.
Loading 1510.05970v2…