PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning

Amit Shaked, Lior Wolf

arXiv:1701.00165v1cs.CV

TL;DR

Stereo matching requires accurate disparity estimates and reliable confidence for refinement. The paper introduces a constant-gated highway matching network, a CNN-based disparity and confidence stage, and reflective loss, reporting improved benchmark accuracy and confidence performance.

  • Problem

    Stereo matching pipelines need to compute matching costs and refine disparity predictions while reliably assessing prediction correctness.

  • Method

    The paper combines a multilevel constant-gated highway matching network with hybrid loss, CNN-based disparity prediction replacing WTA, and reflective confidence learning.

  • Results

    The accurate method achieves 2.29 and 3.42 error rates on KITTI 2012 and KITTI 2015, improving MC-CNN baselines of 2.43 and 3.89.

  • Takeaways & Limitations

    The learned confidence is used to detect and correct outliers during disparity refinement, and the paper reports state-of-the-art benchmark accuracy.

  • Takeaways & Limitations

    The method does not use semantic segmentation, which uses additional training data and runtime; the authors leave possible complementarity for future study.

Abstract

from arXiv · show

We present an improved three-step pipeline for the stereo matching problem and introduce multiple novelties at each stage. We propose a new highway network architecture for computing the matching cost at each possible disparity, based on multilevel weighted residual shortcuts, trained with a hybrid loss that supports multilevel comparison of image patches. A novel post-processing step is then introduced, which employs a second deep convolutional neural network for pooling global information from multiple disparities. This network outputs both the image disparity map, which replaces the conventional "winner takes all" strategy, and a confidence in the prediction. The confidence score is achieved by training the network with a new technique that we call the reflective loss. Lastly, the learned confidence is employed in order to better detect outliers in the refinement step. The proposed pipeline achieves state of the art accuracy on the largest and most competitive stereo benchmarks, and the learned confidence is shown to outperform all existing alternatives.

1. Introduction

The paper improves stereo matching with a multilevel constant-gated highway matching network, a CNN that replaces WTA disparity selection, and reflective confidence learning for refinement. It reports improved KITTI accuracy and fast-architecture results.

  • The pipeline uses a deep matching network plus post-processing to incorporate spatial information, verify plausibility, and selectively smooth and refine disparities.
  • A multilevel highway architecture with constant gating and scaling layers improves the deep matching network over existing residual alternatives.
  • A second network replaces WTA and outputs both the predicted disparity and a confidence score.
  • Reflective loss dynamically labels confidence as 1 for correct predictions and 0 otherwise during training.
  • 2.29 and 3.42 error rates improve the MC-CNN baseline values of 2.43 and 3.89 on KITTI 2012 and KITTI 2015.

2. Related work

Prior stereo systems combine CNN matching with heuristic refinement, while related work explores residual architectures and CNN-based confidence estimation. This paper differs through constant learned highway gates, reflective confidence training, joint disparity-confidence prediction, and confidence-guided outlier correction.

  • Earlier stereo pipelines combined CNN matching costs with heuristic spatial refinement, while later methods added faster computation, semantic segmentation, or adaptive smoothness.
  • Residual networks achieve strong vision results but are not comparably successful for stereo matching.
  • The proposed architecture differs from concurrent residual designs by using learned constant highway gates and scaling layers controlling the receptive field.
  • Unlike prior CNN confidence work, one network produces both a disparity map and confidence, trains confidence with dynamically changing reflective labels, and uses it for outlier correction.
  • The reflective loss is presented as a general confidence-evaluation technique rather than one bounded to stereo matching.

3. Computing the matching cost

The matching-cost stage uses tied two-tower descriptors, multilevel constant highway residual blocks, and a hybrid loss combining decision-network cross-entropy with descriptor hinge loss. The architecture balances accuracy, receptive-field control, and runtime flexibility.

  • Computing the matching cost: For each image position and disparity, the network compares corresponding left and right patches, assigning low cost to patches from the same 3D location.
  • Inner-Outer Residual blocks: The description network uses inner and outer constant highway residual blocks, scaling layers, and preserved spatial resolution instead of pooling with batch normalization.
  • Two-tower matching: Two tied description networks extract descriptors, then a decision pathway concatenates them for cross-entropy training while a second pathway applies hinge loss to their dot product.
  • Runtime: At full-image scale, descriptor maps are computed once, but each candidate disparity requires a decision-subnetwork pass and dominates runtime.
  • Runtime: The fast dot-product pathway trades accuracy for speed, while its training term improves matching performance despite being unused at runtime.
  • Constant highway skip connection: A learned λ weights each residual identity shortcut, replacing the input-dependent highway gate with a constant parameter.
  • Constant highway skip connection: In the unrolled outer block, λ2 controls f1, λ1 balances the input to f2, and λ0 controls the original input path.
  • Hybrid loss: The hybrid loss combines α-weighted cross-entropy with (1 − α)-weighted hinge loss over decision outputs and descriptor similarities.

4. Computing the disparity image

The global disparity network replaces winner-takes-all disparity selection with a CNN that uses the full matching-cost map to predict disparities and confidence. It is trained with disparity and confidence objectives, including reflective learning whose targets change with prediction correctness.

  • 4.1. Global Disparity Network: The global disparity network propagates the full matching-cost map to predict a disparity at every image position instead of applying WTA directly.Its input consists of processed matching-cost maps after CBCA and SGM, while Tanh maps values to [−1, 1].
  • 4.1. Global Disparity Network: FC3 produces disparity scores for possible disparities, while FC5 produces a separate confidence output.LogSoftMax is applied to FC3 for predicted disparities, and FC5 depicts confidence.
  • 4.1. Global Disparity Network: The disparity objective uses weighted cross-entropy with a smooth target distribution centered on the ground-truth disparity.For KITTI, the weighting assigns λ1 = 0.65, λ2 = 0.25, and λ3 = 0.1 for errors within one, two, and three pixels, respectively.
  • 4.2. Reflective confidence: A binary classifier is trained alongside disparity prediction to estimate confidence from the score vector.Training labels are positive when argmax_i y_i differs from the ground truth by less than one pixel.
  • 4.2. Reflective confidence: Reflective loss makes confidence targets depend on both ground truth and the network’s current activations, changing dynamically during training.The reflective loss is combined with the weighted cross-entropy loss using 15:85 weights, respectively.

5. Disparity refinement

The refinement stage detects unreliable disparities using left-right consistency together with learned confidence, then replaces mismatches and occlusions using different neighboring-pixel interpolation rules.

  • 5. Disparity refinement: Refinement begins with a left-right consistency check for outlier detection that incorporates the learned confidence score.The process compares left- and right-reference disparity predictions and their confidence scores.
  • 5. Disparity refinement: A pixel is considered correct when the two predictions agree or when disagreement is outweighed by the greater reliability of the reference prediction.Otherwise, the procedure distinguishes mismatches from occlusions using alternative disparity consistency.
  • 5. Disparity refinement: Mismatches are replaced with the median disparity of nearby correct pixels gathered from 16 directions.This uses reliable neighboring pixels rather than retaining the detected outlier.
  • 5. Disparity refinement: Occlusions are interpolated by moving left to the first correct pixel and using that pixel’s disparity value.The method treats occlusion outliers as likely background pixels.

6. Experimental results

The pipeline is evaluated on KITTI 2012, KITTI 2015, and Middlebury using benchmark comparisons, runtime-constrained submissions, validation experiments, component analysis, and confidence-measure evaluation. It ranks first on both KITTI benchmarks, improves Middlebury validation error, and shows that its global disparity network and constant highway network provide dataset-dependent gains while reflective confidence performs best overall.

  • Benchmark results: The accurate method ranks first on KITTI 2012 and KITTI 2015, reducing mc-CNN error from 2.43 to 2.29 and from 3.89 to 3.42, respectively.KITTI error is the percentage of pixels whose predicted and true disparities differ by more than three pixels.
  • Benchmark results: The fast architecture is compared with methods running under five seconds, with runtime measured on an NVIDIA Titan X Pascal GPU.The paper reports under-five-second comparisons for both KITTI benchmarks.
  • Benchmark results: 8.01 validation error is achieved on Middlebury with λ-ResMatch, improving on the 8.18 error from trained MC-CNN.The experiments use half-resolution, imperfectly rectified image pairs; the accurate architecture's reported 7.91 error was not reproduced.
  • Components Analysis: The global disparity network provides the greatest KITTI 2015 improvement, whereas the constant highway network provides the greatest KITTI 2012 improvement.The paper attributes the KITTI 2015 difference to densely labeled moving vehicles and inclusion of car glass, which make reflective regions more common.
  • Components Analysis: λ-ResMatch achieves the best stereo-matching accuracy across all evaluated data sets, and the hybrid loss further improves its results.The architecture is compared with MC-CNN, conventional highway networks, ResNets, densely connected residual networks, and residual networks of residual networks.
  • Confidence evaluation: The reflective confidence measure has the highest overall AUC on both KITTI data sets and performs better on almost every KITTI 2015 validation image.It is compared with six widely used confidence techniques on 40 random validation images from KITTI 2012 and KITTI 2015.

7. Discussion

The discussion identifies scope boundaries and open questions: the method omits semantic segmentation, while the domain-specific behavior of matching architectures remains incompletely understood. It also suggests reflective loss may extend beyond stereo matching.

  • The method omits semantic segmentation, which uses additional training data and incurs additional runtime.
  • The favorable behavior of moderate-depth, batch-normalization-free, specialized residual architectures remains insufficiently explained.
  • Reflective loss may extend to other problems and gradual learning schemes such as self-paced and curriculum learning.

A. The benefit of color

Unlike prior matching architectures, the authors find that color information improves disparity prediction after deepening their network, especially where object and background colors differ subtly.

  • Color information improves disparity prediction around areas with delicate color differences between objects and their backgrounds.
  • This finding contrasts with prior matching architectures, which reported no benefit from color information.

B. Multilevel constant highway connections

The constant highway connections determine how strongly subnetworks with different depths contribute to the matching description. Training increasingly favors skipping upper blocks while retaining lower-level processing.

  • The input added to each outer block is determined by λ0 + (λ1 · λ2), controlling that block’s contribution to the path ensemble.
  • After full training, deeper outer blocks receive higher skip values than shallow blocks.
  • Training leaves low-level blocks rarely skipped while increasingly skipping upper blocks.

C. Reflective confidence

On 40 random KITTI 2012 validation images, the proposed reflective confidence measure performs better than the comparison measures on almost every image.

  • Reflective confidence performs better than the compared measures on almost every image.The comparison uses 40 random validation images from KITTI 2012.

D. Runtime

Runtime is measured for disparity-map computation on KITTI 2012 image pairs using a single NVIDIA Titan X (Pascal). The fast pipeline is substantially quicker than the accurate pipeline, with parallelization offering further reductions.

  • Evaluation setup: The runtime evaluation uses KITTI 2012 images sized 1242 × 350 with 228 possible disparities.Measurements were obtained on a single NVIDIA Titan X (Pascal) GPU.
  • Pipeline differences: The fast and accurate pipelines differ in description-network depth, decision-network computation, and whether cost aggregation is performed.The fast description sub-network has four outer blocks instead of five, uses a dot product in the decision sub-network, and omits cost aggregation.
  • Runtime comparison: 48 seconds for the accurate method versus 2.84 seconds for the fast method in total runtime.These measurements concern one image-pair disparity-map computation.
  • Bottlenecks and parallelization: Parallel descriptor computation can reduce the fast method’s runtime contribution from descriptor extraction, while parallel disparity passes can reduce up to 90 percent of the accurate method’s runtime.Descriptor computation accounts for 40 percent of the fast method’s runtime; disparity passes add another 11 percent for the fast method.
Loading 1701.00165v1…