PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Training of Hybrid CNN-CRF Models for Stereo

Patrick Knöbelreiter, Christian Reinbacher, Alexander Shekhovtsov, Thomas Pock

arXiv:1611.10229v2cs.CV

TL;DR

Stereo estimation requires reconstructing depth from image pairs, while prior deep CNN methods often relied on post-processing for accurate outputs. This paper trains a hybrid CNN-CRF end-to-end and reports competitive benchmark performance with shallow CNNs and no post-processing.

  • Problem

    Deep CNN stereo methods commonly rely on post-processing to produce accurate results.

    Source: §1 · p. 1

  • Method

    The paper combines CNN-computed unary and pairwise costs with CRF inference, training the complete model using an SSVM formulation and propagated subgradients.

    Source: §1 · pp. 1–2

  • Results

    The hybrid model achieves competitive benchmark performance with shallow CNNs and no post-processing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that the CRF reduces the generalization gap, allowing competitive performance with fewer parameters and a more compact model.

    Source: §1 · p. 2, §6 · p. 7

    The model has no explicit occlusion label; the authors identify adding one and exploring continuous labels for slanted surfaces as future work.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

We propose a novel and principled hybrid CNN+CRF model for stereo estimation. Our model allows to exploit the advantages of both, convolutional neural networks (CNNs) and conditional random fields (CRFs) in an unified approach. The CNNs compute expressive features for matching and distinctive color edges, which in turn are used to compute the unary and binary costs of the CRF. For inference, we apply a recently proposed highly parallel dual block descent algorithm which only needs a small fixed number of iterations to compute a high-quality approximate minimizer. As the main contribution of the paper, we propose a theoretically sound method based on the structured output support vector machine (SSVM) to train the hybrid CNN+CRF model on large-scale data end-to-end. Our trained models perform very well despite the fact that we are using shallow CNNs and do not apply any kind of post-processing to the final output of the CRF. We evaluate our combined models on challenging stereo benchmarks such as Middlebury 2014 and Kitti 2015 and also investigate the performance of each individual component.

1. Introduction

The paper combines learned CNN matching and pairwise costs with CRF inference, then trains the complete model end-to-end. It reports competitive results with shallow networks and no post-processing.

1. Introduction

  • Earlier stereo approaches used engineered matching costs and experimentally selected parameters, while deep CNN stereo methods commonly relied on post-processing to obtain accurate outputs.The paper frames its hybrid approach against both global optimization methods and deep CNN pipelines.Source: §1 · p. 1
  • The model replaces hand-crafted CRF terms with learned unary and pairwise costs, combining complex local matching with parameterized geometric priors in global optimization.The Unary-CNN features are compared by a fixed correlation metric, while the Pairwise-CNN can estimate edge-sensitive costs that encourage or discourage label jumps.Source: §1 · p. 1
  • The learned pairwise terms generalize contrast-sensitive regularizers, while the unary network generalizes manually designed matching costs such as color differences and Census transforms.The CRF integrates long-range interactions.Source: §1 · p. 2
  • The authors use a fast CRF inference method instead of post-processing heuristics, reporting speed competitive with SGM and MGM.They deliberately omit post-processing to assess how much of its performance gain a trained CRF can cover.Source: §1 · p. 2
  • Complete-image SSVM training propagates subgradients through the CNNs and targets inference performance on whole training images.The approach uses approximate subgradients because exact inference is unavailable and fixed-iteration inference is used at test time.Source: §1 · p. 2
  • Shallow 3–7-layer hybrids perform very well, with a smaller generalization gap than models without the CRF, and achieve competitive results without post-processing.The model is often reported to delineate object boundaries accurately and to be robust to occlusions despite lacking explicit occlusion modeling.Source: §1 · p. 2

2. Related Work

Related work spans CNN stereo matching, general-purpose matching networks, and learned inference. The paper positions whole-image CNN-CRF training with approximate subgradient descent as a response to limitations in prior approaches.

2. Related Work

  • CNN stereo systems use siamese feature extraction and fixed correlation, but train on sampled patches and rely on extensive post-processing.The paper instead trains on complete images to support end-to-end optimization for dense matching with a CRF.Source: §2 · p. 2
  • General-purpose matching networks have been applied to patch matching [52], optical flow, and stereo, optical flow, and scene flow [29], often with larger models and additional synthetic data.Some use correlation layers followed by up-convolutions to regress dense matching.Source: §2 · p. 2
  • The model was considered unsuitable for stereo because typical solutions contain slanted rather than piece-wise constant surfaces, and its filtering propagates information in fronto-parallel planes.Source: §2 · p. 3
  • For LP-relaxation inference, blended learning and inference [11] jointly optimizes model parameters and dual variables, but the paper finds it impractical at this scale because of memory and time.The authors instead use stochastic approximate subgradient descent; fast GPU inference [44] helps address the speed constraint.Source: §2 · p. 3

3. CNN-CRF Model

The model combines learned image features and pairwise costs in a CRF that optimizes complete disparity labelings. Its CNN components provide matching confidences and image-adaptive edge weights.

3.1. Unary CNN

  • The shared Unary-CNN extracts dense features from both images for matching at candidate disparities.Its two image branches share parameters.Source: §3 · p. 3

3.2. Correlation

  • The correlation layer softmax-normalizes feature-vector dot products across disparities into matching confidences, and supports testing with a different disparity count.The normalization fixes the unary-cost scale for joint training.Source: §3.2 · p. 3

3.3. CRF

  • Dual MM decomposes the CRF costs into horizontal and vertical chains, then updates their multipliers in parallel to monotonically tighten a lower bound.The resulting labeling is optimal if inference converges and each pixel’s minimizer is unique.Source: §3.3 · p. 4

3.4. Pairwise CNN

  • The Pairwise-CNN learns positive, image-adaptive edge weights from a wider neighborhood, generalizing manually defined contrast-sensitive weights.Figure 2 shows learned low-cost label changes following object outlines while suppressing texture edges.Source: §3.4 · p. 4, Figure 2 · p. 4

4. Training

The paper compares models of increasing generality and trains the joint CNN-CRF using an SSVM-based approach. Training uses approximate inference to obtain subgradients, with implementation details including full-image processing and a reported streaming speed.

4. Training

  • Three model variants compare a pixel-wise Unary-CNN, a Unary-CNN with fixed contrast-sensitive CRF costs, and a full model with a trained Pairwise-CNN.The variants train progressively more components, from unary-only to unary and pairwise CNNs plus global parameters P1 and P2.Source: §4 · p. 4

4.1. Training Unary CNN in the Pixel-wise Model

  • The pixel-wise Unary-CNN is trained with cross-entropy after setting CRF interactions to zero, reducing prediction to independent per-pixel disparity choices.The target uses a one-hot encoding of the ground-truth disparity.Source: §4.1 · pp. 4–5

4.2. Training Joint Model

  • Joint-model training uses SSVM loss with approximate inference, replacing exact-inference subgradients with those from a fixed number of Dual MM iterations.The approximation relies on most pixels eventually agreeing between the chain solutions; unary-cost initialization can improve initial agreement and stabilize training.Source: §4.2 · pp. 5–6

4.3. Training Unary and Pairwise CNNs in Joint Model

  • For trainable pairwise costs, the method computes subgradients for edge weights and P1/P2 using an approximate rather than exact inference minimizer, yielding an end-to-end trainable model.The model has no hand-crafted parameters except training hyperparameters.Source: §4.3 · p. 6

4.4. Implementation Details

  • The fully trained three-layer model runs at 3–4 frames per second on 640 × 480 inputs in a streaming setting.Training uses full RGB images, masks occlusions and invalid pixels, and gradually reuses parameters from lower-complexity models.Source: §4.4 · p. 6

5. Experiments

Experiments compare CNN/CRF variants across Kitti 2015 and Middlebury V3; x in CNNx denotes the number of CNN layers.

5.1. Benchmark Data Sets

  • Kitti 2015 and Middlebury V3 design sets were split 80/20, yielding 160+40 and 122+31 examples, respectively; experiments use badx, the percentage above an x-pixel disparity error.The test-set ground truth was unavailable to the authors.Source: §5.1 · p. 6

5.2. Performance of Individual Components

  • Direct CNN outputs contained too many outliers, while CRF variants achieved good results with shallow CNNs; jointly trained models correctly inpainted occlusions.CNN-only quality depended strongly on layer count, unlike the CNN+CRF versions.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3. Benefits of Joint Training

  • After CRF post-processing narrowed the gap to MC-CNN and ContentCNN, joint training further improved performance; the 7-layer model had 281k parameters versus about 700k and 830k.The parameter counts compare the full joint model with the networks of, respectively.Source: §5.3 · p. 7

5.4. Benchmark Test Performance

  • On Middlebury, CNN7+PW achieved the lowest overall RMS error but slightly worse bad2; Kitti error plots placed most incorrect predictions in occluded areas, without post-processing.The Middlebury model was fine-tuned for half-sized images; the Kitti model included pixel x- and y-coordinates as features.Source: §5.4 · p. 7

6. Conclusion

The paper concludes that a fully trainable, post-processing-free CNN+CRF can achieve competitive stereo performance with fewer parameters. Joint training learns unary and pairwise costs; the authors identify occlusion handling and continuous labels as directions for improvement.

6. Conclusion

  • The clean, fully trainable CNN+CRF achieves competitive performance with significantly fewer parameters and without post-processing.The authors say this design also makes the learned components more interpretable.Source: §6 · p. 7
  • The qualitative comparisons show Middlebury disparity and bad2 error images, Kitti error-threshold images, and a zoomed depth-overlay/error-plot comparison.Source: Figure 4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
  • Joint training learns unary and pairwise costs, and the authors report that the increased generality improves performance.Source: §6 · p. 8
  • The trainable pairwise terms allow more accurate object-boundary delineation.Source: §6 · p. 8
  • The authors propose adding an occlusion label and exploring continuous labels to improve performance in occluded areas and on slanted surfaces.Source: §6 · p. 8

Supplementary Material

The appendix derives an approximate subgradient for the loss upper bound by fixing minimizers across its nested optimization steps. Its approximation neglects a term based on an assumption about distant pixels' influence on the dual variables.

A.1. Approximate Subgradient

  • Fixing minimizers in the outer optimization and the dynamic-programming minima yields a subgradient of the resulting loss upper bound.Source: §A.1 · p. 11
  • The derivation drops pairwise and loss terms that do not depend on the unary costs, leaving the unary-cost contribution.Source: §A.1 · p. 11
  • The approximation neglects a sum because changes to a pixel's unary cost far from a node are expected to have little effect on that node's dual variable.Source: §A.1 · p. 11

B. Training insights

The training plot tracks bad4 error on Middlebury for the seven-layer model through three training stages. The train–validation error gap is significantly smaller after joint optimization.

B. Training insights

C. Additional Experiments

The timing experiments report runtime for individual method components across image sizes and disparity counts. With fixed Unary-CNN layers and disparity range, runtime scales linearly with input pixel count; correlation and CRF runtime also depend on disparity count.

C.1. Timing

  • With Unary-CNN layer count and disparity range held constant, runtime depends linearly on input image pixel count.Source: §C.1 · p. 11
  • Correlation and CRF runtime also depend on the number of estimated disparities; the reported settings use 128 and 256 disparities.Source: §C.1 · p. 11
  • The timing tests use Kitti 2015 at 1242 × 375 and half-resolution Middlebury Jadeplant at 1318 × 994 on a Linux PC with a GTX TitanX GPU.Source: §C.1 · p. 11

C.2. Sublabel Enhancement

The discrete CRF solution can suffer Middlebury discretization artifacts, so the authors investigate sub-label refinement by fitting a quadratic around the discrete solution. Their qualitative example shows smoother transitions after refinement, while the main reported performance remains that of the discrete solution.

C.2. Sublabel Enhancement

  • Discrete CRF disparities can hurt Middlebury quantitative performance, motivating a local quadratic fit to the cost volume around each discrete solution.Source: §C.2 · pp. 11–12
  • Timing experiments compare a 7-layer CNN with five CRF iterations against a real-time configuration using three layers and four iterations.Source: Table C.1 · p. 12
  • On Motorcycle, the sub-label-enhanced result has smoother transitions, including at the floor and rear wheel.Source: Figure C.1 · p. 12
  • The refined disparity is computed from the costs at d-h, d, and d+h around the discrete estimate d.Source: §C.2 · p. 12
  • Quantitative experiments cover KITTI 2015 and Middlebury, but the main paper and submitted images report the discrete solution.Source: §C.2 · p. 12

C.3. Middlebury Stereo v3

On Middlebury Stereo v3, the authors report improving quality as model complexity increases, while the jointly trained 3-layer model looks nearly like the 7-layer model. They also find asymmetric cross-dataset transfer: KITTI-trained performance on Middlebury is inferior, whereas Middlebury-to-KITTI generalization is better.

C.3. Middlebury Stereo v3

  • Middlebury quality increases as more parameters allow a more general model; results use quarter-resolution predictions upsampled to original size.Source: §C.3 · p. 12
  • The jointly trained 3-layer model is visually difficult to distinguish from the 7-layer model, making the smaller model suitable for real-time use.Source: §C.3 · p. 12
  • The Vintage image remains difficult because its nearly untextured top-left region leaves some disparities unrecovered even with the full model.Source: §C.3 · p. 12
  • The KITTI-trained model performs worse on Middlebury, while generalizing from Middlebury to KITTI is reported as much better.Source: §C.3 · p. 12

C.4. Kitti 2015

On the KITTI 2015 design set, the best Middlebury model’s error increases by only about 1.5% on all pixels. Qualitative results show sharp object boundaries, although occlusions and fattened ground truth account for many errors.

C.4. Kitti 2015

  • ≈1.5%: the best Middlebury model’s performance decreases by this amount on all pixels when applied to the KITTI design set.Source: §C.4 · p. 12, Table C.3 · p. 16
  • The qualitative KITTI test-set comparisons include additional results beyond those shown in the main paper.Source: §C.4 · p. 12, Figure C.4 · p. 15
  • Many KITTI errors occur in occluded regions or where the ground truth is fattened, while edge-weight learning produces sharp predictions at object boundaries.Source: §C.4 · p. 12
  • Figure C.3 compares color-coded disparity maps and error plots for Ours, MC-CNN, ContentCNN, and DispNetC; blue marks correct and orange marks wrong.Source: Figure C.3 · p. 14

Related papers

Loading 1611.10229v2…