PaperlayerView exact revision on arXiv

Source-linked AI summary

Single View Stereo Matching

Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, Liang Lin

arXiv:1803.02612v2cs.CV

TL;DR

Monocular depth methods often directly regress depth from one image without explicitly enforcing geometry, making the task difficult and potentially data-intensive. This paper instead synthesizes a right view and applies stereo matching in an end-to-end pipeline, outperforming prior monocular methods and stereo block matching on KITTI with limited real training data.

  • Problem

    Single-view depth inference lacks explicit geometric constraints and can require large amounts of expensive ground-truth depth data.

  • Method

    The method reformulates monocular depth estimation as view synthesis followed by stereo matching, with geometric transformations encoded in both networks and end-to-end training.

  • Results

    The model outperforms previous monocular methods and stereo block matching on KITTI while using only a small number of real training examples, and generalizes to other monocular benchmarks.

  • Takeaways & Limitations

    Decomposing monocular depth estimation into geometrically constrained view synthesis and stereo matching provides a simple, effective alternative to direct depth regression.

Abstract

from arXiv · show

Previous monocular depth estimation methods take a single view and directly regress the expected results. Though recent advances are made by applying geometrically inspired loss functions during training, the inference procedure does not explicitly impose any geometrical constraint. Therefore these models purely rely on the quality of data and the effectiveness of learning to generalize. This either leads to suboptimal results or the demand of huge amount of expensive ground truth labelled data to generate reasonable results. In this paper, we show for the first time that the monocular depth estimation problem can be reformulated as two sub-problems, a view synthesis procedure followed by stereo matching, with two intriguing properties, namely i) geometrical constraints can be explicitly imposed during inference; ii) demand on labelled depth data can be greatly alleviated. We show that the whole pipeline can still be trained in an end-to-end fashion and this new formulation plays a critical role in advancing the performance. The resulting model outperforms all the previous monocular depth estimation methods as well as the stereo block matching method in the challenging KITTI dataset by only using a small number of real training data. The model also generalizes well to other monocular depth estimation benchmarks. We also discuss the implications and the advantages of solving monocular depth estimation using stereo methods.

1. Introduction

Monocular depth estimation is difficult because single-view inference is ill-posed, geometrically ambiguous, and commonly relies on costly real depth data. The paper reformulates it as view synthesis followed by stereo matching, enabling geometric reasoning and end-to-end training.

  • Motivation: Single-view depth estimation is ill-posed and geometrically ambiguous, despite the practical advantages of using one camera.Monocular cameras avoid calibration errors and synchronization problems associated with stereo cameras.
  • Motivation: Current deep models directly relate high-level semantic information to absolute depth without built-in geometric priors.The formulation makes learning difficult even when special constraints are added to the loss.
  • Motivation: Large amounts of expensive real images with ground-truth depth are needed to establish the relationship between scene understanding and depth.This data requirement limits the potential of the current formulation.
  • Approach: The proposed pipeline decomposes monocular depth estimation into view synthesis and stereo matching, with both stages respecting geometric principles.The right view is automatically generated by a view synthesis network before stereo matching.
  • Results: The whole pipeline can be trained end-to-end without expensive real depth data and generalizes to other monocular depth datasets.The paper reports improved performance on KITTI using only a small number of real training examples.

2. Related Works

Related work spans monocular, stereo, multi-view, and temporal depth estimation, with deep networks increasingly used for single-image prediction. Unsupervised methods reduce depth-label requirements, but still directly regress depth or disparity without architectural geometric constraints.

  • Depth estimation methods: Depth estimation research includes single-view, stereo-view, multi-view, and temporal-sequence approaches.The related literature covers several image acquisition settings for estimating depth.
  • Monocular depth estimation: Early monocular methods used supervised learning, Markov random fields, hand-crafted texture features, and later coarse-to-fine or deeper CNN architectures.These methods established single-image depth prediction before the paper’s stereo-based formulation.
  • Unsupervised methods: Unsupervised monocular methods supervise image alignment or consistency losses to reduce reliance on expensive ground-truth depth.Some approaches require approximations such as Taylor expansion to make the loss differentiable.
  • Paper positioning: Despite reducing label requirements, unsupervised methods still directly regress depth or disparity from one image without geometric constraints in the architecture.The paper addresses this limitation by synthesizing a corresponding right view and then performing stereo matching.
  • View synthesis: Novel-view synthesis methods generate unseen or corresponding views using pixels from other views, appearance flow, or disparity distributions.The related methods motivate generating a right view from a monocular input.
  • Stereo matching: Stereo matching methods search for corresponding pixels or directly regress disparity from stereo pairs using convolutional architectures.The proposed method conducts stereo matching between the original left image and its synthetic right view.

3. Analysis and our approach

The approach decomposes monocular depth estimation into view synthesis followed by stereo matching, combining geometric structure with end-to-end learning. A probabilistic synthesis module creates a right view, and stereo matching estimates disparity that can be converted to depth.

  • The pipeline separates monocular depth estimation into view synthesis and stereo matching, with both stages designed around geometric principles.The two networks can then be jointly fine-tuned end-to-end.
  • View synthesis network: The view synthesis network predicts probabilistic disparities and selectively combines shifted pixels from the left image to reconstruct a synthetic right view.This probabilistic selection is differentiable and avoids requiring an already accurate disparity map for reconstruction.
  • View synthesis network: The selection module replaces direct image warping with a probabilistic summation over possible disparity shifts, preserving differentiability throughout reconstruction.The probabilistic disparity tensor has dimensions W × H × C, where C denotes possible disparity shifts.
  • Training: A simple L1 loss on reconstructed appearance supervises view synthesis, while the two pretrained networks are subsequently combined for joint training.The stereo matching sub-network contributes to end-to-end collaboration between the components.
  • Stereo matching network: The stereo matching network compares the original left image with the synthetic right view to estimate accurate disparity, which yields depth using camera settings.The architecture uses horizontal 1D correlation to encode geometric relationships between corresponding pixels.

4. Experiments

The experiments evaluate the method on the KITTI monocular depth benchmark using the Eigen split and standardized cropped evaluation regions. Results are reported under two depth caps to support comparisons with prior methods.

  • Results: The experiments report state-of-the-art monocular depth estimation results on KITTI and include an evaluation on the challenging KITTI Stereo 2015 benchmark.The paper describes this as its first attempt to run the single-view approach on KITTI Stereo 2015.
  • Dataset: The method is evaluated on KITTI, using the Eigen split with 697 test images and the remaining data for training and validation.The view synthesis network is trained with 22,600 stereo pairs.
  • Evaluation metrics: Evaluation metrics measure error and performance on predicted monocular depth, using only pixels with non-empty ground-truth depth.The metric definitions include ARD, SRD, and thresholded accuracy.
  • Evaluation protocol: Results are evaluated on the Eigen cropped region under both 0-80m and 1-50m depth caps to match prior evaluation protocols.Pixels outside the selected depth range are discarded.

4.2. Implementation Details

Training uses separate first-stage optimization for view synthesis and stereo matching, followed by end-to-end fine-tuning with a small set of labeled KITTI disparities. Data augmentation is optional in both stages.

  • Training stages: The model is trained in two stages: separate pretraining of both networks, followed by end-to-end fine-tuning of the combined pipeline.The implementation uses the Caffe framework.
  • Stage one: View synthesis pretraining uses 22,600 KITTI stereo pairs with VGG16 initialized from ImageNet-pretrained weights.The input resolution is set to 640 × 192 to preserve KITTI image aspect ratio.
  • Stage two: End-to-end fine-tuning uses a small number of KITTI Eigen training samples with ground-truth disparity labels.Upsampling is performed inside the stereo matching network because its input has a larger dimension.
  • Data augmentation: Optional augmentation randomly resizes and crops inputs, then multiplies color intensity by a factor between 0.8 and 1.2.

4.3. Depth Estimation by Stereo Matching method

The stereo matching pipeline is evaluated on KITTI using synthesized or real right views, with end-to-end finetuning on only 700 training samples. It outperforms prior monocular methods and improves substantially over the non-finetuned version.

  • With perfect right images, the stereo matching network outperforms state-of-the-art single-image depth methods, despite being mainly trained on rendered data.
  • Without end-to-end finetuning, synthesized right views yield performance better than the unsupervised method and comparable to the state-of-the-art semi-supervised method.
  • 700 KITTI training samples are used to finetune the complete system with left, right, and depth images.
  • 17.5% and 16.8% ARD reductions are achieved versus Godard et al. and Kuznietsov et al., respectively, at the 80 m cap.
  • The finetuned method outperforms all compared methods and achieves the best performance for almost all metrics.

4.5. Analyzing the function of two sub-networks after end-to-end training

Additional experiments test whether end-to-end training preserves the separate functions of view synthesis and stereo matching. The results indicate stronger collaboration, retained functionality, and generalization to unseen scenes.

  • The analysis replaces each finetuned sub-network in turn to test whether the other sub-network retains its original functionality.
  • View synthesis sub-network: Finetuned view synthesis improves over the non-finetuned method, while average synthesized-view PSNR increases from 21.29dB to 21.32dB.
  • Stereo matching sub-network: The stereo matching network remains functional after finetuning: the 700-sample variant outperforms the pretrained stereo matcher with true right images.
  • The two submodules collaborate more effectively after end-to-end training while preserving their individual functionalities.
  • Qualitative results on Cityscape and Make3D demonstrate generalization to unseen scenes after finetuning on KITTI.

4.6. Primitive disparity obtained in the view synthesis network

The view synthesis network produces a primitive probabilistic disparity map, but directly using it for depth estimation performs much worse than the final proposed method.

  • The final method substantially improves over the primitive disparity produced by the view synthesis network.

4.7. Analyzing the effect of training sample number

The method improves as more end-to-end finetuning samples are provided, while also achieving strong results with limited or no high-quality KITTI labels.

  • More end-to-end finetuning samples improve performance, with Finetune-K evaluated at 0, 200, 500, and 700 samples.At K=0, finetuning is not performed on the whole network.
  • Using only 700 samples to finetune the whole network, the method outperforms previous state-of-the-art methods by a clear margin.
  • Table 4 evaluates erroneous-pixel percentages on KITTI 2015 Stereo, counting a pixel as correct when disparity is within 3px of ground truth.
  • Figure 5 reports qualitative results for Make3D in the top two rows and Cityscapes in the bottom two rows.
  • Without real KITTI disparity labels, the method already achieves promising results and still beats the current state-of-the-art method after finetuning.The 200 high-quality KITTI labels increase model capacity to a certain extent, but are not required for improvement under the same condition.

4.9. Comparison with stereo matching method

The proposed single-image method is compared with monocular depth estimators and stereo matching on KITTI 2015. It surpasses block matching while retaining geometric reasoning.

  • The method is the first single-image depth approach reported to surpass the traditional stereo block matching method on the KITTI 2015 Stereo benchmark.
  • Block matching produces noisy disparity maps because it searches matched pixels using low-level image features, although its results remain geometrically correct.
  • The proposed network combines geometric reasoning with high-level image features, enabling it to outperform stereo matching.
  • Godard et al. obtain suboptimal results because their method lacks explicit geometric constraints.

5. Conclusion

The paper reformulates monocular depth estimation as view synthesis followed by stereo matching, explicitly encoding geometric transformations and training the pipeline collectively.

  • Monocular depth estimation is decomposed into view synthesis and stereo matching, with geometric transformations explicitly encoded in both networks.
  • Collective end-to-end training boosts overall performance while preserving both networks’ original functionality.
  • Without a large amount of expensive ground-truth labels, the method outperforms previous methods on a monocular depth estimation benchmark.
  • The monocular method is reported as the first to outperform stereo block matching on a stereo matching benchmark.
Loading 1803.02612v2…