Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Yevhen Kuznietsov; Jörg Stückler; Bastian Leibe

Source-linked AI summary

Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Yevhen Kuznietsov, Jörg Stückler, Bastian Leibe

arXiv:1702.02706v3Feb 9, 2017cs.CV

TL;DR

Monocular depth prediction lacks dense, reliable outdoor ground truth because LiDAR measurements are noisy, sparse, and imperfectly calibrated. The paper combines sparse supervised depth with unsupervised stereo image alignment to train a semi-supervised predictor, achieving state-of-the-art performance on KITTI. Generalization remains limited when the model is applied to substantially different scene domains such as Make3D and NYUDv2.

Problem
Outdoor monocular depth learning lacks dense, reliable ground-truth images because LiDAR measurements are noisy, sparse relative to camera images, and imperfectly calibrated.
Method
The method jointly trains a CNN with sparse supervised depth measurements and an unsupervised direct image-alignment signal from stereo images and predicted depth.
Results
State-of-the-art performance is achieved for single-image depth prediction on the KITTI dataset, with semi-supervised training improving over purely supervised training.
Takeaways & Limitations
Combining supervised, unsupervised, and regularization terms produces detailed depth maps, including for thin and distant objects and image regions without supervised ground truth.
Takeaways & Limitations
Models trained on KITTI generalize imperfectly to Make3D and NYUDv2, where differing scene perspectives and domains cause characteristic prediction errors.

Abstract

from arXiv · show

Supervised deep learning often suffers from the lack of sufficient training data. Specifically in the context of monocular depth map prediction, it is barely possible to determine dense ground truth depth images in realistic dynamic outdoor environments. When using LiDAR sensors, for instance, noise is present in the distance measurements, the calibration between sensors cannot be perfect, and the measurements are typically much sparser than the camera images. In this paper, we propose a novel approach to depth map prediction from monocular images that learns in a semi-supervised way. While we use sparse ground-truth depth for supervised learning, we also enforce our deep network to produce photoconsistent dense depth maps in a stereo setup using a direct image alignment loss. In experiments we demonstrate superior performance in depth map prediction from single images compared to the state-of-the-art methods.

1. Introduction

Monocular depth prediction is ill-posed and supervised learning is constrained by sparse, noisy, and imperfectly calibrated outdoor depth measurements. The paper combines supervised ground truth with unsupervised stereo image alignment to achieve state-of-the-art KITTI performance.

Single-image depth estimation is ill-posed because bottom-up geometric cues alone generally cannot determine depth.
Supervised learning requires vast training data, while outdoor laser measurements are noisy, sparse relative to images, and sensitive to sensor calibration.These measurements also fail to capture high-detail depth variations visible in camera images.
Synthetic depth rendering avoids supplementary sensors but does not fully reproduce realistic scenes or real image noise.
Unsupervised stereo methods remove the need for supplementary ground truth but retain the ill-posedness of image-matching-based depth reconstruction.Regularization schemes may impose depth priors, such as small gradient norms, that real environments do not fully satisfy.
The proposed semi-supervised approach combines sparse ground-truth measurements with unsupervised direct image alignment to use complementary training cues.The paper reports significant improvements over state-of-the-art single-image depth prediction on KITTI urban scenes.

2. Related Work

Related work progresses from supervised monocular depth models using engineered or learned image features to recent unsupervised stereo approaches trained through image alignment. These methods differ in architecture, structured refinement, and the assumptions used to handle photometric reconstruction.

Early supervised methods modeled depth with Markov random fields and hand-crafted multi-scale texture features, combining monocular cues with stereo correspondences.
Deep supervised approaches integrated coarse- and fine-scale prediction, combined learned patch features with hierarchical CRFs, or learned unary and pairwise potentials end-to-end.
ResNet-based encoder-decoder architectures were introduced to produce dense depth maps from images.
Recent unsupervised methods trained single-image depth predictors using stereo image-alignment losses that warp one image into its counterpart using predicted depth.One described method linearizes the loss with a first-order Taylor approximation and therefore requires coarse-to-fine training.

3. Approach

The approach combines sparse supervised LiDAR depth with stereo photoconsistency and regularization in a symmetric semi-supervised loss. A residual encoder-decoder with long skip connections predicts detailed inverse-depth maps from RGB images.

Loss function: Sparse LiDAR supervision provides depth targets, while stereo photoconsistency supplies dense training cues across image pixels.The supervised signal is available only on ΩZ, whereas the image-alignment constraint applies throughout the image.
Loss function: The loss combines supervised, unsupervised, and regularization terms symmetrically for both stereo cameras.This formulation implicitly enforces consistency between the predicted depth maps of the left and right images.
Loss function: Direct image alignment measures photoconsistency in both directions, using Gaussian smoothing and valid warped-pixel sets with linear subpixel interpolation.The smoothing kernel has standard deviation σ = 1 px.
Loss function: An anisotropic smoothness regularizer penalizes depth changes in low-intensity-variation regions while allowing discontinuities at object contours.The regularization is downscaled according to image-intensity variation.
Network Architecture: The network uses a residual encoder-decoder with long skip connections from corresponding encoder layers to the decoder for fine-detail depth prediction.The encoder resembles ResNet-50, while the decoder upprojects features through residual blocks; skip connections slightly improve all metrics without affecting convergence.
Network Architecture: Residual blocks use three successive convolutions, with type 2 blocks additionally applying stride and projecting the input to the residual channel count.Type 1 blocks use stride s = 1 and preserve the input channel count in the residual.

4. Experiments

Experiments evaluate the approach on KITTI and Make3D, compare it with state-of-the-art methods, and analyze training and architectural choices. The method achieves strong quantitative and qualitative results, with semi-supervised training outperforming purely supervised training.

Datasets and setup: The KITTI evaluation uses stereo imagery and Velodyne laser measurements, with 28 testing scenes and 7,346 unique training frames.The training set contains 12,600 frames in total, while the test split follows the protocol proposed by Eigen et al.
Training setup: The encoder is initialized with ImageNet-pretrained ResNet-50 weights, while decoder filters are initialized randomly.The authors report no performance difference when using He initialization instead of Glorot and Bengio initialization.
Comparison with the state of the art: For all metrics and setups on KITTI, the proposed system performs best, including 1.16 m lower RMSE and 0.035 lower log-scale error than Godard et al. at an 80 m cap.At a 50 m cap, the method is 1.586 m more accurate in RMSE than Garg et al.; the unsupervised-loss benefit is larger over 0–80 m.
Comparison with the state of the art: Qualitatively, the method produces crisp predictions for cars, pedestrians, traffic signs, thin poles, and lights, while also predicting depth in image regions lacking ground truth.Some competing predictions appear more detailed, but the paper notes that these details are not always consistent with ground truth.
Ablation study: Removing long skip connections and Gaussian smoothing together reduces performance by 0.119 RMSE relative to the full approach.The ablation also reports that applying the unsupervised loss on all valid pixels improves performance over restricting it to pixels without ground truth.
Ablation study: Semi-supervised training reduces RMSE by 0.235 compared with purely supervised training, while purely unsupervised training produces more outliers and performs worse than the compared unsupervised methods.The method remains effective when laser ground truth is reduced to 50% or 1% of the available measurements.
Generalization to other datasets: The semi-supervised method converges in about one third as many iterations as purely supervised training and generalizes qualitatively to Make3D and Cityscapes.On Make3D, it obtains 8.237 RMSE (m), 0.190 Log10 error, and 0.421 ARD.

5. Conclusions

The approach combines sparse supervised depth cues with stereo image-alignment supervision for monocular depth prediction. Its architecture and loss design support detailed estimates, including regions without supervised ground truth, while future work targets additional tasks and unified monocular-stereo learning.

5. Conclusions: The method complements sparse supervised depth measurements with unsupervised direct image alignment between stereo-camera images.Photoconsistency is quantified for corresponding pixels determined by the CNN's predicted depth.
5. Conclusions: A deep residual encoder-decoder with long skip connections combines supervised, unsupervised, and regularization terms symmetrically across stereo cameras.The symmetric loss design implicitly promotes consistency between depth estimates.
5. Conclusions: The approach achieves state-of-the-art single-image depth prediction on the KITTI dataset.
5. Conclusions: The model predicts detailed depth for thin and distant objects and reasonable depth where supervised ground truth is unavailable.
5. Conclusions: Future work will investigate semi-supervised semantic segmentation and a unified framework coupling monocular and stereo depth cues.

6. Introduction

The supplementary material adds qualitative evaluations across KITTI Raw, Cityscapes, and Make3D datasets.

6. Introduction: Additional qualitative results are provided on the KITTI Raw dataset.
6. Introduction: Additional qualitative results are provided on the Cityscapes dataset.
6. Introduction: Additional qualitative results are provided on the Make3D dataset.

7. KITTI

On KITTI Raw Eigen test images, the semi-supervised approach qualitatively improves predictions over supervised-only training, especially in regions without ground truth, while remaining visually smoother.

7. KITTI: The full semi-supervised model achieves better predictions than supervised-only training in image regions without ground truth.
7. KITTI: The full model's predictions are smoother and visually more appealing than those of the supervised-only variant.
7. KITTI: The supplementary KITTI examples include qualitative failure cases in recovering scene structures.
7. KITTI: Three-dimensional point-cloud visualizations show various results obtained on the test images.

8. Generalization to Other Datasets

The KITTI-trained model is qualitatively evaluated across Cityscapes, Make3D, NYUDv2, and KITTI examples. Results show some cross-dataset generalization, but differing viewpoints, camera parameters, scenes, and object structures produce characteristic failures.

8.1. Cityscapes: On Cityscapes, the KITTI-trained model qualitatively generalizes in the upper six rows, while the bottom two rows show typical failures.The reference depth maps use stereo semi-global matching, and the images are cropped to 847×2048.
8.1. Cityscapes: Cityscapes failures mainly reflect differences in scene perspectives, objects, aspect ratio, camera intrinsics, and vehicle-view pose from KITTI.These camera differences may prevent accurate absolute-depth capture; supervised fine-tuning on Cityscapes is expected to improve results.
8.2. Make3D: On Make3D, the model captures foreground vegetation and cars well in some examples but fails in scenes unlike the KITTI training data.Its different vertical field of view makes close-range ground recovery difficult, and sky depth is often estimated incorrectly.
8.2. Make3D: Make3D generalization is difficult because images are not taken from on-road vehicles and scene perspectives vary more strongly.Supervised fine-tuning on Make3D should improve results significantly.
8.3. NYUDv2: NYUDv2 provides an expected cross-domain limitation because the model was trained on outdoor KITTI scenes but evaluated on indoor images.The predictions were visually rescaled by 0.3 for comparison, and supervised fine-tuning on NYUDv2 could further improve performance.
7. KITTI: KITTI qualitative examples compare the full semi-supervised model with supervised-only training, including failure cases and 3D point-cloud visualizations.