PaperlayerView exact revision on arXiv

Source-linked AI summary

Digging Into Self-Supervised Monocular Depth Estimation

Clément Godard, Oisin Mac Aodha, Michael Firman, Gabriel Brostow

arXiv:1806.01260v4cs.CVstat.ML

TL;DR

Ground-truth depth is difficult to acquire at scale, motivating self-supervised monocular estimation from stereo pairs or video. The paper introduces minimum reprojection, auto-masking, and full-resolution multi-scale sampling, yielding state-of-the-art depth predictions with a simple model across monocular, stereo, and mixed training.

  • Problem

    Large, varied ground-truth depth datasets are difficult to collect, while monocular training additionally faces pose-estimation, occlusion, and camera-motion challenges.

  • Method

    The method combines per-pixel minimum reprojection, auto-masking, and full-resolution multi-scale sampling for self-supervised depth training.

  • Results

    The combined model achieves state-of-the-art depth predictions on KITTI and can be trained with monocular video, stereo data, or mixed supervision.

  • Takeaways & Limitations

    Careful appearance losses and image-resolution choices enable a simple model to produce high-quality self-supervised depth maps across supervision settings.

Abstract

from arXiv · show

Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods. Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.

1. Introduction

Monocular depth estimation avoids costly ground-truth depth collection but must handle pose estimation, occlusions, motion, and artifacts. Monodepth2 combines three simple design choices to improve self-supervised predictions across monocular, stereo, and mixed training.

  • Challenges: Monocular video supervision requires estimating camera egomotion and is vulnerable to occlusions, stationary cameras, and moving objects.These violations can produce severe depth failures, including holes of infinite depth for frequently moving objects.
  • Contributions: Auto-masking ignores pixels with no observed relative camera motion, addressing training pixels that violate the moving-camera, static-scene assumptions.The method is motivated by failures caused by stationary cameras and independently moving objects.
  • Contributions: Minimum reprojection matches each target pixel to the source view where it is visible, avoiding penalties from occluded or disoccluded pixels.Unlike averaging reprojection errors across source views, this choice is intended to produce sharper depth results.
  • Contributions: State-of-the-art KITTI results come from combining minimum reprojection, auto-masking, and full-resolution multi-scale sampling for monocular and stereo self-supervision.The model is reported to produce sharp, high-quality depth maps under monocular, stereo, and joint supervision.
  • Contributions: Full-resolution multi-scale sampling computes losses at the input resolution, reducing texture-copy artifacts from ambiguous low-resolution reconstructions.Intermediate depth predictions are upsampled before reprojection and loss computation.

2. Related Work

Prior work studies supervised, weakly supervised, stereo, and monocular-video approaches to depth estimation, with each supervision source imposing different data or geometric challenges. This paper emphasizes image-reconstruction supervision and shows that careful appearance losses and image resolution can make monocular training competitive with stereo training.

  • Supervised Depth Estimation: Ground-truth depth is difficult to collect at scale and across varied real-world settings, motivating weakly supervised and self-supervised alternatives.Existing alternatives include object-size, sparse-depth, appearance-matching, synthetic, and structure-from-motion supervision.
  • Self-Supervised Training: Self-supervised depth training reconstructs nearby views from predicted depth and pose, using stereo pairs or monocular sequences instead of ground-truth depth.Monocular sequences require a pose network, while stereo training uses known relative pose.
  • Self-Supervised Training: The paper reports that careful appearance losses and image resolution can reach stereo-training performance using only monocular training, while one contribution also improves stereo training.This motivates simpler design choices rather than increasingly complex model components.
  • Self-Supervised Training: Monocular video is less constrained than stereo because it requires jointly estimating camera pose and depth, especially under object motion.Stereo avoids learned pose estimation but introduces occlusion and texture-copy issues.
  • Monocular Video: Prior monocular methods address non-rigid motion with explanation masks or more elaborate motion models, while later work adds geometric consistency and normalization constraints.These approaches illustrate the range of mechanisms explored to close the monocular–stereo performance gap.

3. Method

The method trains a monocular depth network through photometric reprojection, then improves robustness with minimum reprojection, auto-masking, and full-resolution multi-scale sampling. A compact U-Net-style model combines these losses to reduce artifacts and improve depth accuracy.

  • 3.1. Self-Supervised Training: The network predicts depth from one color image and learns by minimizing photometric reprojection error between reconstructed and source views.Stereo training uses known relative pose, while monocular training jointly estimates pose with a separate pose network.
  • Per-Pixel Minimum Reprojection Loss: The minimum reprojection loss selects the lowest photometric error across source images, handling occlusions and improving border artifacts, occlusion-boundary sharpness, and accuracy.Averaging errors can penalize correctly predicted pixels that are invisible in some source images; the minimum avoids that penalty.
  • Auto-Masking Stationary Pixels: Auto-masking removes pixels whose unwarped source appearance is no worse than the reconstructed appearance, excluding stationary-camera and camera-matched object motion.The binary mask is computed during the forward pass and can remove entire frames when the camera is static.
  • Multi-scale Estimation: Full-resolution multi-scale sampling upsamples intermediate depth maps before reprojection, reducing low-texture holes and texture-copy artifacts.Each scale is constrained to reconstruct the high-resolution target image rather than an ambiguous low-resolution image.
  • 3. Method: The final loss combines masked photometric and per-pixel smoothness terms across pixels, scales, and batches.The model uses an encoder-decoder with skip connections and a ResNet18 encoder containing 11M parameters.

4. Experiments

Experiments show that Monodepth2 achieves strong self-supervised depth results across monocular, stereo, and mixed training, while each proposed component improves the model and known data limitations remain.

  • 4.1. KITTI Eigen Split: Monodepth2 outperforms existing state-of-the-art self-supervised monocular methods on KITTI and also achieves high accuracy with stereo-only supervision.The model is evaluated without post-processing; mixed monocular-plus-stereo training further improves accuracy, especially on RMSE-sensitive metrics.
  • Qualitative and Additional Evaluations: Qualitative results support sharper depth maps and fewer artifacts, while odometry performance is reported as comparable to competing methods.The authors note failure cases for Lambertian-violating objects and ambiguous or intricate boundaries.
  • 4.1.1. KITTI Ablation Study: Adding minimum reprojection, auto-masking, and full-resolution multi-scale components significantly improves the monocular baseline.The full model performs best in the ablation, including when trained without ImageNet pretraining or on the full Eigen split.
  • 4.1.1. KITTI Ablation Study: The auto-masking ablation underperforms using no mask, whereas the proposed auto-masking improves results across tested cases.This is particularly relevant for sequences containing stationary cameras, which otherwise create problems for monocular training.
  • Make3D: The method outperforms depth-supervision-free methods on Make3D when trained on KITTI, but evaluation is affected by imperfect image–ground-truth alignment.Results use a center crop and median scaling for the monocular model.

5. Conclusion

The paper concludes that three simple design choices yield state-of-the-art depth predictions across monocular, stereo, and mixed supervision. Qualitative results are strong, but failures remain for distorted, reflective, saturated, and ambiguous regions.

  • 5. Conclusion: The minimum reprojection loss, auto-masking loss, and full-resolution multi-scale sampling together form a simple, efficient model usable with monocular video, stereo data, or both.The components address occlusions, confusing stationary pixels, and depth artifacts, respectively.
  • 5. Conclusion: Monodepth2 produces sharp qualitative depth maps, while failures occur in distorted, reflective, color-saturated, ambiguous-boundary, and intricate-shape regions.These limitations arise in settings that violate the appearance assumptions or make object delineation difficult.

Supplementary Material

An updated implementation separates the pose encoder from the depth encoder and adds auto-masking to handle pixels violating motion assumptions.

  • Supplementary Material: The revised model uses a separate ResNet pose encoder and auto-masking, replacing the earlier shared-encoder design.The authors report higher results with the separate pose encoder and introduced auto-masking to ignore pixels that violate motion assumptions.

A. Odometry Evaluation

The odometry evaluation tests the same lightweight pose architecture used elsewhere, converting pairwise predictions into trajectories for comparison with five-frame methods.

  • A. Odometry Evaluation: The evaluation uses sequences 0–8 for training and 9–10 for testing, reporting average absolute trajectory error and standard deviation.The protocol averages error over overlapping five-frame snippets.
  • A. Odometry Evaluation: Pairwise pose predictions approach the accuracy of methods trained on five-frame blocks when evaluated on KITTI odometry trajectories.Four pairwise transformations are combined to form local trajectories over five-frame test snippets.
  • A. Odometry Evaluation: The pose network accepts two frames and predicts one 6-DoF transformation, unlike baselines that predict transformations from five frames simultaneously.This pairwise design is evaluated by chaining four predictions into five-frame trajectories.

C. Additional Ablation Experiments

The ablations show that combining all components gives the highest-quality model, while deeper encoders improve results at the cost of longer training and testing. On improved KITTI ground truth, the method significantly outperforms previously published methods across all metrics.

  • Ablation Study: All components together produce the highest-quality model, and pretraining further helps in the full ablation study.The ablation turns different system components on and off to assess their combined contribution.
  • Encoder Ablation: ResNet-50 encoders outperform the authors’ ResNet-18 models, but require longer training and test times.ResNet-50 is also a standard encoder used by previous works [15] [50].
  • Qualitative Results: Qualitative comparisons show depth holes in baseline variants that are absent from the authors’ models.The comparison includes both non-pretrained and pretrained monocular variants.
  • Improved Ground Truth: The method significantly outperforms all previously published methods on every metric using improved KITTI ground truth.Evaluation uses 652 improved-ground-truth frames, full images, and predictions clipped to 80 meters.

D.2. Single-Scale Evaluation

The single-scale evaluation addresses the lack of guaranteed metric scale in monocular self-supervised predictions and modifies evaluation to use one scale per method. Under this protocol, the method is compared on improved KITTI ground truth and benchmark submissions, while qualitative results show fewer depth holes and border artifacts.

  • Scale Consistency: Monocular self-supervised predictions have no guarantee of metric scale, unlike stereo-trained predictions whose scale is known [12] [15].The authors therefore examine whether predictions remain consistent across outputs.
  • Evaluation Protocol: Per-image median scaling can hide unstable monocular scale estimates and unfairly favor monocular methods over stereo-trained methods.The authors argue that independently scaling each prediction creates a best-case monocular evaluation.
  • Evaluation Protocol: The modified protocol uses one scale per method, computed as the median of individual depth-median ratios across the test set.The authors note that this still uses ground-truth depth but is fairer and more representative.
  • Benchmark Evaluation: The single-scale evaluation compares existing monocular supervised methods on the Eigen split using improved ground truth.Table 8 specifies the comparison setting and marks newer online-implementation results with †.
  • Qualitative Results: The method generates higher-quality KITTI outputs without the holes or border artifacts visible in several existing baselines [76] [51] [15] [73].These qualitative results are shown on the KITTI test set.

F. Results with Post-Processing

Post-processing, originally introduced for stereo-trained monocular depth estimation, also improves quantitative performance for monocular-only and mixed training. It requires two forward passes per test image, trading computation for accuracy gains.

  • Quantitative Results: Post-processing consistently improves all of the authors’ monocular-trained models and can exceed the gains from higher-resolution training for some metrics.Table 10 evaluates its effect across training settings.
  • Quantitative Results: Post-processing improves quantitative performance in both monocular-only (M) and mixed (MS) training for the first time.The technique was originally motivated only for stereo training.

G. Effect of Image Resolution

Higher input and output resolution improves results, but the full-resolution multi-scale design provides additional benefit beyond resolution alone. Qualitatively, all resolutions recover scene layout, whereas lower resolution struggles with thin objects and depth boundaries.

  • Quantitative Results: Higher resolution improves performance, while full-resolution multi-scale sampling remains beneficial even at the highest resolution.Table 11 compares 640 × 192, 1024 × 320, and 416 × 128 settings and includes the corresponding ablation.
  • Training Setup: The highest-resolution models use 10 epochs at 640 × 192 followed by 5 epochs at 1024 × 320 with a learning rate of 10^-5.A batch size of 4 allows training on a single 12GB Titan X GPU.
  • Experimental Scope: The resolution study evaluates input and output resolutions, with the highest-resolution timing including both training stages.Table 11 reports the ablation and timing scope.
  • Qualitative Results: At 128 × 416, the model captures broad scene shape but struggles with thin objects and depth discontinuities near object boundaries.All predicted disparity maps are resized to the same visualization size.
  • Qualitative Results: All resolutions accurately capture the overall scene shape, but only the highest resolution accurately represents finer structure.The qualitative comparison is shown in Figure 11.

I. Supplementary Video Results

Supplementary results evaluate the model on Wander, a handheld monocular dataset distinct from KITTI, and compare visual depth quality against a contribution-free baseline. The authors report fewer artifacts on Wander and sharpest depth maps for their final approaches on KITTI Eigen results.

  • Wander: The Wander dataset uses handheld monocular videos from a non-European environment rather than car-mounted KITTI footage.Training uses four sequences, with evaluation on a fifth unseen sequence at 128 × 224 resolution.
  • Wander: Our model produces fewer visual artifacts than the baseline on the Wander results.The baseline uses the same model with VGG loss but without the paper’s contributions.
  • KITTI Eigen split: The last three approaches produce the sharpest depth maps in the additional KITTI Eigen split results.The comparison identifies ‘Baseline M’ as the model without the paper’s contributions.
Loading 1806.01260v4…