PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Depth Estimation: A Survey

Amlaan Bhoi

arXiv:1901.09402v1cs.CV

TL;DR

Monocular depth estimation seeks dense depth from a single RGB image despite the ambiguity of mapping 2D images to 3D scenes. This survey reviews five supervised, weakly supervised, and unsupervised approaches, comparing their improvements and identifying directions for more efficient and precise estimation.

  • Problem

    Monocular depth estimation is an ill-posed task because one 2D image may arise from infinitely many distinct 3D scenes, yet depth supports applications including scene understanding and robotics.

  • Method

    The paper reviews and compares five monocular depth estimation methods using supervised, weakly supervised, and unsupervised learning approaches.

  • Results

    The reviewed approaches include multi-scale CNN-CRF models with mean-field updates and achieve 0.586 RMS with 0.987 thresholded accuracy on NYU Depth V2 in one reported method.

  • Takeaways & Limitations

    The survey concludes that monocular depth estimation can still be improved for greater efficiency and precision.

  • Takeaways & Limitations

    The survey notes that supervised methods need multi-scale feature extraction, while unsupervised methods require additional problem information such as camera egomotion and 3D geometric constraints.

Abstract

from arXiv · show

Monocular depth estimation is often described as an ill-posed and inherently ambiguous problem. Estimating depth from 2D images is a crucial step in scene reconstruction, 3Dobject recognition, segmentation, and detection. The problem can be framed as: given a single RGB image as input, predict a dense depth map for each pixel. This problem is worsened by the fact that most scenes have large texture and structural variations, object occlusions, and rich geometric detailing. All these factors contribute to difficulty in accurate depth estimation. In this paper, we review five papers that attempt to solve the depth estimation problem with various techniques including supervised, weakly-supervised, and unsupervised learning techniques. We then compare these papers and understand the improvements made over one another. Finally, we explore potential improvements that can aid to better solve this problem.

1. Introduction

Monocular depth estimation predicts scene depth from one RGB image, supporting applications that lack stereo, motion, or point-cloud data. Its difficulty motivates deep models that exploit monocular cues, multi-scale features, and CRFs.

  • Monocular depth estimation predicts depth from a single RGB image for scene understanding, 3D modelling, robotics, and autonomous driving.
  • A single 2D image can correspond to infinitely many distinct 3D scenes, making computational depth estimation ill-posed.
  • Monocular cues include perspective, texture, object size and location, and occlusions, while stereo or video methods require more resources and data.
  • Recent deep models commonly use convolutional networks, multi-scale feature fusion, and conditional random fields to improve pixel-level predictions.

2. Problem Definition

The paper formulates monocular depth estimation as learning a mapping from RGB images to dense real-valued depth maps. This formulation primarily describes supervised learning, while other methods relax the need for pixel-level ground truth.

  • Given RGB-depth training pairs, monocular depth estimation is a pixel-level continuous regression problem.
  • Standard supervised formulations typically use mean squared error in log-space or related losses.
  • The supervised task learns a nonlinear mapping Φ from the space of RGB images to the domain of real-valued depth maps.
  • Some methods relax the requirement for pixel-level ground truth by introducing alternative requirements and constraints.

3. Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Eigen et al.’s multi-scale deep network separates global scene structure from local refinement and uses a scale-invariant error for depth prediction. On NYU Depth V2, it achieved 7.165 RMSE and 0.967 thresholded accuracy.

  • The network first estimates global scene structure, then refines local details using multi-scale features.
  • The global branch captures vanishing points, object locations, and room alignment, while the local branch refines objects and wall edges.
  • 3.2. Scale-Invariant Error: The scale-invariant error measures relationships between scene points independently of absolute global scale.
  • 3.2. Scale-Invariant Error: 7.165 RMSE and 0.967 thresholded accuracy (δ < 1.253) were achieved on NYU Depth V2.

4. Multi-Scale Continuous CRFs as Sequential Deep Networks for MDE

Xu et al. fuse multi-scale CNN features within continuous CRFs implemented as sequential deep networks. Their framework includes multi-scale and cascade CRFs plus mean-field updates, achieving 0.586 RMS and 0.987 thresholded accuracy on NYU Depth V2.

  • The framework structurally fuses multi-scale estimations from CNN inner semantic layers within a unified CNN-CRF architecture.
  • Multi-scale CRFs and cascaded CRFs progressively combine features across scales, with cascade stages using outputs from the previous scale.
  • It implements mean-field updates as sequential deep models rather than leaving them as a separate shallow inference procedure.
  • The CRF uses unary terms and pairwise potentials to model depth and relationships between hidden variables or neighboring pixels.
  • Pairwise potentials use Gaussian kernels for appearance features and pixel positions, encouraging similar-appearing neighboring pixels to have close depths.
  • 4.1.2 Cascade CRFs: 0.586 RMS and 0.987 thresholded accuracy (δ < 1.253) were achieved on NYU Depth V2 with a ResNet50 front-end.

5. Structured Attention Guided Convolutional Neural Fields for MDE

The method combines multi-scale CNN features with a continuous CRF and structured attention to regulate cross-scale information flow and enforce attention structure. On NYU Depth V2, it achieved an RMS score of 0.593 and thresholded accuracy of 0.986.

  • Method: The framework extracts multi-scale features from a front-end CNN and feeds them into a continuous CRF for structured refinement.Its overall structure follows a CNN front end with CRF-based multi-scale modeling.
  • Contributions: Unlike methods operating only on prediction maps, this approach operates directly at the feature level and claims faster inference than previous approaches.The feature-level design is presented as a distinction from earlier prediction-map-based methods.
  • Method: A structured attention mechanism controls information transfer between intermediate and final scales while enforcing relationships among attention variables.The attention variables regulate cross-scale information flow, and an additional term enforces structural constraints among them.
  • Implementation: Attention-map updates use convolutional message passing, attention-map message passing, and sigmoid normalization, while intermediate-scale mean-field updates are omitted to reduce overhead.Mean-field updates are retained for the final-scale representation but not performed at intermediate scales.
  • Results: 0.593 RMS and 0.986 thresholded accuracy (δ < 1.253) were achieved on NYU Depth V2.The network was jointly optimized using standard stochastic gradient descent.

6. Deep Ordinal Regression Network for MDE

The method reframes monocular depth estimation as ordinal regression over space-increasing depth intervals rather than continuous prediction. Its architecture preserves spatial resolution with dilated convolutions and combines parallel scene-understanding modules, achieving 0.509 RMS and 0.992 thresholded accuracy on NYU Depth V2.

  • Method: Continuous depth values are discretized into intervals, and learning is formulated as ordinal regression to preserve ordering among discrete labels.The ordinal outputs encode ordered threshold decisions rather than treating depth intervals as unrelated classes.
  • Method: Space-increasing discretization allocates wider intervals at larger depths because depth uncertainty increases with ground-truth depth.This design avoids over-strengthening the training loss for large depth values.
  • Architecture: The dense feature extractor removes late downsampling and uses dilated convolutions to enlarge receptive fields without reducing spatial resolution or increasing parameters.This addresses the spatial-resolution loss caused by conventional pooling and striding.
  • Architecture: The scene-understanding module has parallel ASPP, cross-channel learner, and full-image encoder components.The ASPP component extracts features from multiple large receptive fields via dilation.
  • Results: 0.509 RMS and 0.992 thresholded accuracy (δ < 1.253) were achieved on NYU Depth V2.The method also reports results on KITTI and Make3D.

7. Unsupervised MDE with Left-Right Consistency

This approach learns monocular depth without labeled depth maps by training on calibrated stereo pairs, using reconstruction, smoothness, and left-right consistency constraints.

  • 7. Unsupervised MDE with Left-Right Consistency: The network performs unsupervised monocular depth estimation by training on binocular stereo footage while requiring only one image at test time.It predicts pixel-level correspondence between rectified stereo images and converts disparity to depth using camera parameters.
  • 7.2 Results: The authors report that the network generalizes across at least three tested datasets without labeled depth data.The review identifies this cross-dataset generalization as a main contribution.
  • 7.1 Network Architecture: Its fully convolutional encoder-decoder predicts disparities at four scales and uses skip connections for end-to-end learning.Backward mapping with a bilinear sampler reconstructs images from predicted disparities.
  • 7.1.3 Loss Functions: Training combines appearance matching, edge-aware disparity smoothness, and left-right consistency losses to improve reconstructed-image similarity and disparity coherence.Appearance matching combines L1 and single-scale SSIM losses, while smoothness is weighted by image gradients.
  • 7.2 Results: On KITTI, the method achieved 4.935 RMSE and thresholded accuracy of 0.976 for δ < 1.253.At inference, the finest-scale left-view disparity is converted into depth using camera distance and focal length.

8. Datasets

The review describes three conventional datasets used to evaluate monocular depth estimation: NYUD-V2, KITTI, and Make3D, each with distinct image and depth-data characteristics.

  • NYUD-V2: NYUD-V2 contains 1,449 densely labeled aligned RGB-depth pairs, 464 scenes from three cities, and 407K unlabeled frames.Images are commonly down-sampled for faster training; one example uses 288 × 384 resolution.
  • KITTI: The KITTI Eigen split contains 23,488 training images from 32 scenes and 697 testing images from 29 scenes.This split is commonly used instead of the entire KITTI dataset.
  • Make3D: Make3D contains 534 outdoor images, with 400 for training and 134 for testing.Its images are 2272 × 1704, while ground-truth depth maps have 55 × 305 resolution and are often down-sampled.

9. Discussion

The discussion traces monocular depth estimation from multi-scale supervised regression through CRFs, attention, ordinal regression, and unsupervised stereo-based learning, while identifying remaining efficiency and precision gaps.

  • Eigen et al.: Eigen et al. established direct pixel regression with global-coarse and local-fine multi-scale networks, plus a scale-invariant loss for scale-dependent errors.Later deep CNN approaches reuse multi-scale information for global prediction and local refinement.
  • Multi-Scale Continuous CRFs: Xu et al. extended multi-scale depth regression with continuous CRFs implemented as sequential deep networks, including cascading CRFs and mean-field updates.The approach embeds CRF processing within deep network computation rather than treating it as a shallow post-processing model.
  • Structured Attention Guided Convolutional Neural Fields: Structured attention guided convolutional neural fields added attention variables to the multi-scale CRF framework and retained end-to-end encoder-decoder training.The attention mechanism extends the preceding CRF-based approach.
  • Deep Ordinal Regression Network: Fu et al. replaced per-pixel continuous regression with ordinal regression and spacing-increasing discretization, using a dense feature extractor and scene-understanding module.This method also departs from the encoder-decoder structure used by earlier approaches.
  • Unsupervised Monocular Depth Estimation: The unsupervised method trains with stereo pairs and combines appearance matching, disparity smoothness, and left-right consistency without depth maps.The review contrasts this changed input and training scope with supervised methods.
  • Future Scope: For supervised methods, the review identifies multi-scale feature extraction as necessary for decent performance; unsupervised methods may require additional cues such as camera egomotion and 3D geometry.The authors present these as possible directions for further improvement.
  • 9. Discussion: The review concludes that the five examined methods provide varied improvements and pitfalls, but monocular depth estimation remains open to more efficient and precise solutions.The authors frame additional approaches and future work as necessary.
Loading 1901.09402v1…