PaperlayerView exact revision on arXiv

Source-linked AI summary

Monocular Depth Estimation Based On Deep Learning: An Overview

Chaoqiang Zhao, Qiyu Sun, Chongzhen Zhang, Yang Tang, Feng Qian

arXiv:2003.06620v2cs.CV

TL;DR

Monocular depth estimation must infer dense depth from a single image despite traditional methods relying on multiple viewpoints and often producing sparse maps. This review synthesizes deep learning approaches by datasets, metrics, training modes, network frameworks, and loss functions, while discussing challenges and future directions. The surveyed methods demonstrate that deep neural networks can recover pixel-level depth maps from single images, with representative work improving results through network and loss-function design.

  • Problem

    Dense depth estimation from a single image is ill-posed, while traditional structure-from-motion and stereo approaches rely on multiple viewpoints and may produce sparse depth.

  • Method

    The review surveys monocular depth-estimation methods by supervised, unsupervised, and semi-supervised training, alongside datasets, metrics, network frameworks, and loss functions.

  • Results

    Deep neural networks recover pixel-level depth maps from single images end-to-end, and representative methods improve performance through deeper networks and improved loss functions.

  • Takeaways & Limitations

    The field has promising deep learning-based solutions, but progress remains tied to improving accuracy, geometric constraints, network design, and training strategies.

Abstract

from arXiv · show

Depth information is important for autonomous systems to perceive environments and estimate their own state. Traditional depth estimation methods, like structure from motion and stereo vision matching, are built on feature correspondences of multiple viewpoints. Meanwhile, the predicted depth maps are sparse. Inferring depth information from a single image (monocular depth estimation) is an ill-posed problem. With the rapid development of deep neural networks, monocular depth estimation based on deep learning has been widely studied recently and achieved promising performance in accuracy. Meanwhile, dense depth maps are estimated from single images by deep neural networks in an end-to-end manner. In order to improve the accuracy of depth estimation, different kinds of network frameworks, loss functions and training strategies are proposed subsequently. Therefore, we survey the current monocular depth estimation methods based on deep learning in this review. Initially, we conclude several widely used datasets and evaluation indicators in deep learning-based depth estimation. Furthermore, we review some representative existing methods according to different training manners: supervised, unsupervised and semi-supervised. Finally, we discuss the challenges and provide some ideas for future researches in monocular depth estimation.

I. INTRODUCTION

Monocular depth estimation seeks dense depth maps from a single image, addressing limitations of geometry- and sensor-based approaches that often require multiple views or produce sparse measurements. This overview surveys deep learning methods, datasets, metrics, training modes, frameworks, challenges, and future directions.

  • Depth estimation supports SLAM, navigation, object detection, and semantic segmentation.
  • Structure from motion and stereo matching recover 3D structure from multiple viewpoints, but SfM depends on feature correspondences and has monocular scale ambiguity.
  • Geometry-based methods generally estimate sparse depth and depend on image pairs or sequences, leaving dense single-image estimation challenging.
  • Depth sensors provide direct measurements but have limitations including sparse maps, restricted range, sunlight sensitivity, size, or power consumption.
  • Deep neural networks recover pixel-level depth maps from single images end-to-end using architectures including CNNs, RNNs, VAEs, and GANs.
  • The review organizes prior work by learning method and covers datasets, evaluation indicators, network frameworks, challenges, and promising research directions.

A. Datasets

The overview describes commonly used datasets spanning outdoor driving, indoor environments, semantic-segmentation imagery, and monocular RGB-depth testing. Their sensing modalities and annotations determine how they support supervised, unsupervised, semi-supervised, or generalization evaluation.

  • KITTI is a major benchmark and primary training dataset for unsupervised and semi-supervised monocular depth estimation.It contains real images from city, residential, and road categories.
  • NYU Depth contains 464 indoor scenes, with 249 used for training and 215 for testing, and is a primary supervised-learning benchmark.
  • Cityscapes provides 5,000 finely annotated and 20,000 coarsely annotated images from stereo video sequences across 50 cities.Because it lacks depth ground truth, it is used for training several unsupervised methods.
  • Make3D contains monocular RGB and depth images without stereo pairs or monocular sequences, so supervised methods use it for training and unsupervised methods commonly use it for testing.Testing use evaluates network generalization ability.

B. Evaluation metrics

The review uses five commonly accepted indicators to evaluate and compare monocular depth-estimation networks. These metrics compare predicted pixel depths with ground-truth depths across error and accuracy measures.

  • The evaluation protocol includes RMSE, RMSE log, Abs Rel, Sq Rel, and Accuracies.
  • The Accuracies indicator reports the percentage of pixels whose predicted-to-ground-truth depth ratio satisfies a threshold condition.
  • For each pixel, d_i denotes predicted depth, d*_i denotes ground-truth depth, N denotes the number of pixels with real-depth values, and thr denotes the threshold.

III. MONOCULAR DEPTH ESTIMATION BASED ON DEEP LEARNING

Deep learning-based monocular depth estimation learns structural information from supervised signals or alternative training data to predict depth from single images. The review groups methods by their use of ground truth and summarizes their data, supervision, contributions, and benchmark results.

  • Prior work uses geometric priors and CNNs to estimate dense depth maps from single images.
  • Methods are categorized as supervised, unsupervised, or semi-supervised according to their use of ground truth during training.Unsupervised and semi-supervised training can use monocular videos or stereo pairs, while testing predicts depth from single images.
  • The review summarizes training data, supervised signals, method contributions, and quantitative KITTI results for unsupervised and semi-supervised algorithms.

A. Supervised monocular depth estimation

Supervised monocular depth estimation predicts depth from a single image using ground-truth depth maps, with CNN architectures, specialized losses, refinement modules, and adversarial strategies improving estimates. Its main limitation is dependence on expensive labeled training data.

  • Supervised methods treat monocular depth estimation as regression, training networks to predict depth maps from single images against ground-truth maps.
  • Methods based on different architectures and loss functions: Eigen et al. use coarse-scale and fine-scale CNN components, with the fine-scale network refining the coarse depth prediction.
  • Methods based on different architectures and loss functions: Additional losses promote local structural consistency, while Berhu loss combines L1 behavior for small errors with L2 behavior for larger errors.
  • Methods based on different architectures and loss functions: Ordinal regression with spacing-increasing discretization accommodates larger errors at greater depths, while camera models, lightweight networks, and pruning address generalization and deployment.
  • Methods based on conditional random fields: Hierarchical and continuous CRFs refine depth using neighboring super-pixels, local structure, and interactions between depth and semantic labels.
  • Methods based on adversarial learning: Adversarial methods use discriminators with predicted and real depth maps, while conditional GAN variants can refine coarse depth estimates using image information.
  • Ground-truth supervision helps learn 3D structure and scale from single images, but labeled training sets are difficult and expensive to acquire.

B. Unsupervised monocular depth estimation

Unsupervised monocular methods learn depth from geometric constraints in neighboring monocular frames rather than ground-truth depth, while semi-supervised methods use stereo pairs and reconstruction errors. These approaches support practical training without dense labels but face scale ambiguity, occlusions, dynamic scenes, and added modeling complexity.

  • Unsupervised methods use geometric constraints between neighboring monocular frames instead of expensive ground-truth depth annotations.
  • A basic model for unsupervised methods: A depth network predicts depth and a pose network predicts frame transformations, which establish pixel correspondences through camera projection.
  • A basic model for unsupervised methods: During testing, the depth and pose networks can operate independently even though they are coupled during unsupervised training.
  • A basic model for unsupervised methods: View synthesis reconstructs a target frame from a neighboring image, and photometric differences, SSIM, and edge-aware smoothness losses supervise training.
  • Methods based on explainability mask: Explainability and object masks reduce dynamic-object and occlusion effects, but learned masks add computation and complicate training; geometry-based masks are proposed as alternatives.
  • Methods based on multi-tasks framework: Multi-task frameworks use optical flow, object motion, or camera intrinsics as additional supervision, while adversarial methods discriminate reconstructed images against real images.
  • Semi-supervised stereo methods predict inverse depth or disparity from the left image and reconstruct it from the right image through inverse warping.
  • Monocular-sequence training lacks absolute scale information, causing scale ambiguity and scale inconsistency alongside occlusion-related problems.

C. Semi-supervised monocular depth estimation

Semi-supervised methods add signals such as sparse depth, physical priors, known stereo poses, semantic information, adversarial learning, or distillation to improve monocular depth estimation while reducing dependence on dense ground truth.

  • Semi-supervised methods target higher accuracy than unsupervised methods while reducing dependence on expensive dense ground truth and learning scale information from additional signals.
  • Known left-right poses distinguish stereo-pair training from monocular-video training and are treated as supervised signals in this review.
  • Stereo-trained methods estimate disparity between left and right images, then use inverse warping to synthesize one view and supervise reconstruction against the real image.Depth is recovered from disparity using d = fB/D, where f is camera focal length and B is the camera baseline.
  • Left-right disparity prediction and consistency constraints, together with SSIM, improve reconstruction structure similarity and outperform earlier work.Related extensions address occlusions, image borders, semantic prediction, and region-aware depth estimation.
  • Adversarial learning and knowledge distillation are used to improve performance, with distillation transferring knowledge from a complex teacher to a smaller, faster student.
  • Sparse LIDAR supervision and physical orientation priors constrain predicted depths or surface normals, with physical priors reported to improve depth accuracy.The supervision uses available ground-truth pixels and the berHu norm; semi-supervised accuracy nevertheless relies heavily on pose or LIDAR data.

D. Applications

Deep-learning monocular depth estimation has been applied to SLAM and visual odometry to improve mapping, recover absolute scale, and replace RGB-D sensors in dense mapping.

  • Monocular depth prediction can initialize feature-depth means and variances in SVO, supporting mapping improvements and depth-uncertainty handling.

IV. DISCUSSION

Future monocular depth estimation research must balance accuracy, transferability, and real-time performance while clarifying what depth cues networks learn. The review identifies domain shift, computational cost, scale ambiguity, and limited mechanism understanding as continuing challenges.

  • Accuracy: Accuracy remains the dominant focus, with new loss functions and network frameworks driving improvements while unsupervised methods trail effective semi-supervised methods.The review suggests stronger geometric constraints and dynamic-object-aware masking as directions for improving unsupervised methods.
  • Accuracy: Monocular-video unsupervised methods face scale ambiguity and scale inconsistency, while semantic information is proposed as a possible source for learning scale.The review also points to camera-intrinsics estimation as a promising direction.
  • Transferability: Transferability is limited because models trained and tested within one dataset can degrade severely across cameras, scenarios, or datasets.Camera parameters and domain adaptation are identified as approaches that may improve cross-domain performance.
  • Real-time performance: Deeper networks improve performance but require more computation, creating a deployment challenge for real-time estimation on embedded devices.Lightweight networks reduce parameters but can affect performance, making accuracy–speed balance an open subject.
  • Mechanisms and applications: Research on what depth networks learn and which cues they exploit remains limited, with experiments showing that current networks ignore the apparent size of known obstacles.The review connects mechanism studies with potential improvements in accuracy, transferability, and real-time performance.

V. CONCLUSION

This review surveys deep learning-based monocular depth estimation across supervised, unsupervised, and semi-supervised training, including loss functions and network frameworks. It also discusses current topics, challenges, and future research directions.

  • V. CONCLUSION: The review organizes monocular depth estimation research by supervised, unsupervised, and semi-supervised training manners, alongside loss functions and network frameworks.It concludes by discussing current hot topics, challenges, and promising directions.
Loading 2003.06620v2…