PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Real-Time Monocular Depth Estimation for Robotics: A Survey

Xingshuai Dong, Matthew A. Garratt, Sreenatha G. Anavatti, Hussein A. Abbass

arXiv:2111.08600v1cs.RO

TL;DR

Monocular depth estimation is important for robotic 3D perception, but existing surveys lack comprehensive coverage of MDE. This paper reviews 197 articles, methods, resources, evaluations, and applications, then compares representative approaches across accuracy and computational time while identifying practical scope differences.

  • Problem

    Existing depth-estimation surveys do not comprehensively cover monocular methods and their evolution, despite MDE's role in robotic tasks.

  • Method

    The paper surveys 197 articles, organizes MDE methods across classical and deep-learning approaches, summarizes datasets, metrics, implementations, and robotics applications, and compares representative methods.

  • Results

    The comparison reports that deep learning methods outperform Make3D, supervised methods achieve the best accuracy and error metrics, and computational speed varies substantially across methods.

  • Takeaways & Limitations

    The survey helps readers navigate MDE methods and assess their suitability for accuracy-oriented or real-time robotic tasks.

Abstract

from arXiv · show

As an essential component for many autonomous driving and robotic activities such as ego-motion estimation, obstacle avoidance and scene understanding, monocular depth estimation (MDE) has attracted great attention from the computer vision and robotics communities. Over the past decades, a large number of methods have been developed. To the best of our knowledge, however, there is not a comprehensive survey of MDE. This paper aims to bridge this gap by reviewing 197 relevant articles published between 1970 and 2021. In particular, we provide a comprehensive survey of MDE covering various methods, introduce the popular performance evaluation metrics and summarize publically available datasets. We also summarize available open-source implementations of some representative methods and compare their performances. Furthermore, we review the application of MDE in some important robotic tasks. Finally, we conclude this paper by presenting some promising directions for future research. This survey is expected to assist readers to navigate this research field.

I. INTRODUCTION

Monocular depth estimation derives dense depth maps from single RGB images, offering lightweight perception for robotics while avoiding active-sensor and stereo limitations. This survey addresses the lack of a comprehensive MDE review by organizing methods, resources, evaluation, and robotic applications.

  • Depth estimation infers a dense depth map from an input image, supporting 3D structure recovery for ego-motion estimation, obstacle avoidance, and scene understanding.
  • RGB cameras are lightweight and information-rich, whereas active sensors face range, sparsity, precision, size, or energy limitations and stereo requires calibration and has baseline-dependent accuracy.
  • The survey restricts its scope to dense monocular depth estimation and reviews 197 relevant articles published from 1970 through 2021.
  • It classifies MDE methods into structure-from-motion, traditional handcrafted-feature, and deep-learning-based categories.
  • Existing depth-estimation surveys generally mix monocular and stereo methods or cover only a small set of deep models, leaving a comprehensive MDE survey gap.
  • The paper additionally summarizes datasets, metrics, and implementations and reviews MDE applications in ego-motion estimation, obstacle avoidance, and scene understanding.

B. Performance Evaluation

The survey defines common metrics for comparing estimated and ground-truth depth maps, including relative errors, RMSE variants, threshold accuracy, and scale-invariant error.

  • Metric definitions: Evaluation compares an estimated depth map D with ground-truth D∗ over pixels having valid values.The pixel count N includes locations with both valid ground-truth and estimated depth values.
  • Metric definitions: Absolute Relative Difference averages the L1 depth error scaled by the estimated depth.
  • Metric definitions: Squared Relative Difference averages the L2 depth error scaled by the estimated depth.
  • Metric definitions: RMSE measures the square root of the mean squared depth error, while RMSE log evaluates the corresponding error in logarithmic space.
  • Metric definitions: Threshold accuracy reports the percentage of predicted pixels whose relative error is within thresholds typically set to 1.25, 1.25^2, and 1.25^3.
  • Metric definitions: The scale-invariant error measures relationships between scene points independently of absolute global scale by aligning estimates to ground truth in log space.

C. Datasets

The paper situates depth-estimation datasets as essential for development and evaluation, highlighting representative benchmarks and complementary SfM-based data-generation approaches.

  • Datasets: Datasets play a critical role in developing and evaluating depth-estimation methods.
  • Datasets: Representative released datasets include Make3D, NYU depth v2, KITTI, Cityscapes, and Virtual KITTI.
  • Structure from Motion: SfM predicts camera motion or scene structure from image sequences captured from different viewpoints using feature extraction, matching, and outlier rejection.
  • Structure from Motion: SfM depth can be computed from image-region motion together with vehicle translation, with scale and camera translation in depth defining the formulation.
  • Structure from Motion: A sparse SfM method uses 5 to 8 monocular images, multi-scale feature detection, feature matching, two-view geometry, and metric reconstruction.
  • Structure from Motion: A plane-sweep SfSM method produces dense depth maps, but processing takes about 10 minutes per image; ORB reduces runtime to minutes without real-time operation.

B. Summary

The survey summarizes monocular depth estimation’s evolution from handcrafted visual cues and SfM toward deep learning, while emphasizing the accuracy–density–efficiency limits of SfM.

  • SfM limitations: SfM depth quality depends on feature matching, while textureless or low-contrast scenes yield fewer features and usually sparse depth maps.
  • SfM limitations: Sparse SfM maps support localization but are insufficient for applications such as autonomous flight requiring dense maps for frontal-obstacle avoidance.
  • Historical progression: Traditional research progressed from handcrafted feature methods to state-of-the-art deep learning-based methods, as summarized by the survey’s milestones.
  • Traditional methods: Earlier monocular methods infer depth from cues including texture, occlusion, defocus, color or haze, surface layout, and known-object size.
  • Traditional methods: Handcrafted methods over-segment images into superpixels, compute visual and geometric features, estimate depth cues, and refine superpixel depths.
  • Historical progression: Early learning-based approaches used global image structure or patch-level absolute and relative depth features for monocular prediction.
  • Historical progression: Later methods combined depth with semantic labels, unified semantic-depth reasoning, retrieved similar images for label transfer, or refined predictions using contour information.

B. Summary

Deep learning-based MDE spans supervised, unsupervised, semi-supervised, and domain-adaptation settings, with architectures ranging from multi-scale and encoder-decoder networks to CRF- and GAN-based designs. Representative methods incorporate multi-scale features, geometric constraints, discretized depth, adaptive bins, omnidirectional inputs, and recurrent modeling.

  • Handcrafted-feature methods can miss corner cases and require computationally burdensome pre- or post-processing, limiting suitability for real-time robotic control.
  • Learning settings: Supervised MDE learns a dense depth map from an RGB image and ground-truth depth by minimizing a loss between prediction and target.
  • Network architectures: Deep MDE architectures include multi-scale, encoder-decoder, CNN-CRF, and GAN designs, with encoder-decoder models using feature upsampling and convolutional decoding.
  • Architectural and objective innovations: Representative methods improve estimation through dense feature fusion, geometric or hybrid losses, ordinal depth discretization, and adaptive per-image depth bins.
  • Architectural and objective innovations: Adaptive-bin estimation combines bin centers using probabilities, producing smoother depth maps than selecting only the most likely bin center.
  • Extended inputs and models: MDE research also addresses omnidirectional images and temporal dependencies, using 360° inputs or recurrent networks to model broader views or sequential data.

2) Monocular Depth Estimation by Classification:

Classification-based MDE discretizes continuous depth into intervals and may model probability distributions, while related systems combine classification with regression or exploit semantic and geometric auxiliary tasks. These approaches use multi-scale, multi-task, and cross-dataset designs to refine depth and related scene representations.

  • Monocular Depth Estimation by Classification: Classification-based MDE discretizes continuous depth values into segments because different pixels can have different depth distributions.
  • Monocular Depth Estimation by Classification: Mean-variance loss penalizes errors in the estimated distribution mean and sharpens its variance, alongside softmax supervision.
  • Monocular Depth Estimation by Classification: Some methods jointly integrate depth interval classification and regression through a joint-loss layer to produce the final depth map.
  • Multi-task learning: Multi-task MDE jointly estimates depth with semantic segmentation or surface normals because these outputs reveal complementary scene layout and object geometry.
  • Multi-task learning: Cross-dataset training can combine depth and semantic information from heterogeneous datasets when per-pixel labels for both tasks are difficult to collect.
  • Multi-task learning: GeoNet jointly refines depth and surface normals through depth-to-normal and normal-to-depth streams, while other frameworks connect depth with object detection or 3D pose.

4) Real-Time Supervised Monocular Depth Estimation:

Real-time supervised MDE reduces model complexity for embedded deployment, trading accuracy against speed and training-data requirements. Lightweight networks and pruning can achieve high frame rates, while supervised training still depends on costly calibrated depth ground truth.

  • Real-Time Supervised Monocular Depth Estimation: Complex deep networks challenge real-time requirements, motivating lightweight architectures for embedded platforms.
  • Real-Time Supervised Monocular Depth Estimation: A non-bottleneck-1D-based network runs about 30fps on an Nvidia-TX2 GPU, but its accuracy is inferior.
  • Real-Time Supervised Monocular Depth Estimation: 178 fps on an Nvidia-TX2 GPU was reported for a pruned lightweight encoder-decoder on NYU, with RMSE 0.604 and δ1 0.771.
  • Training-data constraints: Supervised methods can achieve high MDE accuracy but require vast amounts of ground-truth depth images for training.
  • Training-data constraints: Collecting supervised depth data requires LIDAR or RGB-D sensors, whose calibration errors can produce inaccurate ground truth.
  • Unsupervised alternatives: Unsupervised methods instead use stereo images or video sequences and formulate depth estimation as image reconstruction with depth as an intermediate product.

1) General Unsupervised Methods:

General unsupervised MDE learns depth from stereo pairs or monocular video through reconstruction, view synthesis, geometric consistency, and related auxiliary signals rather than ground-truth depth. Later methods jointly estimate depth with pose, optical flow, motion, or segmentation while addressing scale inconsistency and dynamic scenes.

  • General Unsupervised Methods: Early unsupervised MDE uses image pairs with known camera motion or stereo correspondence, optimizing photometric or color-constancy reconstruction errors.
  • General Unsupervised Methods: Monocular-video methods use view synthesis, semantic guidance, global orientation, or scale-invariant constraints to supervise depth without direct depth labels.
  • Joint depth and pose: Joint depth-and-pose frameworks inverse-warp source views to reconstruct targets, using predicted depth and relative camera pose as supervisory signals.
  • Joint depth and pose: Scale inconsistency across sequences can degrade joint depth-and-pose estimation, motivating a geometry-consistency loss for scale-consistent predictions.
  • Dynamic scenes and occlusions: A self-discovered mask assigns lower weights to inconsistent pixels to reduce the influence of moving objects and occlusions during training.
  • Joint learning: Unsupervised methods also jointly learn depth with optical flow, camera pose, motion segmentation, or dynamic-object motion using cross-task consistency and competitive collaboration.

3) Adversarial Learning Based Methods:

Adversarial learning formulates unsupervised monocular depth estimation through generator–discriminator training, while lightweight pyramidal designs target real-time operation.

  • Adversarial learning: GAN-based unsupervised MDE trains a generator to infer depth and a discriminator to distinguish synthesized warped images from real inputs.The generator is encouraged to improve depth estimates because they affect synthesized-image quality.
  • Adversarial learning: Joint adversarial learning can combine depth estimation, ego-motion, and spatial-temporal geometric constraints.A depth-pose network with direct visual odometry produces synthesized images and supplies gradients to the depth network.
  • Real-time operation: Complex adversarial architectures often require high-performance GPUs for real-time speed, motivating simpler networks such as MiniNet.MiniNet uses a simple encoder and multiple small pyramidal decoders to generate multi-scale disparity maps.
  • Unsupervised learning: Unsupervised MDE uses stereo images or monocular sequences and geometric constraints instead of ground-truth depth during training.This avoids the expense of collecting depth maps, but accuracy is limited by stereo construction.
  • Semi-supervised learning: Semi-supervised methods combine small labeled datasets with larger unlabeled datasets to improve accuracy while reducing dependence on ground-truth depth.Typical training first produces pseudo-labels from a model trained on labeled data, then links them with unlabeled images.

1) General Semi-supervised Methods:

Semi-supervised MDE combines limited labeled information with abundant unlabeled data, while synthetic-data methods address data availability but face domain-generalization challenges.

  • General semi-supervised methods: Semi-supervised MDE combines supervised and unsupervised objectives using sparse depth or image-depth pairs alongside unlabeled stereo or monocular images.Reported frameworks include adversarial learning, semantic-task sharing, and confidence-guided supervision.
  • General semi-supervised methods: Semi-supervised learning improves accuracy over unsupervised learning by adding a small amount of labeled data, but still requires substantial unlabeled data.The labeled information may consist of sparse depth or semantic maps.
  • Synthetic data: Synthetic 3D scenes can provide large collections of images and corresponding depth maps for training MDE models.Modern graphics platforms and game engines make such scene generation practical.
  • Synthetic data: Models trained on synthetic data normally generalize poorly to real scenes because of the inherent domain gap.The domain gap reflects differences between source and target domains that can impair model performance.
  • Synthetic data: Domain adaptation trains on synthetic data and adapts the resulting MDE networks toward real data to reduce cross-domain effects.The reviewed approaches include fine-tuning and other adaptation strategies.

1) Domain Adaptation via Fine-tuning:

Fine-tuning adapts MDE models from source data to target-domain images, while domain-adaptation strategies reduce reliance on costly real-world depth collection.

  • Domain adaptation via fine-tuning: Fine-tuning methods first train an MDE network on source-domain images, such as synthetic data, and then adapt it using target-domain images.DispNet is identified as an early depth-estimation example of this strategy.
  • Domain adaptation via fine-tuning: Guo et al. train stereo matching on synthetic data, fine-tune it on real data, and use its disparity maps to supervise MDE.Their method reportedly outperforms several supervised and unsupervised baselines.
  • Domain adaptation via fine-tuning: Fine-tuning generally requires target-domain ground-truth depth, which is available for only a few benchmark datasets and is costly to collect in practice.Acquisition requires expensive sensors such as LIDAR and accurate calibration.
  • Domain adaptation via fine-tuning: Data-transformation methods alter source or target image style to reduce visual discrepancies between domains.Some methods use GAN-based style transfer before depth estimation, while geometry-aware variants incorporate epipolar structure.
  • Domain adaptation via fine-tuning: Data-transformation methods can lose accuracy in environments differing from the source in appearance or context, and illumination or saturation changes can impair transformed images.These conditions constrain robustness across target environments.
  • Domain adaptation via fine-tuning: Domain adaptation reduces the cost of acquiring real-world ground-truth depth and addresses the limited availability of labeled real data.The survey presents it as a promising technique for adapting synthetic-trained networks to real data.

B. Sparse depth map from SLAM

Depth-completion methods fuse RGB images with sparse SLAM depth to produce dense maps, while benchmark comparisons expose accuracy–runtime trade-offs across MDE categories.

  • Sparse depth map from SLAM: RGB images and sparse depth maps are fused through convolutional or encoder–decoder architectures to estimate dense depth and confidence outputs.Examples process RGB and sparse depth separately before concatenation or feature fusion.
  • Sparse depth map from SLAM: Depth completion benefits from complementary inputs: RGB supplies appearance and scene structure, while sparse depth supplies rough geometric information.The survey reports better accuracy than monocular depth estimation methods.
  • Accuracy: KITTI comparisons summarize 42 methods using error, accuracy, and GPU running-time metrics.Abs Rel, Sq Rel, RMSE, and RMSE log are lower-is-better; δ1, δ2, and δ3 are higher-is-better.
  • Accuracy: Deep learning methods outperform the traditional Make3D method, while supervised learning achieves the best aggregate error and accuracy metrics among the four categories.The reported ordering is supervised, semi-supervised, domain adaptation, then unsupervised.
  • Accuracy: Small amounts of labeled data can substantially improve depth-network performance, with semi-supervised and domain-adaptation methods showing similar accuracy but variable error metrics.The comparison specifically highlights differences in Sq Rel, RMSE, and RMSE log.
  • Computational time: Real-time suitability depends strongly on model size and hardware: Wang et al. report about 14fps on Jetson AGX Xavier with 1.75M parameters.The survey contrasts this with a 78M-parameter network considered unsuitable for single-GPU real-time inference.

VIII. APPLICATIONS IN ROBOTICS

MDE provides dense depth from a single RGB camera for robotic perception, supporting ego-motion estimation, obstacle avoidance, and scene understanding. Lightweight implementations extend these capabilities to resource-limited platforms, but deployment and generalization remain constrained.

  • Dense monocular depth maps can provide surface distances from a single sensor for autonomous navigation and obstacle avoidance.
  • Lightweight CNN-based MDE systems achieve real-time speed on mobile platforms while producing more accurate depth maps than traditional non-real-time methods.The cited platforms include Nvidia-TX2 and Jetson AGX Xavier GPUs.
  • A. Ego-motion Estimation: CNN-predicted absolute scale helps monocular visual odometry and SLAM address scale ambiguity and drift.Depth priors can also improve depth-filter initialization in CNN-SVO.
  • A. Ego-motion Estimation: Integrating lightweight depth estimation with ORB-SLAM2 enables real-time tracking and mapping on mobile platforms while reducing scale drift and improving monocular SLAM accuracy.
  • B. Obstacle Avoidance: Depth maps support obstacle avoidance by representing obstacle distances and enabling collision-free waypoint selection.An effective depth map combines perceived obstacle distance, braking distance, and confidence-derived measurement error before being converted into collision-free regions.
  • B. Obstacle Avoidance: Obstacle-avoidance CNNs trained from ground-vehicle imagery may generalize poorly to UAV views because their viewing angles and motion types are limited.
  • B. Obstacle Avoidance: Depth errors can substantially affect collision-point and time-to-collision estimates for moving objects, while dynamic 3D obstacle avoidance using estimated depth remains scarcely studied.

C. Scene Understanding

Scene understanding uses monocular depth to recover 3D geometric structure from images and support tasks such as semantic interpretation and UAV landing-zone detection. The survey identifies dataset diversity, real-time efficiency, labeling cost, multimodal fusion, and interpretability as continuing challenges.

  • C. Scene Understanding: Depth maps encode scene 3D structure, helping resolve geometric ambiguities and avoid physically implausible scene labels.
  • C. Scene Understanding: A depth-estimation network can infer relative scene depth from single images and then be fine-tuned for city-scene understanding tasks such as semantic segmentation.
  • C. Scene Understanding: A two-stage UAV landing-zone method estimates depth from aerial images before classifying possible landing zones.The extracted depth map is passed to a second CNN for landing-zone detection.
  • Existing public datasets do not sufficiently represent moving objects, clutter, occlusions, illumination changes, and weather changes in real-world scenes.
  • Highly accurate MDE networks can be computationally expensive for mobile devices, making accuracy-efficiency balance important for small robots.
  • Synthetic-to-real domain adaptation is proposed because supervised MDE requires costly ground-truth depth from LIDAR or RGB-D cameras.
  • Semi-supervised MDE seeks to reduce labeling costs while improving over the lower accuracy of unsupervised methods by combining limited labeled and abundant unlabeled images.
  • Open questions include efficiently fusing sparse depth, optical flow, and surface normals, handling independently moving objects, and explaining what depth networks learn.
Loading 2111.08600v1…