Source-linked AI summary
Single Image Depth Estimation: An Overview
Alican Mertan, Damien Jade Duff, Gozde Unal
TL;DR
Single-image depth estimation is ambiguous because many 3D scenes can produce the same image. This paper surveys learning-based solutions and their research directions, reporting that investigated architectures yield similar findings; those investigations also identify cues models use while leaving their generalization beyond training domains in question.
Problem
Single-image depth estimation is inherently ambiguous because many different 3D scenes can produce the same 2D image.
Source: §1 · p. 1Method
The paper surveys learning-based SIDE solutions, outlining research categories and highlighting their progression and common themes.
Source: §1 · p. 2Results
The investigated architectures produced similar findings, indicating that those findings were not dependent on a particular architecture.
Source: §4.1 · p. 15Takeaways & Limitations
Investigations identify ground-plane connections, vanishing points, and object edges as cues models use to estimate depth.
Source: §4.1 · p. 15How well learned cues generalize outside their training domain remains questionable, and failures on surfaces such as mirrors can disrupt estimates.
Source: §4.1 · p. 15, §5 · p. 16
Abstract
from arXiv · showhide
We review solutions to the problem of depth estimation, arguably the most important subtask in scene understanding. We focus on the single image depth estimation problem. Due to its properties, the single image depth estimation problem is currently best tackled with machine learning methods, most successfully with convolutional neural networks. We provide an overview of the field by examining key works. We examine non-deep learning approaches that mostly predate deep learning and utilize hand-crafted features and assumptions, and more recent works that mostly use deep learning techniques. The single image depth estimation problem is tackled first in a supervised fashion with absolute or relative depth information acquired from human or sensor-labeled data, or in an unsupervised way using unlabelled stereo images or video datasets. We also study multitask approaches that combine the depth estimation problem with related tasks such as semantic segmentation and surface normal estimation. Finally, we discuss investigations into the mechanisms, principles, and failure cases of contemporary solutions.
1 INTRODUCTION
Single-image depth estimation maps one RGB image to a metric depth value for every pixel, but the mapping is ambiguous because many 3D scenes can produce the same image. The overview organizes learning-based approaches and emphasizes how human depth cues and learned statistical priors inform the field.
Problem and scope
- A single RGB image must be mapped to a metric depth value for every pixel, a task the paper calls Single Image Depth Estimation (SIDE).The example depth map uses blueish colors for nearer pixels and reddish colors for farther ones.Source: §1 · p. 1
- Many different 3D scenes can produce the same 2D image, making monocular depth estimation inherently ambiguous.The paper frames this as a one-to-many mapping from RGB images to depth maps.Source: §1 · p. 1
Human depth cues
- Humans estimate depth from static pictorial cues including occlusion, perspective, size, texture gradient, atmospheric effects, shading, and height.The size cue draws on prior knowledge of familiar object sizes, which can be overturned by comparing relative sizes in an image.Source: §1 · p. 1
Overview
- The overview organizes learning-based SIDE research into categories, tracing the progression of solutions and emphasizing shared themes and problem-specific insights.Figure 3 presents the structure used to discuss the works.Source: §1 · p. 2
2 EARLY DAYS
Early work approached single-image depth through hand-crafted features, scene assumptions, and statistical models. These approaches introduced recurring ideas such as multi-scale information, priors, and smoothing, while also revealing challenges in data and domain variation.
Early scene reconstruction
- Hoeim et al. reconstructed outdoor scenes by classifying superpixels as sky, ground, or vertical objects, then placing objects upright on the ground plane.The resulting scenes omitted detail but were described as visually pleasing.Source: §2 · p. 2
Early task-specific depth
- Michels et al. estimated the nearest obstacle distance in each image direction to guide a remote-control car through an uncontrolled outdoor environment.Their supervised system used handcrafted features and synthetic data; its obstacle-distance task differs from dense pixelwise SIDE.Source: §2 · p. 2
Recurring design ideas
- Early methods used log-space errors for depth and relative-depth comparisons, while image coordinates and global context supplied additional scene cues.Log-space errors are more forgiving of mistakes at larger depths; spatial cues can also introduce bias.Source: §2 · p. 2
Training data
- Synthetic data can provide automatically labeled examples and varied textures or lighting, but synthetic-to-real adaptation and diverse natural-scene generation remain challenges.Obtaining labeled RGB images and depth maps is costly, motivating synthetic data use.Source: §2 · pp. 2–3
Patch-based estimation
- Saxena et al. estimated metric depth from handcrafted patch features with a supervised Markov Random Field, incorporating multiple scales, neighboring patches, and image-column information.The model learned separate row-specific parameters for absolute depth, uncertainty, and smoothing between similar neighboring patches.Source: §2 · p. 3
Recurring design ideas
- Multi-scale processing simulates viewing scenes at different distances, while hand-designed filters, assumptions, and loss functions encode human knowledge into models.Smoothing terms encourage similar depths in neighboring pixels but should avoid blurring true depth discontinuities.Source: §2 · pp. 3–4
Superpixel and planar assumptions
- Saxena et al. assumed complex scenes could be represented by small planar surfaces, leading many methods to estimate depth over superpixels and reduce the number of estimation points.Similar-looking neighboring superpixels are often treated as coplanar to smooth depth estimates.Source: §2 · p. 4
Scene and semantic priors
- Saxena et al. [16] also modeled 3D connectivity, coplanarity, and straight lines, while semantic information and environment-specific structure offered further priors for depth estimation.Indoor and outdoor scenes differ statistically, and many methods were designed for only one environment type.Source: §2 · pp. 2, 4
3 SIDE LANDSCAPE
The section introduces commonly used SIDE datasets and organizes research by its primary objective. The three categories cover metric performance, reducing reliance on labeled data, and operation in the wild.
Datasets
- NYU provides 1,449 densely labeled indoor images from 464 scenes, while KITTI contains more than 93,000 outdoor driving images with depth maps.NYU also has a raw collection of over 400,000 images and corresponding depth maps.Source: §3 · p. 4
- Sun RGB-D includes 10,335 indoor images with depth and several other annotations, while SceneNet RGB-D [24] provides five million synthetically generated indoor images with depth maps.SceneNet varies room conditions, lighting, and textures.Source: §3 · p. 4
Research categories
- The review groups SIDE research into improving metric performance, eliminating the need for labeled data, and working in the wild.The first category includes architectural and multitask approaches; the second covers synthetic data and unsupervised learning.Source: §3 · p. 4
3.1 Main consideration: increasing the metric performance
This section surveys methods for improving single-image depth estimation through CNN architectures, losses, image-specific or geometric knowledge, and adaptive or ordinal depth representations. It also covers multitask approaches that use cross-task relationships and consistency to refine depth, including methods for unlabeled and real-image settings.
3.1.1 CNN based works
- Eigen et al. combine global-context coarse prediction with local refinement and scale-invariant loss, while Laina et al. use a real-time residual network with higher-resolution outputs and BerHu loss.The scale-invariant loss addresses scale ambiguity; BerHu acts as L1 below a threshold and L2 above it.Source: §3.1.1 · pp. 5–6
- Cao et al. classify depth ranges and use confidence-guided refinement, while Fu et al. use ordinal threshold predictions to avoid exact-depth regression and preserve output resolution.Depth classes have ordinal relations, which classification-based approaches can exploit to improve performance.Source: §3.1.1 · pp. 6–7
- Bhat et al. [28] learn image-adaptive depth bins and per-pixel bin probabilities, combining bin centers into smoother depth maps while adapting to each image’s depth range.Their method uses a Vision Transformer variant, scaled scale-invariant loss, and chamfer-distance regularization for bin centers.Source: §3.1.1 · p. 7
3.1.2 Multitasking
- Liu et al. use semantic labels and class-specific depth models, while Wang et al. jointly train depth and segmentation, improving both tasks individually.Liu et al. also derive hand-crafted depth constraints from horizon estimation and scene decomposition.Source: §3.1.2 · pp. 7–8
- Ladicky et al. address missing perspective information by classifying pixels by depth and semantic class, using image scaling to simplify joint depth and class estimation.Their method uses depth’s inverse relationship to image scaling to determine the scaling that maps a pixel to a chosen depth.Source: §3.1.2 · pp. 7–8
- Qi et al. use local tangent-plane constraints to refine depth and surface normals, exploiting their geometric relationship without adding learnable parameters to the refinement.The method derives normals from local depth neighborhoods and refines depth from initial normals.Source: §3.1.2 · p. 8
- Xu et al. combine intermediate outputs for complementary tasks as pseudo-multimodal inputs to improve each task’s estimation beyond using RGB alone.Their two-stage framework first encodes the image, then decodes depth, surface normals, contours, and semantic labels.Source: §3.1.2 · pp. 8–9
- Zhang et al. alternate depth and semantic segmentation predictions across scales, using cross-task knowledge rather than only shared layers.Their decoder uses attention-based gating to pass earlier depth and segmentation outputs to higher scales.Source: §3.1.2 · p. 9
- To address feature ambiguity across tasks, Zhang et al. propagate affinities derived from task-specific features, combining them with learnable task-specific weights.Affinity matrices are calculated deterministically without learnable parameters; the propagated features produce final predictions.Source: §3.1.2 · p. 9
- Zhang et al. [35] found that 50%–60% of pixel pairs labeled similar in one task were also similar in other tasks, with the same pattern for dissimilar pairs.Source: §3.1.2 · p. 9
- Chen et al. jointly learn unsupervised SIDE and supervised semantic segmentation with a task-conditioned shared decoder and semantic-guided self-supervised losses.The losses warp segmentation maps to penalize reconstruction inconsistency and regularize depth to favor similar values within the same object; their depth approach resembles Godard et al..Source: §3.1.2 · pp. 9–10
- Taskology [37] trains task-specific models separately and connects them with a consistency loss that requires no ground-truth labels, including for additional datasets.For depth and surface normals, applying consistency loss to real images helps bridge the domain difference from synthetic training data.Source: §3.1.2 · p. 10
3.2 Main consideration: eliminating the need for labeled data
This section surveys ways to reduce reliance on annotated real-world depth: synthetic labeled data and unsupervised learning from stereo images or video. It links label scarcity to limits on data collection, lifelong learning, and fine-tuning.
3.2.1 Usage of synthetic data
- Ren and Lee jointly estimate depth, surface normals, and instance contours using a synthetically generated dataset with labels for all three tasks.Synthetic data supplies the needed ground truths when a dataset with all task labels is hard to find.Source: §3.2.1 · p. 10
- Adversarial training encourages similar low-level feature maps for real and synthetic images to address their domain difference.A discriminator distinguishes the domains from estimator feature maps, and its adversarial loss trains the estimator.Source: §3.2.1 · p. 10
3.2.2 Unsupervised learning
- Unsupervised methods use stereo images or nearby video frames to estimate depth, warp one view toward the other, and train from image reconstruction loss.These methods assume static scenes, no occlusion or disocclusion, and Lambertian surfaces, assumptions that may not always hold.Source: §3.2.2 · pp. 10–11
- Garg et al. learn depth from calibrated stereo pairs, while Godard et al. predict both left and right disparities and use both reconstruction losses.Garg et al. calculate warping from depth and known camera motion; Godard et al. use rectified stereo pairs with known baselines.Source: §3.2.2 · pp. 10–11
- Zhou et al. train on single-camera video by estimating depth and camera motion, warping neighboring frames, and minimizing reconstruction error.They also weight each pixel's loss by a predicted belief to limit the influence of pixels where model assumptions fail, such as moving-car pixels.Source: §3.2.2 · pp. 10–11
- Wang et al. normalize predicted depth before calculating the loss to address scale-related training divergence.They also replace the pose-estimation CNN with differentiable pose optimization, reducing the number of learned parameters.Source: §3.2.2 · pp. 11–12
- Ranjan et al. [51] jointly learn depth, camera motion, optical flow, and motion segmentation through unsupervised Competitive Collaboration.A moderator segments static and moving regions to allocate training data to competing networks that estimate optical flow for each region.Source: §3.2.2 · p. 12
3.3 Main consideration: working in the wild
Depth estimation in the wild is difficult because absolute depth ranges vary widely and diverse absolute-depth labels are costly to collect. The section traces the shift to relative-SIDE, its datasets, and methods for learning pixel-pair depth relations.
3.3 Main consideration: working in the wild
- Relative-SIDE targets better generalization in the wild by estimating pixel-pair relations instead of absolute depths, which vary widely and are costly to annotate across diverse scenes.The section introduces DIW, ReDWeb, and Youtube3D as datasets with relative ground truths.Source: §3.3 · p. 12
3.3.1 Datasets
- DIW crowdsources one relative-depth pixel pair per diverse internet image; ReDWeb derives dense relative-depth annotations from internet stereo, while Youtube3D uses structure from motion on videos.The datasets are presented as routes to automatically collect and annotate data for depth estimation in the wild.Source: §3.3.1 · pp. 12–13, Figure 4 · p. 13, Figure 5 · p. 13
3.3.2 Learning from relative depth annotations
- The common framework regresses a score map from an RGB image and uses pairwise annotations to train predictions that preserve the annotated depth ordering.Chen et al. use a loss that penalizes unequal scores for equal-depth pairs and pushes unequal pairs apart in the correct direction.Source: §3.3.2 · p. 13
- For ReDWeb, Xian et al. randomly sample 3K pixel pairs per image, retain the 75% with highest loss to focus on hard pairs, and balance equal and unequal relations.Their loss uses the relative ranking objective with minor modifications.Source: §3.3.2 · p. 13
- Chen et al. omit the equality term because Youtube3D ground truths do not contain equality relations.Source: §3.3.2 · p. 13
- Xian et al. sample pairs across instance and image edges to emphasize object-boundary supervision and distinguish edges with depth discontinuities from those without.Source: §3.3.2 · p. 13
- Mertan et al. model each pixel depth as a normal distribution and learn per-pixel uncertainty through its standard deviation.Source: §3.3.2 · p. 14
3.4 Other considerably related works
These related works extend single-image depth estimation through optical cues, non-parametric image matching, sensor inputs, and 3D scene reconstruction. Their methods range from focused-image supervision to sparse depth guidance and object-based scene fitting, with stated data and scene constraints.
Depth from Optical Cues
- Gur et al. learn depth from defocus cues by reconstructing focused images, but training requires focused images, which they create from ground-truth depth for mainstream datasets.The section notes that optical cues arise from photography's physical principles and are less tied to scene semantics.Source: §3.4 · p. 14
Non-parametric
- Karsch et al. retrieve similar RGB images, warp their depth maps to a query image using SIFT flow, then merge the warped maps without learning parameters.For video, they add steps that use temporal frame relations.Source: §3.4 · p. 14
Sensor Fusion
- Liao et al. combine an RGB image with a laser-derived reference depth map and estimate a residual depth map, using regression and classification losses.The added laser scan helps alleviate scale ambiguity; the section frames sensor input as relevant to robotics, where metric performance may not ensure task success.Source: §3.4 · p. 14
- Mal and Karaman also study depth completion, using randomly sampled image points with depth values as an additional input instead of 2D laser scans.They present this as sensor super-resolution.Source: §3.4 · p. 14
3D Reconstruction
- Izadinia et al. reconstruct scenes by aligning database CAD objects and fitting an indoor room layout, then refining the scene through rendering-based optimization.Source: §3.4 · p. 15
- Despite failures when objects are missed, absent from the database, or the room is not box-shaped, the method produces qualitatively pleasing 3D scenes from complex natural images.Source: §3.4 · p. 15
4 INVESTIGATIVE WORKS
Investigative works probe learned depth cues by perturbing inputs and masking pixels, finding reliance on position, ground contact, edges, and scene geometry. The discussion cautions that learned regularities may not generalize beyond training domains.
4 INVESTIGATIVE WORKS
- Dijk and Croon [61] found networks relied more on vertical position than apparent size when estimating inserted cars, consistently across architectures and training methods.Changing camera pitch also changed estimated depth, strengthening the vertical-position finding.Source: §4 · p. 15
- Dijk and Croon [61] found that networks use ground contact and object edges, while interior filling persists even when an object's interior is emptied.Grayscale or false coloring had little effect, but uniform coloring degraded performance, suggesting part distinction mattered more than actual color or texture.Source: §4 · p. 15
- Hu et al. used learned input masks to identify informative pixels; masks highlighted edges, some small-object interiors, and outdoor vanishing points.The authors suggest vanishing points may matter because errors there are larger or because they help represent scene geometry.Source: §4 · p. 15
4.1 Discussion
- The investigative works suggest models exploit training-set regularities, including ground-plane connections, vanishing points, and object edges, whose generalization remains uncertain.The authors call for systematic study of pictorial-cue use and learning methods that help models acquire generalizable cues.Source: §4.1 · p. 15
- Similar findings across the tested architectures suggest the observed behaviors are not tied to one particular architecture.The authors therefore do not examine architecture details further.Source: §4.1 · p. 15
- The discussion links dataset bias to learned cues and argues that diverse datasets are crucial for models intended to work in the wild.These investigations are presented as justification for in-the-wild datasets discussed earlier in the paper.Source: §4.1 · p. 16
- Qualitative results in works including show depth-estimation failures on surfaces with figures and reflective surfaces such as mirrors.The authors suggest mirror handling requires identifying the mirror and ignoring its image features in that region.Source: §4.1 · p. 16
5 CONCLUSION
The paper offers an overview of SIDE research directions and investigative work, while explicitly describing itself as non-exhaustive. It identifies open questions spanning datasets, robustness, alternative methods, and evaluation.
5 CONCLUSION
- The paper presents an overview of existing SIDE solutions, research directions, and common themes, and lists open questions and future-work suggestions without claiming exhaustiveness.Its stated aim is to provide an overview of the field.Source: §5 · p. 16
- Absolute depth in the wild remains an important research direction, with a need for a more general dataset for training and testing.The paper notes promising cross-dataset results in but says the topic needs more attention.Source: §5 · p. 16
- Robustness research is needed because missing ground connections, figured surfaces, and mirrors can disrupt estimates, a concern for safe real-world deployment.The paper calls for in-depth investigation of these failures and methods to improve robustness.Source: §5 · p. 16
- The authors suggest investigating differentiable ranking-metric relaxations, [65], [66] and optimal-transport ranking approaches [67], for relative-SIDE.The discussed surrogate losses do not directly optimize ranking measures such as MAP and NDCG because those measures are not differentiable.Source: §5 · p. 16
- The authors recommend investigating relative-depth and optical-cue models, since their methods differ from the examined absolute-depth models,,,.Source: §5 · p. 16
- The paper proposes testing whether off-the-shelf relative-depth estimators plus sparse relative-depth information can yield absolute depth, including metric performance, label needs, and cross-domain adaptation,.Source: §5 · p. 16
- A systematic quantitative measure for 3D-projected depth maps is needed because methods with similar depth-map metrics may differ qualitatively in projection.Source: §5 · p. 16
- The paper proposes interactive depth learning as a future direction, suggesting agents that learn through interaction may build better task-relevant world representations.Source: §5 · p. 16