PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning Techniques for Stereo-based Depth Estimation

Hamid Laga, Laurent Valentin Jospin, Farid Boussaid, Mohammed Bennamoun

arXiv:2006.02535v1cs.CVcs.GR

TL;DR

Deep learning-based stereo depth estimation addresses longstanding difficulties in recovering depth from RGB images, especially where traditional matching struggles. The paper surveys more than 150 studies, organizes their pipelines and training choices, and concludes that these techniques were achieving state-of-the-art results while remaining constrained by scope and resolution limitations.

  • Problem

    Traditional stereo matching is limited in occlusions, featureless regions, and highly textured repetitive regions, motivating learning-based depth estimation.

  • Method

    The paper provides a comprehensive survey and taxonomy of deep learning methods for stereo-based depth estimation, covering their pipelines, architectures, training data, and training procedures.

  • Results

    More than 150 papers published from 2014 to 2019 are reviewed, and the survey reports that deep learning stereo techniques were achieving state-of-the-art results.

  • Takeaways & Limitations

    The survey helps readers navigate an emerging field by comparing methods, datasets, training strategies, generalization, and performance across standard and novel images.

  • Takeaways & Limitations

    Monocular and video-based depth estimation are beyond scope, and high-resolution processing remains constrained by memory, speed, and limited depth resolution.

Abstract

from arXiv · show

Estimating depth from RGB images is a long-standing ill-posed problem, which has been explored for decades by the computer vision, graphics, and machine learning communities. Among the existing techniques, stereo matching remains one of the most widely used in the literature due to its strong connection to the human binocular system. Traditionally, stereo-based depth estimation has been addressed through matching hand-crafted features across multiple images. Despite the extensive amount of research, these traditional techniques still suffer in the presence of highly textured areas, large uniform regions, and occlusions. Motivated by their growing success in solving various 2D and 3D vision problems, deep learning for stereo-based depth estimation has attracted growing interest from the community, with more than 150 papers published in this area between 2014 and 2019. This new generation of methods has demonstrated a significant leap in performance, enabling applications such as autonomous driving and augmented reality. In this article, we provide a comprehensive survey of this new and continuously growing field of research, summarize the most commonly used pipelines, and discuss their benefits and limitations. In retrospect of what has been achieved so far, we also conjecture what the future may hold for deep learning-based stereo for depth estimation research.

1 INTRODUCTION

The article surveys the rapidly developing use of deep learning for stereo-based depth estimation, addressing limitations of traditional matching and organizing the field’s methods and evidence.

  • Depth estimation from RGB images is an ill-posed problem with applications spanning robotics, autonomous driving, augmented reality, and medical diagnosis.
  • Traditional stereo methods remain limited by occlusions, featureless regions, and highly textured regions with repetitive patterns.
  • The survey reviews more than 150 deep learning papers published between January 2014 and December 2019.
  • It provides a comprehensive taxonomy, describes common pipelines, and discusses similarities and differences between methods within each pipeline.
  • The review analyzes training data, network architectures, reconstruction performance, training strategies, generalization ability, and comparative performance on public datasets and new scenarios.

2 SCOPE AND TAXONOMY

The survey defines its scope around deep learning methods that estimate depth from stereo or multiple RGB views and organizes them by their reconstruction pipelines.

  • The paper covers stereo with n = 2 images and multi-view stereo with n > 2 images, while excluding monocular and video-based depth estimation.
  • Learning-based reconstruction trains a predictor fθ to infer a depth map from input images while minimizing distance from the unknown real depth map.
  • One method category explicitly learns pixel correspondences across views, which can be converted into optical flow or disparity and then depth.
  • A second category uses end-to-end trainable stereo pipelines, including direct depth regression and staged methods that mimic traditional stereo matching.
  • Performance depends on network architecture, training datasets, loss functions, and supervision modes including supervised, weakly supervised, and self-supervised training.

3 DATASETS

The survey organizes depth-estimation datasets by size, resolution, annotation type, and domain gap, emphasizing how these properties affect deep learning stereo methods. Dataset construction remains constrained by the difficulty of obtaining large, varied, accurately annotated data.

  • Dataset characteristics: Dataset reviews cover size, spatial and depth resolution, annotation type, and domain gap.These dimensions structure the survey's discussion of datasets for training and testing deep learning-based depth estimation algorithms.
  • Dataset size: 200 stereo pairs in the KITTI datasets can yield thousands of patch-based CNN samples, but only one sample for an end-to-end network.End-to-end architectures therefore require larger datasets, while dense 3D annotation remains time consuming; ApolloScape and A2D2 use LIDAR for dense labels.
  • Dataset construction: Synthetic data expands viewpoints, poses, lighting conditions, and scene variety through rendered 2D and 2.5D views and overlays.Examples include FlyingThings3D, Monkaa, Driving, OmniThings, OmniHouse, and MVS-Synth.
  • Dataset construction: Generating large synthetic datasets with varied real-world appearance and motion is not trivial, motivating unsupervised training and reconstructed annotations such as MegaDepth.MegaDepth uses structure-from-motion and multiview stereo with Internet photo collections, and adds ordinal depth relations from semantic segmentation.
  • Annotation and domain gap: Ordinal depth labels are simpler and faster to obtain than exact Euclidean depths, but synthetic-to-real domain shift can reduce real-world accuracy.Ordinal relations indicate which of two pixels is closer, farther, or equally distant; differing real and synthetic conditions contribute to domain shift.

4 DEPTH BY STEREO MATCHING

Deep stereo depth estimation follows the traditional matching pipeline while replacing hand-crafted components with learned features, similarity measures, and trainable modules. The surveyed variants improve receptive fields, multiscale processing, training, and regularization, but remain constrained by data requirements and computational artifacts.

  • Pipeline formulation: Stereo reconstruction minimizes a matching-cost and regularization energy to estimate disparity, which is converted into depth through triangulation.For rectified pairs, disparity indexes corresponding horizontal pixels; discretization produces a W × H × nd cost volume.
  • Pipeline formulation: Traditional stereo matching comprises feature extraction, cross-image feature matching, disparity computation, and disparity refinement or post-processing.The first two blocks construct the cost volume, while later blocks regularize, estimate, and refine disparity.
  • Learned matching: Early deep methods replace hand-crafted features with CNN features and compute patch similarity using distances, correlations, or jointly trained decision networks.Decision networks learn the similarity measure from data and are more accurate than correlation layers but significantly slower.
  • Network architecture variants: Architecture variants use residual connections, dilated convolutions, spatial pyramid pooling, and multistream designs to improve training, receptive fields, resolution, or multiscale matching.Multistream networks process multiple scales in one forward pass, but require one stream per scale.
  • Training procedures: Supervised deep stereo methods can perform well, but they require very large labeled datasets that are difficult or costly to collect and may contain noisy sensor-derived ground truth.Synthetic augmentation can mitigate data scarcity, although handcrafted synthesis may not capture target-scene stereo regularities.
  • Regularization and disparity estimation: Learned regularization can replace manually tuned penalties, while scanline-based SGM remains vulnerable to streaking artifacts and high memory use.SGM-Net predicts pixelwise penalties and reports better penalties than hand-tuned methods.

5 END-TO-END DEPTH FROM STEREO

End-to-end stereo methods learn disparity directly from image pairs through unified networks. They include fast encoder-decoder regressors and approaches that retain explicit stereo-matching stages.

  • End-to-end stereo networks train a pipeline jointly to estimate disparity from the input images.
  • Early methods stack the left and right images into a 6D volume and use a single encoder-decoder to regress disparity.
  • Single encoder-decoder regressors are fast at runtime but require large amounts of difficult-to-obtain training data.
  • Other end-to-end methods mimic traditional stereo matching by decomposing the problem into multiple stages.

5.1 Feature learning

Feature-learning methods process whole images to produce feature maps for cross-view matching. They use either image-branch or disparity-branch architectures, with design trade-offs in flexibility and context capture.

  • Feature-learning networks process entire images in one forward pass and produce feature maps at the input or a lower resolution.
  • Multi-branch image networks assign one branch to each input image and restrict correspondence searches to horizontal scanlines after rectification.
  • Multi-branch disparity networks assign branches to disparity levels, comparing a reference image with right-image re-projections at different depth planes.
  • Disparity-branch architectures cannot vary the number of disparity levels without changing the network and retraining it.
  • Feature extractors use fully convolutional or residual networks, with dilated convolutions and multi-scale modules adding broader unary-feature context.

5.2 Cost volume construction

Stereo systems convert learned features into cost volumes, regularize them with spatial and disparity context, and estimate disparity from the refined volumes. Architectures trade richer representations and context for memory and computation.

  • 5.2 Cost volume construction: Matching scores form cost volumes that may be 3D, 4D, or hybrid, commonly at 1/8-th of the input resolution.
  • 5.2.1 3D cost volumes: A 3D cost volume compares left and right features across a predefined disparity range using L1, L2, or correlation distances.
  • 5.2.1 3D cost volumes: Raw cost volumes are noise-contaminated, so regularization incorporates spatial and/or disparity context before disparity estimation.
  • 5.2.1 3D cost volumes: 2D convolutions efficiently regularize 3D volumes along spatial dimensions, while recurrent regularization additionally processes context along disparity.
  • 5.2.1 3D cost volumes: 3D convolutions regularize cost volumes across spatial and disparity dimensions, but memory requirements motivate low-resolution estimation and progressive refinement.
  • 5.2.2 4D cost volumes: 4D cost volumes preserve feature dimensions so subsequent networks can learn similarity measures instead of relying on hand-crafted ones.
  • 5.2.2 4D cost volumes: 4D volumes contain richer information than 3D volumes, but concatenated features require more parameters to learn feature similarity.
  • 5.2.2 4D cost volumes: 3D convolutions regularize 4D volumes by exploiting height, width, and disparity correlations to produce 3D cost volumes.

5.3 Disparity computation

Disparity computation converts a regularized cost volume into pixelwise estimates using discrete or differentiable operators. Soft argmin enables sub-pixel estimates but depends on distributional assumptions.

  • Argmin selects the disparity with minimum cost, but it is nondifferentiable and cannot provide sub-pixel accuracy.
  • Soft argmin is differentiable and approximates the sub-pixel MAP solution when the disparity distribution is unimodal and symmetric.
  • When disparity distributions are multimodal, soft argmin can blend modes, producing estimates far from all modes and over-smoothing boundaries.
  • The sub-pixel MAP approximation computes a weighted mean around the disparity with maximum posterior probability.
  • The sub-pixel MAP method is used only at inference and permits changing the disparity range without retraining.

5.4 Variants

The survey describes variants that address low resolution, noise, missing details, boundary smearing, computational cost, and limited high-resolution processing in stereo depth pipelines.

  • Resolution and quality: Initial disparity estimates can be low-resolution, incomplete, noisy, over-smoothed, and missing fine details, especially at object boundaries.Variants target resolution, runtime, disparity completion, and denoising.
  • Resolution and quality: High-resolution regression is computationally expensive and difficult to train because upscaling cost volumes requires many parameters.State-of-the-art methods therefore face memory or speed limitations on high-resolution imagery.
  • Resolution and quality: Bottom-up methods refine disparity locally using patches or regions, then merge estimates when needed.Split-and-merge methods estimate regional depths and fuse them in the Fourier domain.
  • Resolution and quality: Top-down methods progressively upsample low-resolution disparity estimates and use residual refinement to recover small details and thin structures.Refinement can be cascaded through multiple blocks.
  • Resolution and quality: Coarse-to-fine cost-volume hierarchies can refine both spatial and disparity resolution while processing high-resolution inputs.Yang et al. construct four 4D cost volumes with increasing spatial and depth resolutions.
  • Completion and denoising: Depth completion and denoising address noisy, incomplete disparities and boundary smearing using post-processing, additional training data, or alternative depth representations.Depth Coefficients are designed to help convolutions avoid inter-object depth mixing.
  • Efficiency: PatchMatch unrolled as an RNN reduced KITTI2015 computation time from 600ms to 60ms per image while achieving comparable performance to state-of-the-art methods.The approach treats each unrolling step as an iteration of PatchMatch.

5.5 Learning confidence maps

Confidence estimation identifies likely disparity failures and supports sparsification, replacement, or refinement. Methods range from consistency-based and supervised predictors to locally and globally reasoned models and algorithm ensembles.

  • Purpose and uses: Confidence maps are used to detect and remedy failure cases by removing potential errors and replacing them with reliable neighboring disparities.They can also guide disparity refinement pipelines.
  • Estimation methods: Traditional confidence estimation relies on assumptions and heuristics, whereas recent methods learn confidence directly from disparity space.Learned confidence can be ad hoc or integrated end-to-end with disparity estimation.
  • Estimation methods: Left-right consistency compares left and right disparity maps after warping one into the other, making it suitable for detecting occlusions.The pixel-wise difference is converted into probabilities.
  • Estimation methods: PBCP enforces both left-right and local consistency using a two-channel network trained as a classifier.It requires disparity maps estimated from both left and right images.
  • Estimation methods: LRCR integrates left-right consistency with disparity estimation using parallel convolutional LSTMs and a subsequent 3D convolutional LSTM.The model focuses subsequent processing on left-right mismatched regions.
  • Efficiency: Single-disparity confidence predictors avoid the runtime cost of estimating two disparity maps, using reference images, local patches, or the left disparity alone.Left-right checks are expensive because they estimate two disparity maps.
  • Broader reasoning: Confidence estimation can combine local and global reasoning, or ensemble multiple confidence maps and stereo matchers to improve robustness.LGC-Net combines local and large-receptive-field networks; other methods combine 23 confidence maps or four stereo matchers.

6 LEARNING MULTIVIEW STEREO

Deep multiview stereo methods address feature matching across wide view angles through volumetric or plane-sweep representations, with different fusion strategies and computational trade-offs.

  • Motivation and taxonomy: Multiview stereo follows the depth-from-stereo pipeline but must match features across multiple images without restricting correspondences to horizontal epipolar lines.Large view-angle differences motivate volumetric scene-geometry representations.
  • Motivation and taxonomy: Single-pass methods are classified as volumetric or Plane-Sweep Volume-based according to how multiple-view information is represented and aggregated.Plane-Sweep Volumes avoid intermediate volumetric representations of 3D geometry.
  • Volumetric methods: Volumetric methods fuse feature grids, pairwise cost volumes, or reconstructed surfaces across views.Fusion may use recurrent networks, confidence-weighted sums, or learned view-pair selection.
  • Volumetric methods: Colored Voxel Cubes reduce memory requirements for high-resolution volumetric reconstruction but require multiple forward passes and increase computation time.The scene is split into small cubes and regressed cube by cube.
  • Volumetric methods: Volumetric reconstruction took 4 hours for a typical DTU scene, compared with approximately 25 minutes for the alternative approach of Paschalidou et al.Post-reconstruction fusion methods have higher reconstruction errors than early-fusion methods.
  • Plane-Sweep Volumes: Plane-Sweep Volume methods directly estimate depth maps without intermediate volumetric geometry, improving computational efficiency while retaining the challenge of multiview feature matching.They back-project images or learned features onto depth planes.
  • Plane-Sweep Volumes: PSV architectures aggregate information across depth planes or views using per-plane branches, pairwise cost volumes, variance-based metrics, and 3D regularization.MVSNet back-projects learned features into a camera frustum and regularizes the resulting cost volume with a 3D U-Net.
  • Limitations: PSV methods eliminate the need for rectified images but require camera parameters, preset disparity ranges and discretization, and often complex architectures.Camera parameters may be supplied or estimated with Structure-from-Motion.

7 TRAINING END-TO-END STEREO METHODS

The survey reviews supervised, self-supervised, and cue-augmented end-to-end stereo training. These methods use disparity, reconstruction, consistency, smoothness, and semantic signals, but remain sensitive to calibration and domain shift.

  • Training objectives: Training optimizes network parameters with a loss combining disparity data terms and regularization or smoothness terms under different supervisory cues.The survey also compares supervised and self-supervised training formulations.
  • Supervised methods: Supervised methods minimize the distance between ground-truth and estimated disparity, optionally weighting valid pixels, regions of interest, initial estimates, and refined estimates.Distance choices include L1, L2, smooth L1, and robust variants.
  • Self-supervised methods: Self-supervised methods use image reconstruction losses based on projective geometry and coherence across multiple images.The right image is warped onto the reference view using estimated disparity.
  • Self-supervised methods: Reconstruction losses may compare image identity, learned features, gradients, or structural dissimilarity using L1 or L2 distances.Gradient-based losses are less sensitive to lighting and acquisition variations than photometric losses.
  • Limitations: Stereo-based supervision avoids ground-truth 3D labels but requires calibrated stereo pairs during training.
  • Weak supervision: Weakly supervised methods use auxiliary signals and confidence-based selection to reduce manual ground-truth labeling.Left-right consistency supplies confidence for selecting high-confidence matches as additional labeled data.
  • Additional cues: Additional constraints include smoothness, consistency, maximum depth, scale-invariant gradients, and semantic cues such as normals, segmentation, and edges.These cues guide disparity estimation through losses or additional network inputs.
  • Additional cues: Scale-invariant gradient loss penalizes relative neighboring-pixel depth errors, emphasizing discontinuities and sharp edges while increasing smoothness in homogeneous regions.

8 DISCUSSION AND COMPARISON

The survey compares 16 stereo methods on standard and novel scenarios, examining accuracy, runtime, memory, and supervision. Results show substantial domain-gap effects, broadly similar accuracy, and major deployment trade-offs.

  • The evaluation uses ApolloScape stereo images and four in-house images designed to test generalization to novel scenarios, including outdoor, bicycle, desk, and moving-camera scenes.
  • 16 stereo-based methods published between 2018 and 2019 are evaluated using authors’ provided network weights.
  • More than 1 second per disparity map and 3GB–10GB runtime memory make slow methods difficult to deploy on mobile platforms.
  • The reconstruction error exceeds 8 pixels on ApolloScape, suggesting that a significant training–testing domain gap can substantially reduce accuracy.
  • More than 25% of pixels have errors larger than 5 pixels, while foreground pixels exceed 55% above 3 pixels and fewer than 5% fall below 2 pixels.
  • SegStereo achieves similar or better performance than many supervised methods, while unsupervised adaptation achieves strong foreground performance.

9 FUTURE RESEARCH DIRECTIONS

Future work targets camera calibration, difficult lighting and materials, resolution, real-time processing, disparity parameterization, training data, and automated architecture design. The survey emphasizes accurate, efficient, and less annotation-dependent reconstruction.

  • Camera parameters: Known camera parameters remain necessary for rectification and Plane-Sweep Volumes, but estimating them in the wild is challenging.
  • Lighting conditions and complex material properties: Poor lighting and complex material properties remain difficult, motivating combinations of recognition, scene understanding, and low-level feature learning.
  • Spatial and depth resolution: Most methods produce low spatial and depth resolution, limiting reconstruction of thin or distant structures; real-time high-resolution accuracy remains unresolved.
  • Realtime processing: 3D and 4D cost volumes impose substantial memory and processing costs, leaving lightweight end-to-end networks as an open direction.
  • Disparity range: Uniform disparity discretization can create meter-scale depth errors at far ranges, and changing disparity ranges requires retraining.
  • Training: Ground-truth depth annotation is expensive and labor intensive, and limited training data can affect performance, generalization, and domain overfitting.
  • Automated architecture design: Automatically learning architectures, activation functions, and parameters could reduce the need for manual network design.

10 CONCLUSION

The survey organizes rapid developments in deep-learning stereo depth estimation and helps readers navigate a fast-growing field. Its scope excludes several related areas, including monocular and video-based depth estimation.

  • More than 150 papers were published from 2014 to 2019, making the survey a guide to understanding differences and similarities across the fast-growing field.
  • The survey does not cover image-based 3D object reconstruction, monocular or video-based depth estimation, photometric stereo, or active stereo.
Loading 2006.02535v1…