Source-linked AI summary
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Clément Godard, Oisin Mac Aodha, Gabriel J. Brostow
TL;DR
Ground-truth depth is costly and difficult to collect for monocular depth estimation. This paper trains a network from calibrated stereo footage using image reconstruction and left-right consistency, achieving state-of-the-art KITTI results that outperform supervised baselines.
Problem
Monocular depth methods commonly require large collections of costly, difficult-to-acquire ground-truth depth data for training.
Method
The network learns disparity from calibrated stereo pairs through image reconstruction while enforcing left-right consistency directly in its differentiable training loss.
Results
The method achieves state-of-the-art monocular depth estimation on KITTI and outperforms fully supervised baselines.
Takeaways & Limitations
Binocular stereo data can support single-image depth estimation without aligned ground-truth depth data.
Takeaways & Limitations
Training requires rectified and temporally aligned stereo pairs, so existing single-view datasets cannot be used directly for training.
Abstract
from arXiv · showhide
Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality depth data in a range of environments is a challenging problem. In this paper, we innovate beyond existing approaches, replacing the use of explicit depth data during training with easier-to-obtain binocular stereo footage. We propose a novel training objective that enables our convolutional neural network to learn to perform single image depth estimation, despite the absence of ground truth depth data. Exploiting epipolar geometry constraints, we generate disparity images by training our network with an image reconstruction loss. We show that solving for image reconstruction alone results in poor quality depth images. To overcome this problem, we propose a novel training loss that enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Our method produces state of the art results for monocular depth estimation on the KITTI driving dataset, even outperforming supervised methods that have been trained with ground truth depth.
1. Introduction
Monocular depth estimation is a fundamental machine-perception problem with broad applications, but many existing methods depend on multiple observations or large image collections with corresponding pixel depths. This work addresses these limitations with a novel training objective and network architecture, producing dense 512×256 depth maps in about 35 milliseconds on a modern GPU.
- Motivation: Many established depth-estimation approaches require multiple scene observations from different viewpoints or lighting conditions.These approaches include structure from motion, shape-from-X, binocular stereo, and multi-view stereo.
- Problem: Existing learning-based methods have been restricted to settings with large image collections and corresponding pixel-depth data.The paper identifies the availability of paired imagery and depth as a limitation for broader deployment.
- Motivation: Monocular depth estimation supports applications including object insertion, computational photography, robotic grasping, surgery, and 2D-to-3D film conversion.The paper frames single-image scene-shape understanding as a fundamental problem in machine perception.
- Method: The proposed training objective and network architecture improve final depth quality while avoiding limitations such as nondifferentiable components and poor scaling to large output resolutions.The introduction presents these changes as improvements over prior methods.
- Efficiency: 35 milliseconds is the approximate prediction time for a dense depth map at 512×256 resolution on a modern GPU.This demonstrates the method’s stated inference speed for the specified image size and hardware.
2. Related Work
Related work spans multi-image depth estimation, supervised monocular prediction, and reconstruction-based methods that avoid ground-truth depth. The paper positions its fully differentiable monocular approach against these methods, emphasizing left-right consistency to improve depth quality.
- Multi-image depth estimation: Multi-image methods use stereo pairs, multiple viewpoints, temporal sequences, or fixed-camera assumptions, and therefore generally require more than one scene image.The paper instead focuses on monocular depth estimation from a single input image.
- Stereo estimation: Stereo algorithms typically estimate disparity through pixel matching along rectified scan lines, but usually require accurate ground-truth disparity and stereo training data.Because such real-world data is difficult to obtain, these approaches often rely on synthetic training data that still requires manual content creation.
- Supervised monocular depth estimation: Monocular depth methods evolved from patch-based plane estimation trained on laser scans to dense pixel prediction with deep networks trained on images and depth values.Later work built on dense deep prediction using techniques such as CRFs to improve accuracy.
- Reconstruction-based methods: Reconstruction-based methods avoid ground-truth depth but may require multiple posed images, target novel-view synthesis, or produce disparity distributions from binocular input.DeepStereo requires several nearby posed images at test time and is therefore unsuitable for monocular depth estimation.
- Reconstruction-based methods: The concurrent method of Garg et al. also uses reconstruction loss for monocular depth, but its non-differentiable image formation requires a Taylor approximation that complicates optimization.The proposed model instead uses a fully differentiable formulation with bilinear sampling.
- Proposed approach: The proposed network estimates disparity without ground-truth depth, while left-right consistency addresses the poor depth that can result from minimizing photometric reconstruction loss alone.The method is loosely inspired by the supervised DispNet architecture.
3. Method
The method trains a single-image depth network without ground-truth depth by reconstructing calibrated stereo views from predicted disparities. It combines differentiable image reconstruction, smoothness, and left-right consistency losses across multiple output scales.
- Training objective: Each output-scale loss combines appearance reconstruction, disparity smoothness, and left-right disparity consistency terms, with left and right variants computed from only the left convolutional input.The total loss sums the scale-specific losses across four output scales.
- Stereo reconstruction: Training uses calibrated left-right image pairs and reframes depth estimation as reconstructing one stereo view from the other through dense disparity correspondences.The network estimates the right image from the left and similarly the left image from the right during training.
- Left-right consistency: The network infers both left-to-right and right-to-left disparities from only the left input image, then enforces their mutual consistency to improve depth quality.The right disparity is estimated during training but is not used at test time.
- Differentiable image formation: A fully differentiable bilinear sampler forms reconstructed images by backward mapping pixels from the opposite stereo image using the predicted disparity map.The appearance cost combines an L1 term with single-scale SSIM, while smoothness uses an edge-aware penalty on disparity gradients.
- Inference: At test time, the network outputs the finest-scale left disparity at input resolution, which is converted to depth using the training camera baseline and focal length.The right disparity is not used during inference.
4. Results
On KITTI, the proposed unsupervised monocular method outperforms Deep3D and existing methods, including supervised approaches, while left-right consistency and post-processing improve results. The method also generalizes reasonably to Make3D and adapts to a separate urban stereo dataset, though it requires aligned stereo training data and struggles with occlusions, specular, and transparent surfaces.
- Post-processing: Post-processing improves accuracy and reduces visual artifacts, but doubles test-time computation.The procedure combines predictions from the original and horizontally flipped images to reduce stereo-disocclusion effects.
- KITTI Split: Our loss outperforms Deep3D baselines, and left-right consistency improves performance across all measures.Deep3D can produce plausible reconstructions but inferior disparities compared with the proposed method.
- KITTI Split: The method outperforms all existing methods on KITTI, including approaches trained with ground-truth depth, while Cityscapes pre-training further improves results.Quantitative results are reported in Table 2, with qualitative examples in Figure 4.
- Stereo model: The stereo-input model outperforms the monocular model on every metric, especially D1-all, but becomes unstable when trained beyond 12 epochs.The stereo model receives concatenated left and right views.
- Make3D: On Make3D, a Cityscapes-only model achieves reasonable results despite dataset and camera differences, beating on one metric and on three.Make3D contains RGB/depth pairs but no stereo images, so it cannot be used directly for training this method.
- Limitations: The method requires rectified, temporally aligned stereo pairs for training and remains vulnerable to occlusion-boundary artifacts and inconsistent depths on specular or transparent surfaces.Fine-tuning on application-specific ground-truth depth remains possible.
5. Conclusion
The paper presents an unsupervised deep neural network for single-image depth estimation trained with binocular stereo data instead of aligned ground-truth depth, using a consistency-enforcing loss that improves predictions and surpasses fully supervised baselines. Future work includes extending the model to videos with temporal consistency, exploring sparse input, and predicting full scene occupancy.
- Conclusion: The method estimates single-image depth without aligned ground-truth data by training on binocular stereo and enforcing consistency between predicted depth maps from both camera views.This approach addresses the rarity and cost of aligned depth data while improving predictions.
- Conclusion: The results are superior to fully supervised baselines, supporting future research that does not require explicit ground-truth depth.The conclusion describes this outcome as encouraging for future research.
- Future work: Future work would extend the model to videos, where temporal consistency could improve estimates currently produced independently for each frame.The authors also propose investigating sparse input as an alternative training signal.
- Future work: The authors also propose predicting full scene occupancy in addition to the model’s current per-pixel depth estimates.This is identified as another direction for future investigation.
Supplementary Material · 1. Model architecture
The supplementary material specifies the network architecture through layer-wise kernel sizes, strides, channel counts, and input/output downscaling factors. It also defines layer inputs using concatenation and 2× upsampling.
- 1. Model architecture: The architecture table specifies each layer’s kernel size, stride, and input/output channel counts.Here, k denotes kernel size, s stride, and chns the number of input and output channels.
- 1. Model architecture: Each layer is described by input and output downscaling factors relative to the input image.These factors indicate the resolution scale associated with each layer.
- 1. Model architecture: Layer inputs may combine features through concatenation or apply 2× upsampling.The table denotes concatenation with + and 2× upsampling with ∗.
2. Post-Processing
Post-processing combines the input-image disparity with the horizontally flipped disparity of the flipped input using per-pixel weights. The weight construction uses normalized pixel coordinates, and the resulting disparities and weight map are illustrated in an example.
- 2. Post-Processing: The post-processed disparity is a per-pixel weighted sum of the input disparity and the flipped disparity from the flipped input.The components are denoted dl and d′′l, respectively.
- 2. Post-Processing: The weight map for the input disparity is defined using normalized pixel coordinates, while the second component’s weight map is obtained by horizontally flipping it.The coordinates are denoted i,j, and the flipped weight map is denoted w′l.
- 2. Post-Processing: An example presents the disparities dl, d′′l, d, and the weight map wl side by side.The figure shows these maps from left to right.
3. Deep3D Smoothness Loss
The paper enhances Deep3D with smoothness constraints for its per-pixel disparity distributions. Because these distributions cannot be smoothed directly, the method applies constraints to gradients of their cumulative sums.
- 3. Deep3D Smoothness Loss: Deep3D represents each pixel’s disparity as weights over offset input-image copies, forming a probability distribution whose weights sum to one.The weights are interpreted as a discrete probability distribution over disparities for each pixel.
- 3. Deep3D Smoothness Loss: Assuming each distribution has a maximum of one, the cumulative sum of its weights directly indicates the maximum disparity location.The associated weight is αmax=0.02.
- 3. Deep3D Smoothness Loss: In the example, the maximum occurs at disparity 3, and Equation 2 computes d=8−6=3.This observation motivates the smoothness constraint for the Deep3D image formation model.
- 3. Deep3D Smoothness Loss: The method applies smoothness constraints to gradients of the cumulative sums of the weights at each pixel.The associated smoothness-loss weight is αds=0.1.
4. More KITTI Qualitative Results
This section presents additional qualitative KITTI comparisons on the Eigen split and notes that sparse ground-truth velodyne depth was interpolated for visualization.
- Additional qualitative comparisons are shown for the KITTI dataset using the Eigen split.
- Ground-truth velodyne depth is very sparse, so it is interpolated for visualization.
5. Disparity Error Maps
The section analyzes disparity uncertainty by applying 50 random color augmentations to each test image and measuring per-pixel disparity standard deviation. The network is most confused around nearby objects, textureless regions, occlusion boundaries, and a very dark tunnel image.
- Disparity Error Maps: 50 random augmentations per test image were used to visualize the per-pixel standard deviation of predicted disparities.The same color-augmentation principle used during training was applied at test time.
- Disparity Error Maps: The network becomes confused around close-by objects, texture-less regions, occlusion boundaries, and a very dark tunnel image.The tunnel image was captured in a tunnel and produced a very dark image.