Unsupervised Monocular Depth Estimation with Left-Right Consistency

• Supervised monocular depth methods require extensive image collections with corresponding ground-truth depth, limiting them to settings where such data is available.Source: §1 · p. 1
• Accurate single-image depth supports applications including graphics, photography, robotics, surgery, 2D-to-3D conversion, and autonomous driving.Source: §1 · p. 1
• The method trains a fully convolutional model to infer pixel correspondences from rectified stereo images, using reconstruction rather than depth labels.Source: §1 · p. 1
• The method predicts dense depth for a 768 × 256 image in about 50 milliseconds on a modern GPU.Source: §1 · p. 1
• Its network uses an end-to-end loss with a left-right depth-consistency constraint and evaluates alternative losses and image-formation models.Source: §1 · p. 2
• The paper reports state-of-the-art results on a challenging driving dataset and generalization to three additional datasets, including a new outdoor urban dataset.Source: §1 · p. 2

2 RELATED WORK

• Existing image-based depth methods use stereo pairs, multiple viewpoints, temporal sequences, or fixed-scene assumptions, whereas this work targets single-image inference without scene-geometry assumptions.Source: §2 · p. 2

Learning Based Stereo

• Learning-based stereo methods such as DispNet directly predict disparity but require large amounts of accurate ground-truth disparity and stereo training data.Source: Learning Based Stereo · p. 2
• Synthetic training data can reduce this burden, but it still requires manually creating content for each application scenario.Source: Learning Based Stereo · p. 2

Supervised Single Image Depth Estimation

• Supervised single-image methods learn dense depth from pixel-aligned depth labels, while this work instead uses an added binocular color image during training.Source: Supervised Single Image Depth Estimation · p. 2

Unsupervised Depth Estimation

• DeepStereo requires several nearby posed images at test time, making it unsuitable for monocular depth estimation.Source: Unsupervised Depth Estimation · p. 2
• Deep3D reconstructs a neighboring view but scales poorly as candidate disparities and output resolution increase.Source: Unsupervised Depth Estimation · p. 2
• The proposed model uses a fully differentiable bilinear image sampler and adds left-right consistency during training because photometric reconstruction alone can yield poor depth.Source: Unsupervised Depth Estimation · p. 3

3 METHOD

• The method is designed as an end-to-end training loss with left and right image variants, while its architecture is documented through convolutional layers, skip connections, and upsampling.The network architecture uses concatenation and 2× upsampling in its decoder.Source: §3 · p. 3, §3.3 · p. 4, Table 1 · p. 4

3.1 Depth Estimation as Image Reconstruction

• Calibrated stereo pairs provide training supervision by letting the network predict pixel correspondences, convert disparity to depth, and avoid ground-truth depth.For rectified images, disparity is a scalar per pixel; depth is recovered from disparity using the camera baseline and focal length.Source: §3.1 · p. 3

3.2 Depth Estimation Network

• The fully convolutional network predicts disparities at four scales, with encoder-decoder skip connections for higher-resolution details and separate maps aligned to each stereo view.At every output scale, one disparity map is aligned with the network input and the other with its stereo partner.Source: §3.2 · p. 3

3.3 Training Loss

• The total multiscale loss combines appearance matching, disparity smoothness, and left-right consistency, with left and right variants for each component.Scale weights control the relative importance of losses at different output resolutions.Source: §3.3 · pp. 3–4

Appearance Matching Loss

• Bilinear spatial-transformer sampling reconstructs the neighboring stereo image through a locally fully differentiable image-formation model.The appearance cost combines L1 and single-scale SSIM terms, while edge-aware disparity-gradient penalties encourage smoothness except near image gradients.Source: Appearance Matching Loss · p. 4, Disparity Smoothness Loss · p. 4

Left-Right Disparity Consistency Loss

• Predicting both left and right disparities and penalizing their disagreement enforces coherence while the convolutional network receives only the left view.The consistency term is mirrored for the right disparity and evaluated at every output scale.Source: Left-Right Disparity Consistency Loss · p. 4
• At test time, a single forward pass outputs the finest-scale left disparity, which is upsampled to input resolution and converted to depth using the training camera parameters.The right disparity is estimated during training but is not used for prediction.Source: Left-Right Disparity Consistency Loss · p. 4

4 RESULTS

• The evaluation compares the approach with supervised and unsupervised single-view depth methods using rectified stereo training pairs without ground-truth depth.Datasets lacking stereo pairs are described as unsuitable for this evaluation.Source: §4 · p. 4
• A Deep3D image-formation variant and a baseline without left-right consistency are included to evaluate the image-formation model and consistency constraint.These comparisons isolate two design choices in the proposed training approach.Source: §4 · p. 4

4.1 Implementation Details

• The network has 31 million parameters and trains for 50 epochs on 30 thousand images using two NVIDIA GeForce GTX TITAN X GPUs.Training takes on the order of 20 hours.Source: §4.1 · pp. 4–5
• Inference takes less than 50 ms for a 768 × 256 image, corresponding to more than 20 frames per second including GPU transfer times.The reported speed is measured on the described implementation.Source: §4.1 · p. 5
• The model constrains disparities with a scaled sigmoid, uses α_ap = 1 and α_lr = 1, and scales smoothness weighting by output resolution.The maximum disparity is set to 0.3× the image width at each output scale.Source: §4.1 · p. 5
• ELUs replace ReLUs because ReLUs prematurely fixed intermediate-scale disparities, while optimizing all four scales jointly produced more stable convergence.Batch normalization was tested but excluded after no significant improvement.Source: §4.1 · p. 5
• On-the-fly augmentation horizontally flips and swaps stereo views, with random gamma, brightness, and per-channel color shifts.The augmentation preserves the correct left-right ordering after flipping.Source: §4.1 · p. 5

4.2 KITTI

• The KITTI evaluation uses full-resolution inputs, 40,046 training images, an approximately 80-meter prediction cap, and depth and D1-all disparity metrics.The authors note that depth-space errors can be sensitive to large errors caused by small-disparity predictions.Source: §4.2.1 · p. 5

4.2.1 KITTI Split

• Stereo inputs produce sharper, more accurate disparity maps than single images but can show artifacts in low-textured road regions.The cited example contains a hole in the road disparity map.Source: Figure 4 · p. 5
• The proposed loss outperforms Deep3D on KITTI, and adding left-right consistency improves performance across all measures.The consistency term also improves visual quality, especially around object boundaries.Source: §4.2.1 · p. 5, Figure 5 · p. 6

4.2.2 Eigen Split

• The evaluation also uses the Eigen split of 697 test images, with velodyne-reprojected depth that is sparse and affected by motion, rotation, and occlusion errors.Predictions and baselines are resized and evaluated using a common crop and input resolution.Source: §4.2.2 · pp. 5–6

4.2.1 KITTI Split

• Table 2 compares image-formation models on the official 200-image KITTI 2015 test set, including KITTI and Cityscapes training variants.The table also includes a model trained and tested with two input images.Source: Table 2 · p. 6

4.2.2 Eigen Split

• Our left-right-consistent model outperforms all existing methods on the Eigen split, including methods trained with ground-truth depth.Cityscapes pretraining further improves results over KITTI-only training.Source: §4.2.2 · p. 6

4.3 Stereo

• The stereo model performs significantly better on the D1-all disparity measure than the single-image model.Its distance-based depth errors are worse for some measures.Source: §4.3 · p. 6
• Textureless road regions explain the stereo model’s higher distance-based errors because their depths are sometimes estimated incorrectly.The paper links this behavior to artifacts visible in the qualitative stereo results.Source: §4.3 · p. 6

4.4 Make3D

• Despite dataset and camera dissimilarities, the Cityscapes-trained model achieves reasonable Make3D results without Make3D training data.Evaluation uses a central crop because Make3D has a different aspect ratio.Source: §4.4 · p. 6
• The model’s numerical results are worse than the supervised baselines, but its qualitative outputs compare favorably to the supervised competition.The authors expect more relevant training data could improve the numerical results.Source: §4.4 · p. 6
• Table 4 reports Make3D results, with starred methods using ground-truth Make3D depth and evaluation restricted to depths below 70 meters in a central crop.The table’s C1 metric is computed only within that evaluation region.Source: Table 4 · p. 7

4.5 Generalizing to Other Datasets

• The Cityscapes-trained model produces visually plausible depths on CamVid despite differences in location, image characteristics, and camera calibration.The result demonstrates qualitative cross-dataset generalization.Source: §4.5 · p. 7
• Finetuning on a 20,000-frame synchronized-camera dataset produces visually convincing depth images on test footage captured with the same rig on another day.The training data came from paired videos recorded with consumer cameras.Source: §4.5 · p. 7
• On Make3D, the model can confuse a shadow and synthesize depth for a human instead of a tree.This failure appears in the last qualitative-result row.Source: Figure 7 · p. 8
• Figure 8 shows qualitative outputs on Cityscapes, CamVid, and two handheld urban-video scenes for a model trained on Cityscapes.The figure covers both in-domain and cross-dataset visual evaluations.Source: Figure 8 · p. 8

4.6 Limitations

• Occlusion boundaries and the left image border can still contain artifacts because corresponding pixels may be invisible in the other view.The authors note that explicit occlusion reasoning could improve these issues.Source: §4.6 · p. 8
• Training requires rectified, temporally aligned stereo pairs, so existing single-view datasets cannot currently be used for training.The model can nevertheless be fine-tuned with application-specific ground-truth depth data.Source: §4.6 · p. 8
• Pixel-space reconstruction assumes brightness constancy, which is violated on specular and transparent surfaces.The authors suggest alternative similarity measures as a possible improvement.Source: §4.6 · p. 8

5 CONCLUSION

• The method replaces rare, costly aligned depth data with binocular stereo data and uses a loss enforcing consistency between depth maps from both camera views.This is the paper's central unsupervised training strategy.Source: §5 · p. 8
• The results are superior to fully supervised baselines, and the model generalizes to unseen datasets while producing visually plausible depth maps.The conclusion reports both the benchmark comparison and cross-dataset generalization.Source: §5 · p. 8
• Future work targets temporally consistent video depth, sparse-input training signals, and full occupancy prediction for visible and occluded regions.The current model estimates per-pixel depth independently for each frame.Source: §5 · p. 8