Source-linked AI summary
Self-Supervised Monocular Depth Hints
Jamie Watson, Michael Firman, Gabriel J. Brostow, Daniyar Turmukhambetov
TL;DR
Self-supervised stereo depth learning can be trapped by ambiguous photometric reprojections, limiting depth quality. The paper introduces Depth Hints—stereo-derived alternative depth suggestions that selectively enhance the photometric loss—and reports improvements across leading methods, including state-of-the-art KITTI performance. The approach is scoped to settings with suitable stereo information and is not naturally compatible with monocular-video-only data.
Problem
Photometric reprojection losses can have multiple local minima, producing limited-quality depth maps and errors around thin structures and sharp boundaries.
Method
Depth Hints use complementary depth suggestions from off-the-shelf stereo algorithms to guide training when their reprojection is better, while reverting to the default loss when unhelpful.
Results
Depth Hints improve several leading self-supervised-from-stereo models and achieve top scores on the self-supervised KITTI Eigen benchmark.
Takeaways & Limitations
Depth Hints provide a practical way to guide networks toward better solutions, especially for thin structures and sharp boundaries.
Takeaways & Limitations
Depth Hints are not naturally compatible with monocular-video-only data, and pose hints are left as a future extension.
Abstract
from arXiv · showhide
Monocular depth estimators can be trained with various forms of self-supervision from binocular-stereo data to circumvent the need for high-quality laser scans or other ground-truth data. The disadvantage, however, is that the photometric reprojection losses used with self-supervised learning typically have multiple local minima. These plausible-looking alternatives to ground truth can restrict what a regression network learns, causing it to predict depth maps of limited quality. As one prominent example, depth discontinuities around thin structures are often incorrectly estimated by current state-of-the-art methods. Here, we study the problem of ambiguous reprojections in depth prediction from stereo-based self-supervision, and introduce Depth Hints to alleviate their effects. Depth Hints are complementary depth suggestions obtained from simple off-the-shelf stereo algorithms. These hints enhance an existing photometric loss function, and are used to guide a network to learn better weights. They require no additional data, and are assumed to be right only sometimes. We show that using our Depth Hints gives a substantial boost when training several leading self-supervised-from-stereo models, not just our own. Further, combined with other good practices, we produce state-of-the-art depth predictions on the KITTI benchmark.
1. Introduction
Self-supervised monocular depth learning avoids costly ground-truth depth, but ambiguous photometric reprojections can trap training in poor solutions. Depth Hints address this ambiguity and improve multiple stereo-based methods, achieving state-of-the-art KITTI results.
- Motivation: Ground-truth depth is difficult and expensive to acquire, motivating self-supervised learning from commodity color-camera data.Stereo training derives relative poses from camera calibration while optimizing network depth predictions.
- Ambiguous reprojections: Photometric reprojection losses can assign similarly low errors to multiple depths, especially around repeating structures and uniform textures.These local alternatives make optimization harder and reduce prediction accuracy.
- Depth Hints: Depth Hints provide selective alternative depth suggestions, following the hint where its reprojection is better while retaining the photometric loss otherwise.Hints can be generated from the same stereo images using off-the-shelf stereo estimates such as OpenCV’s.
- Ambiguous reprojections: Existing self-supervised regression methods can struggle to find the global optimum when minimizing photometric reprojection loss.The resulting artifacts are not limited to generalization failure or overfitting, as they can occur on training images.
- Results: Selective Depth Hint training improves multiple leading self-supervised algorithms and, with network design choices, outperforms baseline and prior results on KITTI.The paper reports state-of-the-art results on the KITTI dataset.
2. Related Work
Related work spans stereo, monocular-video, supervised, synthetic, SLAM, semantic, and estimated-depth supervision. Depth Hints are positioned as a stereo-derived enhancement compatible with many stereo-trained models but not naturally with monocular-video-only data.
- Self-supervised approaches: Self-supervised depth methods exploit photometric consistency from binocular stereo, consecutive video frames, or stereo video.These settings differ in whether depth, pose, or both are optimized or predicted.
- Stereo training: Stereo methods use photometric losses and additional consistency or fidelity techniques, including left-right consistency, post-processing, higher-resolution loss computation, and subpixel convolutions.Depth Hints are computed from binocular data and can enhance stereo-derived models using DSSIM+L1 losses.
- Monocular training: Monocular-video methods jointly estimate depth and camera motion, with later work adding minimum reprojection, masking, geometric regularization, optical flow, and consistency constraints.Some methods combine monocular video and stereo pairs for improved pose and depth estimation.
- Scope: Depth Hints are not naturally compatible with monocular-video-only data, and extensions are left for future work.This scope boundary distinguishes the method from approaches trained solely on monocular videos.
- Additional supervision: Additional supervision has used sparse LiDAR or SLAM depths, synthetic data, semantic labels, and proxy depths from traditional stereo matching.These approaches combine or replace photometric self-supervision with estimated or externally generated signals.
3. Background
The background formulates self-supervised depth as reconstructing a reference image by reprojecting another view using predicted depth and known camera geometry. Training commonly uses DSSIM+L1 photometric error, while estimated-depth supervision provides an alternative loss signal.
- Self-supervised formulation: A depth network predicts a depth map from one image, then reprojects a second image into the reference view using camera intrinsics and relative pose.The objective is to reconstruct the reference image through photometric consistency.
- Camera geometry: In stereo training, estimated depth is the only unknown parameter in the reprojection function; video training additionally requires predicting camera pose.The paper does not pursue pose hints, although it identifies them as a natural extension.
- Photometric loss: The commonly used DSSIM+L1 loss measures per-pixel agreement between the reference and warped images.SSIM is computed over a 3x3 pixel window with α set to 0.85.
- Supervised alternatives: Supervised training compares predicted continuous depth d_i with an available depth d′_i from LiDAR, Kinect, stereo, or SLAM systems.Stereo- and SLAM-derived labels may count as self-supervision because they are inferred rather than ground-truth measurements.
- Supervised alternatives: The paper compares several supervised losses, including L1, L2, and berHu-like choices, and reports that log L1 works well with estimated depths.The same losses can also be applied to inverse depth, or disparity.
4. The Need for Depth Hints
Photometric self-supervision can produce plausible but incorrect depth solutions, especially around thin structures, because optimization becomes trapped in local minima. Depth Hints provide alternative depth values when they yield better reprojections, helping training approach the best minimum without requiring ground-truth depth labels.
- Thin structures are missing from depth maps predicted by a stereo-trained network despite the image belonging to its training set.The failure is therefore not attributed to generalization or overfitting.
- DSSIM+L1 can assign lower loss to more appropriate depths while the network remains at an inferior solution.This indicates that the observed errors are not simply caused by an inadequate photometric loss value at the correct depth.
- Depth Hints address this ambiguity by supplying alternative depth values when they produce more plausible reprojections than the current prediction.The hints are noisy and may be ignored as training progresses, while the main objective remains self-supervised reprojection loss.
5. Method
Depth Hints adapt an existing stereo-based photometric loss by consulting an auxiliary stereo depth only where it improves reprojection. The hints come from off-the-shelf stereo estimation and are incorporated selectively rather than treated as ground-truth depth labels.
- The method modifies an existing self-supervised stereo loss only where the network’s estimated depth is worse than a Depth Hint.A Depth Hint is a depth map estimated by a third-party binocular stereo algorithm.
- Depth Hints are not regression targets; they provide supervisory input only when their reprojection is superior to the network prediction.This preserves optimization of the original algorithm’s loss instead of training directly against a hinted depth map.
- The loss uses the hint when lr(hi) < lr(di), and otherwise retains the network reprojection loss lr(di).Here di is the inferred network depth, hi the hint, and lr evaluates photometric quality.
- Computing Depth Hints: The hints are generated from stereo pairs using heuristic off-the-shelf methods because perfect hints and exhaustive disparity searches are impractical.The paper uses Semi-Global Matching (SGM), available in OpenCV, rather than relying on supervised modern stereo algorithms.
- Computing Depth Hints: Random SGM generates hints online with randomly selected hyperparameters, while Fused SGM selects per-pixel depths from 12 SGM configurations.The 12 configurations combine three block sizes with four disparity resolutions.
- For stereo video, the same approach augments depth-and-pose training by applying the hint mechanism alongside reprojections from neighboring frames.The depth network receives the current frame, while the pose network uses the current, previous, and next frames.
6. Experiments
Experiments evaluate Depth Hints across stereo self-supervision, alternative loss formulations, existing methods, and datasets using KITTI and FlyingThings3D. The results support selective use of noisy hints and show improvements across modern stereo-trained models, with a semi-supervised LiDAR-based case as an exception.
- The evaluation covers four settings: local-minimum behavior, Depth Hint ablations, integration with existing models, and comparison with state-of-the-art methods.Experiments use CNNs trained from binocular stereo data, with KITTI evaluated on LiDAR point clouds using the Eigen split.
- 6.1. Solution with Depth Hints: Depth Hints are used by 85% of available pixels initially, decreasing to 50% by the end of training.This pattern is consistent with hints being consulted selectively rather than enforced throughout optimization.
- 6.2. Baseline Loss Functions: Treating Depth Hints as noisy and using them only when superior to the network prediction clearly outperforms alternative hint-integration strategies.The comparison includes direct proxy supervision, summed losses, pretraining followed by self-supervision, and uncertainty modeling.
- 6.2. Baseline Loss Functions: A direct Fused SGM proxy-supervised baseline scores better than 3Net and SuperDepth and is highly competitive with Monodepth2 on all metrics, albeit with pretraining.This result shows that even simpler hint-based baselines can be competitive with established methods.
- 6.3. Depth Hints for Existing Methods: Depth Hints noticeably improve all evaluated stereo and stereo-video methods, while the semi-supervised method shows no improvement and is not harmed.The semi-supervised exception may relate to its existing LiDAR data, according to the table caption.
- Depth Hints produce substantial improvements on FlyingThings3D, where many thin structures make the ambiguity especially consequential.The authors report that these results demonstrate gains for monocular depth estimation across domains.
7. Depth From Color Tournament
On KITTI, Depth Hints improve results across stereo, high-resolution, and stereo-video settings, with especially strong gains for thin structures and boundary-sensitive errors.
- 7. Depth From Color Tournament: Depth Hints achieve state-of-the-art results across KITTI’s stereo, high-resolution, and stereo-video categories.The approach is evaluated with and without pretraining across multiple training settings.
- Low-res Stereo: Ours Resnet50 wins every metric in Low-res Stereo, including without pretraining against the highest-scoring non-pretrained network.The non-pretrained model scores better than 3Net on all seven metrics.
- High-res Stereo: Ours HR Resnet50 improves over existing methods on all metrics, while the non-pretrained model beats SuperDepth in six of seven metrics.It ties SuperDepth on the remaining metric and compares favorably with monoResMatch.
- Stereo Video MS: High-resolution stereo-video training receives a substantial boost from Depth Hints, whereas lower-resolution gains over Monodepth2 are smaller.Stereo-video models use both temporal frames and stereo images.
- Error Metrics: SqRel and RMSE benefit most because Depth Hints recover thin structures and delineate object boundaries more accurately.AbsRel gains are smaller because only a minority of image pixels improve.
- Depth Supervised: The depth-supervised category remains outside the method’s competition because leading alternatives use LiDAR, synthetic depth, or SLAM-derived supervision.The comparison is therefore focused on self-supervised categories.
8. Conclusion
The paper introduces Depth Hints to help self-supervised depth models escape photometric-loss local minima. Experiments support improvements for thin structures, existing self-supervision schemes, and KITTI benchmark performance.
- 8. Conclusion: Depth Hints guide self-supervised monocular depth models toward better solutions by addressing local minima in reprojection losses.The method reverts to the default reprojection loss when proposals are unhelpful.
- 8. Conclusion: Depth proposals provide a strong baseline, while selective training applies them only when useful and otherwise retains the default reprojection loss.This mechanism treats proposals as potentially noisy rather than fully trusted labels.
- 8. Conclusion: Depth Hints help most with thin structures and sharp boundaries, and improve existing self-supervision schemes.The conclusion reports support from extensive experimentation.
- 8. Conclusion: Combined with a common network architecture, Depth Hints achieve top scores on the self-supervised KITTI Eigen benchmark by a significant margin.Pretraining is preferred but not required for the reported result.