Source-linked AI summary
Learning monocular depth estimation infusing traditional stereo knowledge
Fabio Tosi, Filippo Aleotti, Matteo Poggi, Stefano Mattoccia
TL;DR
Monocular depth estimation is difficult because a single image lacks direct geometric correspondence, while ground-truth depth labels are expensive. monoResMatch addresses this by synthesizing a virtual stereo view, matching features end-to-end, and using traditional stereo proxy labels; the combined approach achieves state-of-the-art self-supervised performance.
Problem
Monocular depth estimation is challenging because it is ill-posed, and obtaining direct ground-truth depth annotations is expensive.
Method
monoResMatch estimates depth from one image by synthesizing virtual-right-view features, performing stereo matching, and training with proxy labels from traditional stereo algorithms.
Results
monoResMatch with proxy supervision achieves state-of-the-art self-supervised monocular depth estimation and outperforms self-supervised competitors on the Eigen split.
Takeaways & Limitations
End-to-end virtual stereo matching combined with traditional stereo proxy supervision provides a more accurate self-supervised route to monocular depth estimation.
Abstract
from arXiv · showhide
Depth estimation from a single image represents a fascinating, yet challenging problem with countless applications. Recent works proved that this task could be learned without direct supervision from ground truth labels leveraging image synthesis on sequences or stereo pairs. Focusing on this second case, in this paper we leverage stereo matching in order to improve monocular depth estimation. To this aim we propose monoResMatch, a novel deep architecture designed to infer depth from a single input image by synthesizing features from a different point of view, horizontally aligned with the input image, performing stereo matching between the two cues. In contrast to previous works sharing this rationale, our network is the first trained end-to-end from scratch. Moreover, we show how obtaining proxy ground truth annotation through traditional stereo algorithms, such as Semi-Global Matching, enables more accurate monocular depth estimation still countering the need for expensive depth labels by keeping a self-supervised approach. Exhaustive experimental results prove how the synergy between i) the proposed monoResMatch architecture and ii) proxy-supervision attains state-of-the-art for self-supervised monocular depth estimation. The code is publicly available at https://github.com/fabiotosi92/monoResMatch-Tensorflow.
1. Introduction
Monocular depth estimation avoids stereo hardware but is ill-posed, motivating self-supervised approaches that use stereo pairs and proxy labels. monoResMatch combines virtual stereo matching with traditional stereo-derived supervision in an end-to-end architecture.
- Motivation: Stereo estimates disparity from multiple viewpoints, while monocular depth avoids stereo rigs but remains challenging because the problem is ill-posed.Disparity is the inverse of depth, and SGM is a popular traditional stereo algorithm.
- Motivation: Self-supervised monocular methods replace costly ground-truth depth annotations with image synthesis from stereo pairs or monocular sequences.Stereo-pair supervision is reported as more effective than monocular-sequence supervision because it provides more constrained geometry.
- Proposed approach: monoResMatch maps one image into feature space, estimates an initial depth, synthesizes features from a virtual right viewpoint, and performs stereo matching for refinement.The architecture is trained end-to-end and uses a single network rather than separately trained synthesis and stereo components.
- Proposed approach: Traditional stereo knowledge supplies proxy labels for self-supervised training, despite outliers, and the paper reports superior accuracy to image-warping approaches.The proxy labels are generated using traditional stereo algorithms such as SGM.
2. Related Work
Related work spans conventional and learned stereo, self-supervised monocular depth, single-view stereo, and proxy-label distillation. monoResMatch differs from prior single-view stereo by training its architecture end-to-end without ground-truth supervision.
- Stereo depth estimation: Conventional stereo commonly follows four processing steps, with SGM notable for balancing accuracy and efficiency.Deep stereo methods replace or learn matching costs and formulate correspondence through similarity prediction or classification.
- Stereo depth estimation: Learned stereo architectures include siamese matching-cost networks and models that select disparities from a cost volume.These approaches include DispNetC- and GCNet-derived designs, with domain-shift mitigation addressed through offline, online, or external-depth guidance.
- Monocular depth estimation: Self-supervised monocular depth estimation replaces expensive ground-truth labels with multiple views and image synthesis.The related literature includes supervision from monocular sequences, stereo pairs, or both.
- Single View Stereo: Single View Stereo synthesizes a second view from one image and computes disparity between the real and synthetic views, but its synthesis and stereo networks are trained independently.The cited SVS framework uses Deep3D for view synthesis and DispNetC for disparity estimation.
- Proxy labels distillation: Proxy-label distillation replaces difficult-to-source ground-truth labels with labels generated by conventional methods across stereo, confidence estimation, optical flow, and monocular depth.The paper positions monoResMatch as requiring no ground-truth labels while improving on prior SVS.
3. Monocular Residual Matching
monoResMatch is an end-to-end architecture that estimates monocular depth by synthesizing a virtual stereo representation and refining disparity through stereo matching. Its training combines image reconstruction, smoothness, and proxy-supervised losses across multiple scales.
- Architecture: monoResMatch extracts multi-scale features from one image and estimates initial disparities aligned with both the input and a virtual right view.The virtual view emulates a binocular setup for subsequent stereo-domain processing.
- Training strategy: The complete architecture is trained from scratch end-to-end, unlike SVS, whose image-synthesis and disparity-estimation components are trained independently.SVS additionally requires supervised depth labels from synthetic imagery for its disparity estimator.
- Architecture: A refinement module predicts residual corrections to the initial disparity, using real and back-warped synthesized features plus their error signal.The final output is a left-view-aligned disparity map.
- Training loss: The total loss combines initial-disparity and refinement terms, with predictions upsampled to full resolution so the objective is applied at multiple scales.The initial term includes left and right components, while refinement uses left components only.
- Training loss: Image reconstruction, edge-aware disparity smoothness, and proxy-supervised losses provide the principal training signals.The proxy-supervised term uses reverse Huber loss on proxy disparity maps.
- Training loss: Figure 3 illustrates SGM supervision filtered by left-right consistency, followed by monoResMatch prediction without post-processing from [11].The sequence is source image, filtered proxy labels, and final disparity map.
4. Proxy labels distillation
Proxy labels are generated with conventional stereo matching and filtered by left-right consistency before being used to supervise monocular depth estimation. This filtering removes many erroneous assignments without requiring a learned confidence model or ground-truth depth.
- Proxy label generation: SGM generates proxy disparity labels from rectified stereo pairs without training, using census-based matching costs and scanline optimization along eight paths.A winner-takes-all step produces the final disparity map for each reference-image pixel.
- Proxy label filtering: Left-right consistency filtering removes many incorrect assignments, especially near depth discontinuities, without requiring additional training.The resulting proxy-label process does not rely on ground-truth depth.
- Proxy label filtering: The distilled labels can retain some outliers, yet the trained network recovers accurate disparity values in occluded regions on the image’s left side.Figure 3 depicts filtered outliers as black pixels and shows the resulting prediction.
5. Experimental results
Experiments evaluate monoResMatch on KITTI and CityScapes, studying proxy supervision and architectural components before comparing against self-supervised and sparsely supervised methods. The framework consistently outperforms the compared approaches, including on the KITTI stereo benchmark.
- Datasets and protocol: The evaluation uses standard monocular depth metrics on KITTI and CityScapes, with KITTI Eigen-split testing based on sparse LiDAR ground truth.Metrics include Abs rel, Sq rel, RMSE, RMSE log, and threshold accuracy δ < ζ.
- Proxy-supervised loss analysis: SGM proxy loss improves the monodepth baseline over the original model and outperforms virtual trinocular supervision on most reported metrics.The exception is δ < 1.25, where 3Net performs better; coupling proxy distillation with cropping also addresses disparity ramps on the left border.
- Component analysis: Correlation-based refinement provides the largest component contribution, while post-processing and CityScapes pretraining further improve the full model.The architecture also outperforms the compared monodepth baseline without proxy supervision and exceeds the proxy-supervised ResNet50 configuration without refinement.
- Comparison with self-supervised frameworks: On the Eigen split, monoResMatch significantly outperforms all compared self-supervised monocular, stereo, and combined-supervision methods.The comparison excludes methods using ground-truth labels during training and includes competitors trained on CityScapes and KITTI, with one exception using different KITTI subsplits.
- Few-label supervision: With few KITTI annotations, monoResMatch consistently achieves better scores in direct comparisons and attains the best overall accuracy across all metrics with 200 accurate labels plus 500 KITTI samples.This configuration also outperforms methods trained with substantially more supervised data, according to the reported comparison.
- Single-view stereo evaluation: On the KITTI 2015 stereo benchmark, monoResMatch reduces the total D1 error by 2.72% relative to the compared methods.The reported margins exceed 3% on background pixels and are near 1% on foreground pixels; Figure 4 shows input, predicted depth, and error maps with a severity color code.
6. Conclusions
monoResMatch combines correlation-based refinement for stereo-like monocular depth estimation with proxy ground-truth labels from SGM, trained end-to-end. Together, these choices achieve state-of-the-art self-supervised monocular depth estimation.
- monoResMatch uses a correlation-based refinement module to approach monocular depth estimation analogously to stereo matching.
- Proxy ground-truth labels generated by SGM provide more robust self-supervised training without requiring expensive depth labels.
- The architecture is trained end-to-end, unlike the state-of-the-art models cited by the authors.
- Combining monoResMatch with proxy supervision achieves state-of-the-art self-supervised monocular depth estimation.