Source-linked AI summary
Learning Monocular Depth by Distilling Cross-domain Stereo Networks
Xiaoyang Guo, Hongsheng Li, Shuai Yi, Jimmy Ren, Xiaogang Wang
TL;DR
Monocular depth estimation needs substantial depth annotations or suffers from accuracy limitations, while synthetic data introduces a synthetic-to-real domain gap. The paper uses a stereo matching network as a proxy, fine-tunes it across domains, and transfers its disparity supervision to monocular depth estimation. The resulting pipeline achieves state-of-the-art monocular depth estimation on KITTI.
Problem
Supervised monocular depth methods require difficult-to-obtain depth annotations, unsupervised methods face correspondence problems and limited accuracy, and synthetic data transfers poorly across domains.
Method
A stereo matching network is pretrained on synthetic data, fine-tuned on realistic data with supervised or unsupervised strategies, and used to supervise a monocular depth network.
Results
The pipeline achieves state-of-the-art monocular depth estimation performance on KITTI in both unsupervised and semi-supervised settings.
Takeaways & Limitations
Using stereo matching as a proxy enables cross-domain synthetic data to provide dense supervision for monocular depth learning.
Takeaways & Limitations
The pipeline could be further improved with more advanced stereo algorithms, better fine-tuning, and confidence-based filtering of stereo outputs.
Abstract
from arXiv · showhide
Monocular depth estimation aims at estimating a pixelwise depth map for a single image, which has wide applications in scene understanding and autonomous driving. Existing supervised and unsupervised methods face great challenges. Supervised methods require large amounts of depth measurement data, which are generally difficult to obtain, while unsupervised methods are usually limited in estimation accuracy. Synthetic data generated by graphics engines provide a possible solution for collecting large amounts of depth data. However, the large domain gaps between synthetic and realistic data make directly training with them challenging. In this paper, we propose to use the stereo matching network as a proxy to learn depth from synthetic data and use predicted stereo disparity maps for supervising the monocular depth estimation network. Cross-domain synthetic data could be fully utilized in this novel framework. Different strategies are proposed to ensure learned depth perception capability well transferred across different domains. Our extensive experiments show state-of-the-art results of monocular depth estimation on KITTI dataset.
1 Introduction
Monocular depth estimation is valuable but constrained by difficult depth annotation and weak unsupervised correspondence. The paper addresses synthetic-to-real domain gaps by using cross-domain stereo matching to supervise monocular depth learning.
- Depth estimation supports 3D geometry understanding and downstream tasks including object detection, tracking, and recognition.
- Supervised methods require large annotated datasets, while sparse LIDAR and strong-light limitations complicate depth acquisition.
- Unsupervised methods struggle with low texture, repeated patterns, and occlusions because photometric losses lack correspondences there.
- Synthetic depth data can help, but transfer to realistic scenes requires matching contents, appearance, and viewpoints across domains.
- The proposed pipeline predicts synthetic-data disparity and occlusion masks, adapts the stereo network on realistic data, then uses it to supervise monocular depth.
- Stereo guidance enables monocular predictions with sharp boundaries and clear thin structures, while the full pipeline achieves state-of-the-art KITTI performance.
2 Related Work
Related work covers monocular depth estimation, stereo-based depth recovery, and deep stereo matching. These approaches span classical graphical models, view-synthesis training, and synthetic-to-real stereo-network adaptation.
- Stereo depth methods estimate disparity from image pairs and convert it to depth using camera focal length and baseline.
- Monocular depth estimation predicts depth from one image despite ambiguity in object sizes, using supervised or unsupervised learning.
- Early monocular methods combined multiscale features, graphical models, and conditional random fields to recover detailed 3D structure.
- Unsupervised monocular methods use view-synthesis losses, left-right consistency, or joint depth-and-camera-pose learning from stereo pairs or video.
- Stereo matching computes disparity from rectified image pairs through matching cost, aggregation, optimization, and refinement stages.
- Deep stereo networks commonly train on synthetic Scene Flow data before fine-tuning on realistic data, including DispNet-style architectures.
3 Method
The method uses a stereo matching network trained on synthetic data as a proxy, adapts it to realistic data, and distills its disparity predictions into a monocular depth network. Occlusion-aware unsupervised fine-tuning and regularization address cross-domain transfer and ambiguous photometric supervision.
- The pipeline pretrains a DispNet variant on synthetic Scene Flow data, fine-tunes it on realistic data, and uses the stereo network to supervise monocular depth estimation.Realistic-data fine-tuning can be supervised or unsupervised depending on available depth data.
- Directly training monocular networks on synthetic data is limited by scene sensitivity and the synthetic–realistic domain gap.Stereo matching generalizes better across domains because it learns pixelwise matching rather than directly inferring depth from semantic features.
- The proxy stereo network jointly predicts multiscale disparity maps and occlusion masks from stereo image pairs.Disparity uses L1 regression, while occlusion masks are trained as a binary classification task.
- Unsupervised fine-tuning masks photometric supervision in occluded regions and adds absolute and relative regularization terms.The absolute term keeps predictions close to the unfine-tuned disparity, while the relative term regularizes smoothness using unfine-tuned disparity gradients.
- The proposed unsupervised fine-tuning preserves sharper boundaries and more accurate occlusion-region predictions than Godard et al..The paper attributes the improvement to explicit occlusion handling and improved regularization, reporting better quantitative and qualitative results.
- The monocular network outputs multiscale virtual disparity maps and is trained by matching them to the fine-tuned stereo network’s predictions.Depth is recovered from virtual disparity using ˜d = bf/˜D, where b is camera baseline and f is lens focal length.
4 Experiments
Experiments evaluate the proxy stereo network, its fine-tuning strategies, and the distilled monocular model on KITTI, with additional qualitative comparisons and cross-dataset testing. Results show that stereo-to-monocular distillation uses synthetic data more effectively than direct monocular pretraining and remains competitive with limited depth supervision.
- Datasets and setup: Scene Flow provides synthetic stereo data for pretraining, while KITTI supplies realistic stereo, calibration, and LIDAR data for evaluation and fine-tuning.The Scene Flow collection contains more than 39,000 training stereo pairs and 4,000 testing pairs; KITTI Eigen split contains 22,600 training, 888 validation, and 697 testing image pairs.
- Implementation details: The proxy stereo network predicts disparity and occlusion masks, is trained with DispNetC-style components, and receives supervised or unsupervised KITTI fine-tuning.The network adds an occlusion-mask branch and uses randomized flipping, swapping, resizing, and cropping during stereo training; supervised fine-tuning uses 100 sampled KITTI images.
- Cross-domain synthetic data: Direct monocular training on Scene Flow fails on KITTI, whereas stereo-to-monocular distillation improves performance using no or only limited realistic depth data.The reported analysis attributes this difference to stronger stereo-network generalization and a smaller synthetic-to-real domain-transfer problem.
- Comparison with state-of-the-art methods: The best unsupervised model outperforms by 0.62 meters in RMS and reduces Sq Rel by 21.7%.StereoUnsupFt→Mono pt is trained on Cityscapes and KITTI; the comparison is against method.
- Comparison with state-of-the-art methods: Using only 100 sampled KITTI images for supervised fine-tuning, StereoSupFt100→Mono pt surpasses and previous supervised methods on all evaluation metrics.The paper also evaluates a model fine-tuned with all supervised data for reference and reports similar superiority under a 50-meter maximum-depth cap.
- Proxy stereo fine-tuning analysis: Unsupervised fine-tuning removes incorrect occlusion-region supervision and improves regularization, while stronger proxy stereo networks further improve monocular depth estimation.The proposed fine-tuning method surpasses quantitatively and qualitatively; replacing the proxy with PSMNet also improves the pipeline.
5 Conclusion
The proposed pipeline uses a stereo matching network as a proxy to provide dense supervision for monocular depth estimation. Fine-tuning strategies, including an unsupervised method, improve prediction detail and support state-of-the-art KITTI results, while future improvements remain possible.
- The method uses a deep stereo matching network as a proxy to learn depth from synthetic data and supervise the monocular network densely.
- The proposed unsupervised fine-tuning method preserves detailed prediction structures and improves final performance.
- State-of-the-art monocular depth estimation results are reported on the KITTI dataset.
- Future work includes more advanced stereo algorithms, improved fine-tuning strategies, and confidence measurement to filter stereo-output noise.
1 More Visualization Results on KITTI
The paper presents additional KITTI visualizations of its models, with depth ground truth obtained by interpolating sparse LIDAR measurements.
- Additional visualizations show the models’ depth results on the KITTI dataset.
- The KITTI depth ground truth is produced by interpolating sparse LIDAR depth maps.
2 Visualization Results on Cityscapes
The paper shows visualization results for its monocular depth model trained on the Cityscapes dataset.
- The StereoNoFt→Mono monocular depth model is visualized after training on the Cityscapes dataset.
- The Cityscapes visualization results are presented in Fig. 3.
3 PSMNet as the Proxy Stereo Network
Using PSMNet as the proxy stereo network improves monocular depth results over DispNetC in both supervised and unsupervised settings. This indicates that the pipeline’s performance increases with the performance of its stereo algorithm.
- PSMNet forms a 4-D feature volume by concatenating left-right features across disparity levels and uses stacked hourglass 3D convolutions to predict disparity.
- In both supervised and unsupervised settings, PSMNet produces more accurate monocular depth results than DispNetC.
- The pipeline’s performance increases with the performance of the proxy stereo algorithm.
- Table 1 reports KITTI results using PSMNet as the proxy stereo network.