Source-linked AI summary
SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes
Libo Sun, Jia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, Chunhua Shen
TL;DR
Self-supervised monocular depth estimation struggles where occlusions and moving objects violate multi-view consistency, producing inaccurate dynamic-region predictions and blurred boundaries. SC-DepthV3 uses a pretrained monocular model to provide pseudo-depth for auxiliary self-supervised constraints. The paper reports significantly better performance than previous alternatives across six challenging datasets.
Problem
Multi-view consistency is violated around moving objects and occlusions, where existing self-supervised methods have poor accuracy and blurred object boundaries.
Source: §1 · p. 1Method
SC-DepthV3 uses pretrained monocular pseudo-depth to guide self-supervised learning with Dynamic Region Refinement and Local Structure Refinement.
Source: §1 · pp. 1–2Results
SC-DepthV3 significantly outperforms previous alternatives across six challenging datasets, and ablations demonstrate the efficacy of the proposed modules.
Source: §1 · p. 2, §5 · p. 11Takeaways & Limitations
The pretrained depth model is trained once and used off the shelf in new scenes, so the method adds almost no extra cost compared with purely self-supervised methods.
Source: §1 · p. 1Using pseudo-depth somewhat degrades the claim of self-supervised learning, although the pretrained model adds almost no extra cost in practice.
Source: §4.3 · p. 11
Abstract
from arXiv · showhide
Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions and occlusions. Consequently, existing methods show poor accuracy in dynamic scenes, and the estimated depth map is blurred at object boundaries because they are usually occluded in other training views. In this paper, we propose SC-DepthV3 for addressing the challenges. Specifically, we introduce an external pretrained monocular depth estimation model for generating single-image depth prior, namely pseudo-depth, based on which we propose novel losses to boost self-supervised training. As a result, our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes. We demonstrate the significantly superior performance of our method over previous methods on six challenging datasets, and we provide detailed ablation studies for the proposed terms. Source code and data will be released at https://github.com/JiawangBian/sc_depth_pl
1 INTRODUCTION
Multi-view consistency supports self-supervised depth learning but breaks around moving objects and occlusions, leaving dynamic regions inaccurate and object boundaries blurred. SC-DepthV3 uses pretrained single-image pseudo-depth to guide self-supervised training through dynamic-region and local-structure refinements.
1 INTRODUCTION
- Multi-view consistency losses are violated by occlusions and moving objects, contributing to poor accuracy on dynamic datasets and blurred object boundaries.Prior approaches often exclude dynamic regions, which leaves them insufficiently regularized at inference, or model object motion, which is challenging.Source: §1 · p. 1
- SC-DepthV3 uses an off-the-shelf pretrained monocular depth model to generate pseudo-depth and designs losses that use this imperfect prior to constrain self-supervised depth estimation.The pretrained model is trained once and can be reused in new scenes.Source: §1 · p. 1
- Dynamic Region Refinement (DRR) samples dynamic–static point pairs and uses pseudo-depth ordinal relations to regularize depth predictions in dynamic regions.It uses the self-discovered mask from SC-Depth rather than requiring an external segmentation network.Source: §1 · p. 2
- Local Structure Refinement (LSR) matches surface normals and relative normal angles from pseudo-depth to improve depth details, especially at object boundaries.The relative-angle constraint is applied to sampled point pairs around image edges.Source: §1 · p. 2
- Experiments and ablations on six challenging datasets demonstrate the efficacy of SC-DepthV3; qualitative comparisons show sharper boundaries and more robust results in dynamic scenes than SC-Depth [1].The qualitative comparison covers six datasets, with dynamic scenes in the first three columns.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK
Prior self-supervised methods struggle with dynamic objects and boundaries, while zero-shot monocular depth models can provide useful but quantitatively imperfect priors. SC-DepthV3 incorporates such a prior into its self-supervised pipeline rather than relying on region exclusion, motion modeling, or conventional transfer approaches.
Self-supervised Monocular Depth Estimation
- Earlier self-supervised methods handle dynamic objects by excluding regions or modeling object motion, but exclusion weakens accuracy in those regions and motion modeling can be ill-posed.SC-DepthV3 instead leverages a pretrained single-image prior to address multi-view ambiguities.Source: §2 · p. 2
SC-Depth Series Methods
- The SC-Depth series progresses from scale-consistent depth in V1 to rotation handling in V2, with V3 addressing dynamic objects and blurred boundaries.The cited overview also describes the V1 and V2 methods as accurate in indoor and outdoor scenes.Source: §2 · pp. 2–3
Zero-shot Monocular Depth Estimation
- Zero-shot models such as LeReS and DPT generalize to unseen data, but their scale-shift-invariant predictions can lack quantitative accuracy for out-of-distribution data and 3D reconstruction.The paper identifies useful attributes in these predictions that can boost self-supervised depth estimation.Source: §2 · p. 3
Knowledge Transfer
- Unlike finetuning or mixed-dataset semi-supervised transfer, SC-DepthV3 uses a teacher trained once to generate pseudo-depth for new scenes, and its student has higher accuracy than the teacher.The method overview combines SC-Depth self-supervised losses, generated pseudo-depth, and DRR and LSR constraints on the network prediction.Source: §2 · p. 3, Figure 2 · p. 3
3 METHOD
SC-DepthV3 augments SC-Depth’s video-based self-supervision with LeReS pseudo-depth, extracting ranking, normal, and edge-based constraints to refine dynamic regions and local structure.
3.1 Self-supervised Depth Learning (SC-Depth)
- SC-Depth jointly trains depth and pose networks from video frames, using warped-image photometric and geometry consistency losses, edge-aware smoothness, and masking to support self-supervision.The self-discovered mask downweights geometrically inconsistent regions, including dynamics and occlusions.Source: §3.1 · pp. 3–4
3.2 Single-Image Depth Prior
- LeReS pseudo-depth supplies ordinal, smoothness, and boundary cues that SC-DepthV3 converts into Dynamic Region Refinement and Local Structure Refinement supervision.LeReS generalizes to unseen scenes but can have poor quantitative accuracy, motivating its use as a source of cues rather than direct depth targets.Source: §3.2 · p. 4
3.3 Dynamic Region Refinement
- Dynamic Region Refinement samples dynamic–static point pairs and applies depth-ranking loss to constrain predicted depth using pseudo-depth ordinals; mask-derived segmentation selects dynamic regions without extra computational cost.The self-discovered mask’s lowest 20% of weights are selected as potential dynamic regions; each is paired with a randomly sampled static point, alongside random pairs sampled across the whole image.Source: §3.3 · pp. 4–5, Figure 4 · p. 5
3.4 Local Structure Refinement
- Local Structure Refinement matches predicted-depth and pseudo-depth surface normals for overall structure, then constrains relative normal angles of edge-sampled pairs around object boundaries.The method samples only from edges because pseudo-depth is not accurate enough to maintain high-quality global structures.Source: §3.4 · pp. 5–6
3.5 Training
- Training combines the baseline losses with normal matching and both refinement losses, using empirical weights α=1, β=0.5, and γ=δ=ϵ=0.1.The depth and pose networks use ResNet-18 backbones; training runs for 100k iterations on each dataset.Source: §3.5 · p. 6
4 EXPERIMENT
Experiments span dynamic, almost-static, and boundary-focused evaluations. SC-DepthV3 outperforms prior methods on dynamic datasets, and ablations support the proposed refinement modules.
4.1 Datasets and Evaluation Metrics
- Evaluation covers three dynamic datasets, KITTI and NYUv2 as almost-static datasets, and iBims-1 for boundary and plane quality.Depth is evaluated after median-scale alignment; dynamic and static regions are evaluated separately.Source: §4.1 · pp. 6–7
4.2 Evaluation Results
- On DDAD, BONN, and TUM, SC-DepthV3 outperforms previous methods, particularly on DDAD dynamic regions and challenging indoor datasets.BONN and TUM contain a large proportion of dynamic regions; DDAD has fewer stopping cars and longer depth ranges than KITTI.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7
- On NYUv2, SC-DepthV3 outperforms previous approaches, whereas on KITTI it is comparable but does not surpass prior state-of-the-art methods.The authors attribute limited KITTI gains partly to many stopping cars and the small impact of dynamic objects there.Source: §4.2 · p. 9, Table 5 · p. 9
- On iBims-1, SC-DepthV3 significantly outperforms Monodepth2 and SC-DepthV2 on boundary and plane measures, while removing LSR degrades performance.All models are trained on NYUv2 for this comparison.Source: Table 6 · p. 9, §4.2 · p. 10
4.3 Ablation Studies
- On DDAD, replacing DRR's dynamic-focused sampling and confident ranking terms with random sampling and existing ranking loss significantly degrades performance.The compared alternatives are random sampling and the original ranking loss from.Source: Table 7 · p. 9, §4.3 · p. 10
- Replacing LSR's normal matching loss with edge-aware depth smoothness, or adding random sampling to edge-guided sampling, degrades depth accuracy.These variants are evaluated on DDAD.Source: §4.3 · pp. 10–11, Table 8 · p. 10
- LeReS, DPT, and a smaller LeReS variant produce inaccurate pseudo-depth on DDAD, yet all support high-accuracy results when used in the proposed training method.LeReS with ResNet-101 performs best among the tested variants; the authors hypothesize that shared normal information explains its advantage over DPT.Source: Table 9 · p. 10, §4.3 · p. 11
- Using an external pretrained monocular model somewhat weakens the self-supervised claim, though the authors say off-the-shelf models add almost no cost in new scenes.Those models are trained once on large-scale datasets and can be used in previously unseen scenes.Source: §4.3 · p. 11
5 CONCLUSION
SC-DepthV3 uses pseudo-depth from a pretrained monocular depth network to support self-supervised depth estimation in challenging dynamic videos. Across six challenging datasets, it significantly outperforms previous alternatives and predicts sharp, accurate depth maps, including from highly dynamic videos.
5 CONCLUSION
- SC-DepthV3 significantly outperforms previous alternatives across six challenging datasets, including dynamic and static scenes.The conclusion also reports that ablations demonstrate the proposed modules are effective.Source: §5 · p. 11
- SC-DepthV3 uses pseudo-depth generated by a pretrained monocular depth network to address challenges in self-supervised monocular depth estimation.The pseudo-depth is used for training from challenging dynamic videos.Source: §5 · p. 11
- The method predicts sharp, accurate depth maps even when trained from highly dynamic videos.The conclusion identifies dynamic objects and blurred object boundaries as issues the method addresses.Source: §5 · p. 11