Learning to Discriminate Scene Structures Makes Self-Supervised Depth Learning Scalable

1 Introduction

• Across diverse monocular video domains, existing methods can perform worse than models trained on a single dataset instead of benefiting from added diversity.The paper describes this as a barrier to scaling self-supervised depth estimation.Source: §1 · p. 2
• Under self-supervision, high-level depth features can encode dataset-specific structural biases rather than foreground–background relations and perspective geometry, creating conflicts across domains.On a homogeneous dataset, those biases can compensate for weak supervision; diverse datasets may require inconsistent biases.Source: §1 · p. 2
• MaSS combines contextual features with a decoupled structural pathway guided by contrastive scene-structure discrimination, which organizes representations by scene-layout similarity.The resulting structural features aim to capture global layout and perspective geometry rather than dataset-specific biases.Source: §1 · p. 2
• MaSS outperforms prior state-of-the-art self-supervised methods by a large margin and is the first self-supervised approach to gain clearly from more diverse training data.The gains include improved generalizability, particularly on out-of-distribution scenes.Source: §1 · p. 2

2 Related Works

• Figure 2 contrasts dataset-specific biases in a conventional depth model and insufficient geometric cues in DINOv2 with MaSS’s contextual and transferable structural representations.The contextual representation captures object detail and foreground–background relations, while the structural representation encodes global scene layout.Source: Figure 2 · p. 3
• Prior self-supervised depth approaches commonly train on single domains with limited-scale datasets, while early work has explored heterogeneous videos and self-supervised depth foundation models.The cited heterogeneous-video approach jointly estimates depth, camera motion, object motion, and camera intrinsics [21]; recent scaling efforts are [6] [7].Source: §2 · p. 3
• Related approaches draw on pretrained representations, contrastive organization of depth features, or scene-conditioned depth distributions; MaSS combines these directions by decoupling context from structure.Its structural features are organized by scene-layout similarity, and depth prediction is conditioned on those representations.Source: §2 · p. 3

3 Methodology

• Dataset-specific inductive biases appear in intermediate representations and can impair predictions on scenes with different depth distributions, motivating explicit learning of generalized structural representations.The paper gives examples of driving-scene models predicting a vanishing point and small-room models failing to capture the more uniform depth of large halls.Source: §3 · p. 3

3.1 MaSS Model

• MaSS bridges a trainable depth encoder and a frozen pretrained vision foundation model through bidirectional interaction, then uses anchored and decoupled projectors for contextual and structural representations.The interaction injects geometric cues from the depth encoder into frozen VFM features; the contextual pathway preserves semantic relationships, while the structural pathway focuses on complementary geometry.Source: Figure 3 · p. 4, §3.1 · pp. 4–5

3.2 MaSS Learning Objective

• Scene-structure discrimination uses contrastive learning to make structural descriptors similar for samples with similar global depth distributions and dissimilar for structurally different samples.The discrepancy metric compares depth distributions using histograms of downsampled predicted depth maps, with soft binning used to reduce sensitivity to high-frequency artifacts.Source: §3.2 · pp. 5–6

3.3 Depth Prediction from Learned Representations

• MaSS concatenates the contextual and structural feature pyramids, reduces their channel dimension with a 1 × 1 convolution, and feeds them to the depth decoder.The contextual pathway adapts VFM features into depth-related context, while the structural pathway captures generalized scene geometry.Source: §3.3 · p. 6

4.1 Experimental Setups

• Experiments use four training-and-evaluation datasets, eight additional zero-shot datasets, and median or least-squares alignment for scale-invariant or affine-invariant evaluation.Training freezes DINOv2 and requires a one-epoch initialization stage before applying scene-structure discrimination.Source: §4.1 · pp. 6–7

4.2 Compare with SoTA Self-supervised Methods

• MaSS outperforms prior methods on all four individually trained datasets and improves clearly under mixed-dataset training, while reproduced prior methods drop significantly.Mixed batches sample from four datasets, with training epochs reduced to one-third to keep iteration counts comparable.Source: §4.2 · pp. 7–8, Table 1 · p. 7

4.3 Zero-Shot Generalizability Evaluation

• MaSS maintains strong generalization across all zero-shot datasets, especially out-of-distribution scenes, and MaSS† consistently outperforms individually trained MaSS.It is competitive with supervised depth foundation models on several outdoor and out-of-distribution datasets, but underperforms them on standard indoor datasets.Source: Table 2 · p. 8, §4.3 · p. 9
• On zero-shot tests, MaSS produces sharper object boundaries and finer details while retaining scene geometry in unseen complex environments.Table 2 compares specialized indoor and outdoor methods, mixed-dataset MaSS†, large-scale self-supervised methods, and supervised depth foundation models.Source: §4.3 · pp. 8–9, Table 2 · p. 8

4.4 Ablation Study

• Removing contrastive supervision, the decoupled structural pathway, or the interaction architecture degrades performance; MaSS also beats alternative backbone-and-head designs.Replacing DINOv2 with DINOv3 changes performance only marginally, consistent with pretrained VFMs supplying semantic priors rather than directly decoding depth.Source: §4.4 · pp. 9–10, Table 4 · p. 9

4.5 Scalability Analyses

• Adding 50K TartanAir samples, expanding training from 150K to 200K, leaves training-domain performance largely stable and improves the most challenging unseen domains.The reported average changes on DIML and DIODE are +4.3% in Abs Rel and +3.0% in δ1; the authors describe this as preliminary evidence because the scale remains far below supervised depth foundation models.Source: §4.5 · p. 10, Table 5 · p. 10

• MaSS couples a depth encoder with a pretrained VFM and decouples structural from contextual representations, learning generalized scene structure through contrastive learning.The conclusion identifies dataset-specific structural bias as the key barrier to large-scale mixed-domain self-supervised depth estimation.Source: §5 · p. 10

Outline

• Appendices A–C cover technical details, supplementary experiments, and limitations with future-work directions, respectively.Appendix A includes objective analyses, training details, and dataset descriptions; Appendix B adds alignment-protocol, generalizability, and geometric-robustness experiments.Source: Outline · p. 14

A.1 Details of Learning to Discriminate Scene Structures

• The structural discrepancy metric selects in-batch positive and negative pairs, and Fig. S1 illustrates how the objective guides structural-representation learning.The examples show positives can cross indoor/outdoor labels when layouts are similar, while scenes within the same broad category can remain negative; metric settings and masking align with the illustrated relationships.Source: Figure S1 · p. 14, §A.1 · p. 15

A.3 Details of Training-Evaluation Datasets

• Training uses KITTI, DDAD, NYUv2, and nuScenes, spanning driving, dynamic urban, indoor, adverse-weather, and varied-lighting data.The passage reports image-triplet counts and evaluation splits for these datasets.Source: §A.3 · p. 15

• Outdoor zero-shot evaluation uses Waymo Open, DrivingStereo, and Cityscapes, with test or validation splits of 2,216, 2,000, and 1,525 images, respectively.DrivingStereo’s split includes 500 images per weather condition; Cityscapes uses ground-truth depth from [64].Source: §A.4 · p. 15
• Evaluation covers 7-Scenes (1,700 images), DIML’s 500 close-range outdoor images, and DIODE’s 771-image official test set (325 indoor, 446 outdoor), supporting indoor and out-of-distribution assessment.DIML’s outdoor images have depth characteristics that differ significantly from typical driving datasets. DIODE’s indoor subset spans a larger depth range than commonly used indoor benchmarks, while its outdoor subset mainly contains closer-range scenes than typical driving environments.Source: §A.4 · p. 16

A.5 Data Distribution and Model Generalizability Analyses

• On zero-shot outdoor datasets, depth distributions resemble training data but RGB distributions differ, while MaSS substantially improves generalization amid appearance shifts.The passage attributes prior methods’ degradation largely to appearance shifts rather than depth-range differences, and describes conventional encoder architectures with restricted pretraining as less robust.Source: §A.5 · p. 16, Figure S3 · p. 17
• Zero-shot indoor datasets align closely with NYUv2 in both RGB and depth distributions, a pattern that favors supervised depth foundation models.The passage connects their larger performance gap from self-supervised approaches to similar scene statistics and mappings in supervised training data.Source: §A.5 · p. 16
• MaSS performs comparably to fully supervised depth foundation models on DIML and DIODE, which differ substantially from training data in both RGB and depth distributions.These datasets are presented as challenging tests of out-of-distribution generalization.Source: §A.5 · p. 17, Figure S3 · p. 17

B.1 Additional Quantitative Results using Different Alignment Protocols

• LSQ* aligns in inverse depth and downweights distant regions, reducing the influence of high-variance pixels on the fitted parameters.Depth-domain LSQ gives equal importance to absolute errors, so distant regions can dominate its fit.Source: §B.1 · pp. 17–18
• Under LSQ* alignment, KBR’s DIODE results match its originally reported values, the error metrics are substantially reduced, and MaSS still outperforms prior methods.The paper retains median alignment in its main text because it evaluates scale-invariant depth without additional shift fitting; LSQ* changes the evaluation objective.Source: §B.1 · p. 18, Table S1 · p. 18
• Under LSQ*, MaSS’s gap with DA v2 [37] narrows; MaSS outperforms it on several outdoor and out-of-distribution datasets but underperforms on relatively easier indoor scenes.The paper cautions that direct comparison may not be fully well-defined because DA v2 is trained for scale- and shift-invariant inverse depth.Source: §B.1 · p. 18

B.2 Adapting Depth/Geometry Foundation Models

• With DepthAnythingV2 [37] or VGGT [38] replacing the original VFM encoder, MaSS gains performance and outperforms those models’ original depth-prediction heads.The authors frame MaSS as annotation-free adaptation for foundation models that output affine-invariant depth or disparity, which cannot be directly deployed in many practical scenarios.Source: §B.2 · pp. 18–19, Table S2 · p. 19

B.3 Generalizability Evaluation on In-the-Wild Data

• MaSS produces fine-grained depth predictions with consistent structural layouts across challenging in-the-wild scenes, while the compared methods show structural confusion, indoor bias, or errors under extreme RGB shifts.The scenes include reflections, dense vegetation, complex structures, thin power lines, extreme lighting, and motion blur.Source: §B.3 · p. 19
• SC-DepthV3† deteriorates under mixed-dataset training, whereas Jasmine [9] can produce significant depth errors when extreme RGB shifts destabilize its diffusion process.SC-DepthV3 trained on NYUv2 performs poorly on complex unseen structures and tends toward an indoor bias; Jasmine [9] otherwise produces detailed predictions and improved generalization from diffusion priors.Source: §B.3 · p. 19

B.4 Geometric Fidelity Evaluation on In-the-Wild Data

• With no ground-truth depth for the in-the-wild images, geometric fidelity is assessed by synthesizing novel viewpoints with slight changes from the original view.The comparison uses monocular reconstruction approaches VGGT [38] and MoGe [62].Source: §B.4 · p. 19
• MaSS achieves reconstruction results comparable to VGGT [38] and MoGe [62], with significantly less training data.VGGT produces noisy depth maps and reconstruction artifacts, while MoGe yields high-quality reconstructions in most cases with minor local inconsistencies.Source: §B.4 · p. 19, Figure S4 · p. 20

C Discussion and Future Work

• Systematically curated datasets spanning diverse scenes remain scarce, and collection-platform constraints and sensor calibration and annotation costs often leave existing datasets relatively homogeneous.The cited examples are typical indoor rooms and common driving environments.Source: C Discussion and Future Work · p. 20
• The authors identify large-scale curation of diverse datasets from in-the-wild imagery as a promising direction for advancing SSMDE and extending self-supervision to other vision tasks.They state that limited data diversity may constrain the potential benefits of self-supervision.Source: C Discussion and Future Work · pp. 20–21