Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation

1. Introduction

• Self-supervised MDE methods use video geometry but face constrained data diversity and limited model capacity, leaving performance gaps relative to supervised approaches.Supervised methods, in turn, rely on large datasets with per-pixel depth annotations.Source: §1 · pp. 1–2
• CLIP captures global semantic context while DINO extracts local spatial details, but naive fusion can misalign these features and produce suboptimal depth representations.CLIP lacks spatial precision for depth estimation, whereas DINO features lack contextual depth hierarchies.Source: Figure 1 · p. 1, §1 · p. 2
• Hybrid-depth progressively transfers CLIP and DINO priors into depth estimation, using language alignment to harmonize their different-grained features.The framework proceeds from coarse depth sensing to fine depth estimation.Source: §1 · p. 2
• Coarse sensing uses close–distant patch comparisons along lane markings, aligning CLIP and DINO features with depth-related text prompts through contrastive losses.The proxy task leverages geometric priors from autonomous-driving scenes.Source: §1 · p. 2, Figure 3 · p. 3
• Fine estimation combines auxiliary camera pose information with pixel-wise alignment of hybrid-grained features to learnable depth prompts.The framework is designed to integrate as an enhanced depth encoder in existing self-supervised MDE pipelines.Source: §1 · p. 2, Figure 2 · p. 2
• Hybrid-depth is reported to significantly improve existing self-supervised MDE methods as a compatible depth encoder.The contribution summary also describes language-guided aggregation of CLIP semantics and DINO spatial details as addressing granularity mismatch.Source: §1 · pp. 2–3

2.1.1. Supervised Monocular Depth Estimation

• Supervised MDE methods have improved depth regression and generalization, but collecting large amounts of manually labeled data is resource-consuming.Approaches include multi-scale fusion, classification formulations, and additional priors such as segmentation and geometry.Source: §2.1.1 · p. 3

2.1.2. Self-supervised Monocular Depth Estimation

• Self-supervised MDE uses monocular-video geometry and has expanded to multi-frame cost volumes and supplementary cues, yet methods can face performance bottlenecks and overfitting due to limited data.ManyDepth uses multiple test-time frames and geometric constraints through a cost volume.Source: §2.1.2 · p. 3

2.2. Foundation Models and Prompt Learning

• Foundation models learn transferable image-text or visual representations, while few studies apply them to self-supervised MDE and prior CLIP depth approaches remain limited.DepthCLIP's zero-shot depth predictions are described as ambiguous; cited learnable-prompt methods still underperform previous MDE methods.Source: §2.2 · pp. 3–4

3. Hybrid-depth

• Hybrid-depth aggregates CLIP global semantics and DINO local spatial details under contrastive language guidance.This is the framework's initial coarse-feature stage.Source: §3 · p. 4
• For fine estimation, Hybrid-depth uses auxiliary camera pose information and pixel-wise language alignment of hybrid-grained features.Source: §3 · p. 4
• As a plug-and-play depth encoder, Hybrid-depth is reported to improve continuous depth capture and the accuracy of self-supervised MDE methods.Source: §3 · p. 4

3.1. Coarse Depth Sensing

• Lane geometry supplies a proxy for depth ordering, while multi-scale CLIP and DINO features are resized and concatenated into fused features.CLIP contributes global semantics and DINO spatial details; depth increases along lane markings.Source: §3.1 · p. 4

3.1.1. Patch Selecting

• Seven fused-feature patches are sampled along lane labels, ordered from higher to lower image rows, with random columns and regular spacing within each row.This sampling strategy is intended to reduce selection of patches with nearly identical depths in ego-view images.Source: §3.1.1 · p. 4

3.1.2. Depth Prompt Design

• Rank-distance text prompts, from “very distant” to “very close,” encode the ordinal depth relationships assigned to image patches.The prompts are formatted as “This patch appears to be [depth token]” and encoded by CLIP’s text encoder.Source: §3.1.2 · p. 4

3.1.3. Contrastive Learning

• Intramodal and language-guided contrastive losses encourage depth-ordered patch similarities and align visual features with corresponding text cues.The text cues also act as a granularity calibrator for CLIP–DINO fusion; only the visual encoder is trained in this stage.Source: §3.1.3 · pp. 4–5

3.2.1. Learnable Depth Prompt

• Learnable depth tokens replace manually crafted depth words and are encoded as depth text embeddings.The tokens are initialized by random sampling from a normal distribution with mean 0 and standard deviation 0.02.Source: §3.2.1 · p. 5

3.2.2. Pixel-level Language Alignment

• Four-scale CLIP–DINO features are aligned pixel-wise with learnable depth prompts, producing depth features that replace the original DepthNet features.The alignment uses inner products between reshaped visual features and depth text embeddings.Source: §3.2.2 · p. 5

3.2.3. Reconstruction and Smoothness Loss

• The aligned features are upsampled to predict depth with an auxiliary camera pose, while training uses image reconstruction and edge-aware smoothness losses.The photometric error combines pixel-space L1 distance and SSIM; the text encoder is frozen while the visual encoder, learnable tokens, and upsampling layers are trained.Source: §3.2.2 · p. 5, §3.2.3 · pp. 5–6
• Hybrid-depth-augmented methods improve the original methods and outperform other evaluated approaches across all metrics.The reported comparisons cover KITTI Eigen split self-supervised methods and prior MDE methods using CLIP.Source: Table 1 · p. 6, Table 2 · p. 6

4.1. Implementation Details

• Experiments use ResNet-50-based CLIP and DINOv2 encoders, train for 10 epochs with batch size 16, and set 256 learnable depth and pose tokens.The reported implementation uses AdamW, an initial learning rate of 1e-4, and a single NVIDIA A800 GPU.Source: §4.1 · p. 6

4.2.1. Results on KITTI

• Hybrid-depth improves three classical MDE methods and outperforms other self-supervised approaches across all evaluation metrics; it also exceeds CLIP-based few-shot or supervised methods.The reported comparisons include Monodepth2, ManyDepth, and Mono-VIFI augmented with Hybrid-depth.Source: §4.2.1 · pp. 6–7

4.2.2. Qualitative Results

• On challenging KITTI images, Hybrid-depth produces smoother ground-region depth while retaining sharp object boundaries and reducing edge artifacts.The figure caption specifically notes continuous ground-region predictions and sharp object edges.Source: Figure 5 · p. 7, §4.2.2 · p. 7

4.3. Ablation Studies

• Ablations report that coarse-to-fine learning improves performance beyond a stronger backbone, and removing feature alignment lowers performance across metrics.The alignment comparison covers cross-modal contrastive loss and intramodal contrastive learning.Source: §4.3 · p. 7, Table 3 · p. 7, Table 4 · p. 8

Q3: Is depth instruction necessary as a granularity cali-

• Removing depth instruction significantly reduces performance, while single-encoder variants underperform the CLIP–DINO combination.The paper attributes complementary strengths to CLIP’s semantic understanding and DINO’s fine-grained spatial details.Source: Q3: Is depth instruction necessary as… · p. 8, Table 5 · p. 8, Table 6 · p. 8

Q5: What is the effect of learnable depth tokens’ count?

• Performance varies non-monotonically with learnable depth-token count, improving up to an optimal range such as 256 tokens before degrading.The authors describe a capacity–overfitting trade-off as token count increases.Source: Q5: What is the effect of learnable… · p. 8, Table 7 · p. 8

Q6: Can Hybrid-depth benefit other 3D perception tasks such as BEV prediction?

• Hybrid-depth significantly improves the reported BEVDet and FB-BEV baseline configurations on the nuScenes validation set.The comparison uses input images at 256 × 704 resolution.Source: Q5: What is the effect of learnable… · p. 8, Table 8 · p. 8

• Hybrid-depth aggregates CLIP and DINO features under contrastive language guidance, then uses auxiliary camera-pose information for fine depth estimation and depth-instruction calibration.The framework is presented as a plug-and-play depth encoder intended to improve capture of continuous depth variations.Source: §5 · p. 8