Source-linked AI summary
DINOcular: Self-Supervised Visuospatial Representations
Farkhat Almukhamedov, Sami Azirar, Hermann Blum
TL;DR
Modern vision foundation models largely use RGB despite embodied systems having explicit depth, leaving monocular spatial understanding constrained. DINOcular learns joint RGB-D representations by combining depth-derived geometric priors with visual features through inter-patch and intra-patch fusion and self-supervised training. The resulting features improve 3D geometry performance over comparable-scale prior methods while remaining competitive on semantic segmentation.
Problem
Modern vision foundation models are trained almost exclusively on RGB even though many embodied systems have depth sensors, leaving depth-based geometric information outside the learned representation.
Method
DINOcular combines depth-derived geometric priors with visual features through inter-patch and intra-patch fusion, trained with a scalable self-supervised RGB-D objective.
Results
DINOcular improves over prior representations of comparable scale on multiple 3D geometry benchmarks while remaining competitive on standard RGB-D semantic segmentation tasks.
Takeaways & Limitations
The study demonstrates that RGB and depth can be combined into generalizing features that transfer across semantic and geometric tasks.
Takeaways & Limitations
The study has not validated whether its gains persist when scaled to substantially larger and more diverse data, and spatial and semantic specialization can trade off.
Abstract
from arXiv · showhide
We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern vision foundation models are trained almost exclusively on RGB images, many embodied systems have access to explicit depth sensing, which provides geometric information that monocular inputs cannot recover. Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry benchmarks, and remains competitive when probed for standard RGB-D semantic segmentation tasks.
1 INTRODUCTION
Modern vision foundation models largely rely on RGB despite widespread depth sensing, leaving spatial understanding constrained by monocular scale ambiguity. DINOcular addresses this gap with self-supervised RGB-D fusion that improves 3D awareness while preserving semantic transfer.
- Motivation: Modern vision foundation models are trained almost exclusively on RGB, despite depth sensors being available in many embodied systems.The passage identifies this modality mismatch in robots, autonomous vehicles, mixed-reality headsets, and smartphones.
- Motivation: Depth data is difficult to use at foundation scale because sources are scarce, heterogeneous, noisy, and governed by different failure modes.Recent learned reconstruction enables large-scale depth generation and homogenization of different depth sources.
- Prior limitations: RGB-only geometric methods and task-specific RGB-D fusion architectures do not fully exploit direct geometric sensing or automatically generalize across tasks.The related approaches either infer geometry from RGB or use task-specific fusion mechanisms.
- Approach: DINOcular learns joint visuospatial representations from RGB-D observations using a scalable self-supervised framework.Its architecture integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch interactions.
- Results: DINOcular improves over comparable-scale prior methods on 3D geometry benchmarks while remaining competitive on RGB-D semantic segmentation tasks.The reported evaluations include Probe3D, NYU DepthV2, and SUNRGBD.
2 RELATED WORK
Vision foundation features are strongly semantic but weakly geometric, while geometry-prediction systems use RGB to predict structure rather than represent it. Related RGB-D methods are often task-specific, motivating generalizing representations that combine RGB and depth inputs.
- Self-supervised vision models: Self-distillation and masked reconstruction are the two dominant scalable families for self-supervised vision representation learning.DINO and iBOT exemplify self-distillation, while masked reconstruction targets pixel-level recovery.
- Spatial understanding: Frozen vision foundation model features are strongly semantic but only weakly geometric across viewpoints.Probe3D and Lexicon3D report limited geometric consistency, while view consistency emerges after explicit multi-view finetuning.
- Geometry prediction: RGB geometry-prediction systems shape feature spaces to predict 3D structure rather than represent it directly.CroCo-derived systems such as DUSt3R, MASt3R, and VGGT use RGB as input and geometry as output.
- Research gap: Existing RGB-only and geometry-prediction families leave depth outside the feature extractor, whereas generalizing RGB-D representations remain absent.This motivates learning representations that combine RGB and depth inputs directly.
- RGB-D fusion: RGB-D segmentation methods use architectures and training objectives specialized to segmentation, limiting clean transfer to other tasks.Their representations inherit the inductive bias of the segmentation target.
- RGB-D fusion: DFormerv2 injects depth through pooled-patch attention biases, while DINOcular embeds depth into token positions.The distinction is whether depth modifies attention weights or enters the feature representation through positional encoding.
- Self-supervised RGB-D learning: Other self-supervised methods treat depth as a reconstruction target or require point clouds at inference rather than providing a native RGB-D image backbone.MultiMAE reconstructs depth, while Concerto and Sonata rely on point-cloud-based pipelines.
3 METHOD
The method combines depth-derived geometry with RGB features at inter-patch and intra-patch levels, then trains the representation with semantic and multiview self-supervised objectives.
- Geometry Prior: 3D RoPE encodes patch coordinates and average depth as a principled inter-patch geometry prior for self-attention.It extends rotary positional encoding into a third dimension, allowing attention to use relative spatial relationships and patches that are not metrically close.
- Geometry Prior: Local depth features are fused with patchified RGB embeddings to capture intra-patch shape information such as edges and surface smoothness.A lightweight depth patch-embedding layer is combined with RGB features through a linear projection.
- Learning Objective: The semantic objective combines DINO image-level distillation, iBOT patch-level prediction, and KoLeo regularization.Masked iBOT inputs include both RGB and intra-patch depth, while masked tokens retain positional encoding based on average patch depth.
- Learning Objective: Multiview training gives two students different views of the same object and uses known point maps and depth to identify corresponding spatial regions.The setup also uses global views, local crops, and randomly sampled 3D points for correspondence learning.
- Learning Objective: Lmv-contrastive is selected as the multiview loss most compatible with Lsem, producing the full objective L = LDINO + LiBOT + λ1Lmv-contrastive + λ2LKoLeo.Training batches are sampled to contain image pairs to which the multiview loss applies.
- Learning Objective: Combining multiview training with intra-patch depth initially biased features toward spatial tasks and reduced visual information, so depth-embedding dropout was added.The dropout prevents the multiview objective from relying exclusively on depth information.
4 EXPERIMENTS
The experiments evaluate DINOcular across semantic and geometric tasks, using ablations and comparisons with RGB-only, RGB-D, supervised, and self-supervised baselines. Results show that depth-aware, multi-view training improves geometric awareness while preserving strong semantic transfer, with trade-offs across objectives and model sizes.
- Evaluation Setup: The evaluation covers RGB-D semantic segmentation, 3D correspondence estimation, object pose estimation, and depth regression/reconstruction.These tasks probe semantic transfer, viewpoint consistency, pose prediction, and correlation with depth input.
- Architecture Ablation: Table 1 ablates inter- and intra-patch geometry encoding using NYUDepthv2 segmentation and NAVI and ScanNet correspondence estimation.The comparison isolates architectural choices for encoding depth at different spatial scales.
- Loss Ablation: The multi-view objective causes segmentation collapse when used alone, while combining it with DINO preserves segmentation and further enhances 3D correspondence.MV-Cosine and MV-iBOT also reduce correspondence in this setting, whereas the contrastive objective improves correspondence and is compatible with DINO.
- Semantic Segmentation: DINOcular features significantly outperform DFormerv2 and DINOv2 on semantic probing, while DINOcular S can surpass the three-times-larger DINO ViT-B in most cases.Leaving out the multi-view objective yields slightly better semantic probing on these datasets, although the general model retains an advantage at comparable data scale.
- 3D Correspondence Estimation: Multi-view training provides a clear benefit for correspondence under large viewpoint changes, and DINOcular-S outperforms all reference models on Probe3D-ScanNet.The authors suggest depth may be especially advantageous in cluttered, larger-scale environments such as ScanNet.
- 3D Object Pose Estimation: DINOcular-L outperforms all baselines and DINOv3 ViT-B at the two coarser pose-error thresholds, but the smaller model is weak at the 1 cm-1° threshold.The smaller-model weakness may relate to feature blurriness across the image plane.
5 LIMITATIONS
DINOcular’s evaluation is limited to moderate data scale, with larger and more diverse scaling left unvalidated. The method also exposes a spatial–semantic specialization trade-off and complicates exact data-scale comparisons.
- Scaling DINOcular to substantially larger and more diverse data remains unvalidated.The study focuses on moderate data scale.
- DINOcular outperforms RGB-D and RGB-only foundation models, including DINO, at the studied data scale.Exact same-scale comparison is difficult because DINO training data is unavailable and DINOcular requires multi-view data.
- Weakening or removing the multiview objective improves semantic performance but reduces the representations’ 3D awareness.
6 CONCLUSION
DINOcular learns self-supervised visuospatial features from RGB-D observations that generalize across semantic and geometric tasks. At the same data and model scale, it shows an advantage over other approaches while motivating future scaling studies.
- DINOcular learns visuospatial features from RGB-D observations in a self-supervised manner.
- The learned features generalize across semantic and geometric tasks.
- DINOcular shows a significant advantage over other approaches at the same data and model scale.
- The results motivate scaling the approach to larger data sources and models.
A DIFFERENT DEPTH SOURCES AS INPUT
The study tests DINOcular with multiple depth sources, including monocular estimates and sensor-derived depth, and reports robust probing performance across depth-input variations.
- DINOcular inference is tested with MapAnything, DepthAnything3, and sensor-derived depth sources.The sensor comparison includes raw sensor depth, densified sensor depth, and monocular depth on NYU in Table 5.
- The resulting features appear robust to variations in depth input on ADE20k and NYU Depth v2 RGB-D semantic segmentation.
B EFFICIENCY
DINOcular is compared with RGB-only and depth-biased alternatives using a shared core backbone. It is faster and more memory-efficient than DFormerv2, while adding modest overhead relative to RGB-only.
- DINOcular is faster and uses less memory than DFormerv2.
- DINOcular has modest latency and memory overhead relative to the RGB-only backbone.
- Table 6 compares performance and efficiency, reporting latency and memory as relative differences compared with DINOcular.
C QUALITATIVE RESULTS
DINOcular exhibits robust 3D correspondence and cross-view feature consistency, while producing clean semantic segmentation across indoor and outdoor scenes, including lighting changes.
- 3D Correspondence and Feature Consistency: DINOcular establishes dense, accurate 3D correspondences under severe viewpoint variations, where DINO ViT-B and DUNE struggle.The qualitative comparison reports fewer outliers and more accurate matches than DINO and DUNE.
- 3D Correspondence and Feature Consistency: Its extracted features remain structurally aligned across environments and viewpoints while separating semantic parts such as horns, body, and legs.PCA visualizations show stable feature maps and reduced noisy, pixelated artifacts relative to other feature spaces.
- Semantic Segmentation: Compared with DFormerv2 and MultiMAE, DINOcular produces clean, contiguous semantic regions with sharply delineated object boundaries in complex indoor scenes.The baselines show spatial fragmentation and noisy patch assignments, especially on textured floors and furniture.
- Semantic Segmentation: DINOcular generalizes to outdoor urban layouts, correctly identifying primary categories and maintaining coherent parsing under illumination shifts such as shadows.The outdoor qualitative results specifically highlight robustness to lighting changes.
D CODE
The paper provides its code on the project page.
- The authors provide code through the project page.
E TRAINING DETAILS
Training uses four nodes, each equipped with four A100 GPUs; additional hyperparameters and training details appear in Table 7.
- Models are trained on 4 nodes with 4 A100 GPUs per node.
- Table 7 lists the hyperparameters and pre-training details, with highlighted rows indicating differences by model size.
- The qualitative figures include PCA feature visualizations, semantic segmentation comparisons, and outdoor urban-scene results.