PaperlayerView exact revision on arXiv

Source-linked AI summary

DeFM: Learning Foundation Representations from Depth for Robotics

Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar, Cesar Cadena, Marco Hutter

arXiv:2601.18923v1cs.ROcs.CV

TL;DR

Robotics lacks general-purpose depth encoders despite depth’s value for geometry and sim-to-real transfer. DeFM trains a self-supervised depth foundation model and distills it into compact variants, achieving strong results across perception and control tasks. Its pretrained encoders support off-the-shelf depth-based robotic learning, while fine-detail obstacle sensitivity remains bounded by low-resolution policy inputs.

  • Problem

    General-purpose depth encoders are unavailable, while RGB adaptation and task-specific depth encoders limit geometric fidelity and generalization across tasks, environments, and sensors.

  • Method

    DeFM uses DINO-style self-distillation on 60 M depth images, metric-aware input normalization, and distillation into compact CNN and ViT models.

  • Results

    DeFM generalizes across classification, segmentation, navigation, locomotion, and manipulation, including robust sim-to-real transfer, without task-specific fine-tuning.

  • Takeaways & Limitations

    DeFM provides an off-the-shelf depth encoder family, and its distilled variants make strong depth representations practical for resource-constrained robotic systems.

  • Takeaways & Limitations

    Policies occasionally struggle with very small or thin obstacles, which the authors attribute primarily to the 64 × 40 depth input resolution used during policy training.

Abstract

from arXiv · show

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/

I. INTRODUCTION

Depth is an important robotic modality for robust sim-to-real transfer, yet general-purpose depth encoders remain unavailable. DeFM addresses this gap with self-supervised depth pretraining, metric-aware normalization, and compact distilled models that generalize across tasks and sensors.

  • Motivation: Depth is invariant to lighting, texture, and color variation, making it effective for robotic generalization and sim-to-real transfer.Depth sensors support locomotion, navigation, and manipulation across robotic platforms.
  • Motivation: Existing approaches repurpose RGB encoders for depth or train task-specific encoders from scratch, limiting geometric fidelity and cross-task generalization.The paper identifies the absence of pretrained general-purpose depth encoders as the central gap.
  • DeFM: DeFM is a self-supervised foundation model trained exclusively on depth images using a DINO-style objective.Its learned embeddings show emergent semantic clustering despite lacking texture and color.
  • Results: DeFM achieves state-of-the-art performance across classification, segmentation, locomotion, navigation, and manipulation, with robust sim-to-real transfer.The encoder can be applied frozen across these robotics tasks.
  • DeFM: 60 M depth images support DeFM pretraining, while distilled models from 3 M to 30 M parameters span CNN and ViT architectures for broad robotic adoption.The compact variants target resource-constrained robotic systems.
  • DeFM: A novel input normalization strategy preserves metric awareness across multiple scales.The strategy is presented as a key design choice for depth pretraining.

2) Self-Distillation:

Robotic visual representation learning builds on self-supervised objectives and increasingly specialized foundation models. Prior work uses masking, contrastive learning, language alignment, geometric prediction, and distillation to obtain transferable or action-relevant representations.

  • Self-Distillation: Self-distillation processes augmented views through student and teacher branches, with momentum teacher updates preventing representational collapse.DINOv2 extends this framework with large-scale curated data, improved training recipes, and regularization.
  • Self-Distillation: Masked-image methods reconstruct hidden regions from visible patches, while iBOT combines masked-image modeling with DINO-style self-distillation for dense features.These objectives encourage global structure and patch-level representation learning.
  • Vision-Language Modeling: Vision-language models align visual representations with captions, providing semantic grounding and zero-shot transfer for classification and retrieval.Their objective uses contrastive learning on large web-scraped datasets.
  • Geometric Models: Geometric foundation models unify outputs such as depth, relative pose, and pointmaps, targeting multi-view reconstruction and camera-pose estimation.Examples include DUSt3R, MASt3R, VGGT, and Depth Anything 3.
  • Model Distillation: Agglomerative vision models distill multiple heterogeneous foundation models into one backbone that consolidates complementary capabilities.RADIO combines strengths such as zero-shot text grounding and dense correspondence.
  • Robotics Representations: Robotics requires grounded, action-relevant features emphasizing geometry and physical affordances beyond general semantics.Action-centric video pretraining, including R3M and VIP, introduces temporal and kinematic priors for embodied tasks.
  • Robotics Representations: Masked video modeling extends self-supervised learning to robot demonstrations, multicamera views, and multi-view geometric consistency.MVP and its extensions adapt masked modeling to real-world robotic scenarios.
  • Robotics Applications: DINO features have been used for semantic retrieval, pixel-level alignment, voxel representations, and zero-shot planning in robotics.These applications reflect the value of dense, robust feature spaces for downstream robotic tasks.

1) Locomotion and Navigation:

Depth supports geometric control and sim-to-real transfer, but existing robotics systems largely rely on task-specific encoders. DeFM addresses this gap with transferable depth representations for locomotion, navigation, and manipulation.

  • Locomotion and Navigation: Depth is valuable for locomotion and navigation because it captures terrain geometry and is invariant to lighting, texture, and color variation.Earlier systems often converted depth into height or elevation maps, which can be noisy and require accurate odometry in real scenes.
  • Locomotion and Navigation: Navigation systems use depth or depth-derived features to support reinforcement learning in diverse, large-scale environments.Habitat enables large-scale training for PointGoal Navigation, where depth-based agents can perform strongly in unseen environments.
  • Manipulation: Manipulation methods use depth to construct voxel grids, point clouds, or object meshes, but these representations can struggle in cluttered or dynamic scenes.Such pipelines also require careful perception calibration, motivating approaches that consume egocentric visual observations directly.
  • Manipulation: Dexterous-hand policies often distill privileged teacher information into visual students, with synthetic-image training and augmentation increasing training demands.This teacher-student strategy is used to address the high-dimensional control space of dexterous hands and visual sim-to-real gaps.
  • Shared Representation Gap: Across these applications, depth encoders are typically trained from scratch with each policy, limiting efficiency and generalization across tasks, environments, and sensors.The paper argues that large-scale self-supervised pretraining can provide universal depth features for locomotion, navigation, and manipulation.

3) Objective Enhancements:

DeFM adapts DINO-style self-distillation to depth using multi-crop, masked-patch objectives and regularization. It also uses scale-aware normalization and distillation into compact architectures for robotic deployment.

  • Model and Input Design: DeFM trains a ViT-L model with 307 M parameters on curated depth data before distilling it into smaller ViT-S and ResNet models.The compact models target resource-constrained robotic systems.
  • Model and Input Design: A novel input normalization strategy is introduced to learn metric-aware features across diverse spatial scales.The normalization is presented as a critical design choice for retaining metric awareness in depth representations.
  • Multi-Crop Training: For each depth image, large global crops, smaller local crops, and partially masked global crops are processed by momentum teacher and student networks.The teacher produces target distributions from global crops, while the student processes local and masked global views.
  • Image-Level Objectives: The global-crop loss aligns student representations of masked global views with teacher representations of unmasked global views using class-token features.This objective encourages consistency between differently processed views of the same depth image.
  • Image-Level Objectives: The local-crop loss aligns student representations of local crops with teacher representations of global crops through class-token features.The formulation connects fine-scale student views to broader teacher context.
  • Patch Objective and Regularization: The iBOT patch loss trains masked-patch predictions against teacher targets from corresponding visible patches to promote dense spatial features.The total objective combines the three primary losses with a KoLeo regularizer to prevent feature collapse.

B. Dataset

DeFM’s pretraining dataset combines 60.4 M depth images from 18 datasets, while its three-channel log-compressed input represents relative and metric depth across operational scales. The mixture balances diversity, scale, and sensor noise to support generalization across robotic environments.

  • Dataset construction: 60.4 M depth images from 18 datasets provide broad coverage across sensor fidelity and domain.The collection combines monocular-depth-estimated, synthetic, and real depth sources.
  • Dataset construction: MDE-derived images add object-centric semantic variety, while synthetic data supplies clean metric depth and domain diversity.These sources complement each other by expanding scale and object coverage.
  • Dataset construction: Real depth data exposes the model to sensor artifacts, missing data, and noise profiles needed for sim-to-real robustness.The real-data category includes measurements from diverse depth sensors.
  • Dataset construction: The combined sources balance diversity, scale, and noise fidelity to maximize generalization across unknown robotic environments.This is the stated purpose of the dataset mixture.
  • Input normalization: A three-channel log-compressed representation captures depth structure across operational regimes while preserving metric scale.The channels combine global relative depth with mid-range and far-range metric depth cues.
  • Input normalization: The stacked channels preserve fine-grained near-field structure, metric depth at multiple scales, and stable gradients for pretraining.This design targets manipulation-scale detail and long-range navigation regimes.

D. Implementation Details

DeFM is trained with a DINO-style self-distillation setup using a ViT-L/14 teacher-student framework. The teacher is also distilled into smaller transformer and CNN encoders, with BiFPN features preserving dense spatial information.

  • Pretraining: DeFM pretraining uses a DINO-style self-distillation objective with global and local crops of each depth image.The implementation uses two global crops and eight local crops.
  • Pretraining: The ViT-L/14 model is trained for 625k iterations on 96 NVIDIA GH200 GPUs with a distributed batch size of 3,072.Training uses AdamW, float16 precision, warmup, and cosine schedules.
  • Distillation: DeFM-L/14 serves as a frozen teacher for distillation into ViT-S and lightweight ResNet, RegNet, and EfficientNet students spanning 3 M to 30 M parameters.The student EMA weights are retained as the final distilled models.
  • Distillation: A BiFPN is added to CNN students so they retain dense spatial features required for prediction beyond global pooled representations.The BiFPN fuses feature maps from three backbone resolutions.
  • Distillation: DINO losses match CNN global pooled features to the teacher class token, while iBOT losses match BiFPN spatial outputs to teacher patch tokens.Student and teacher feature maps are aligned at 16 × 16 spatial resolution.

V. EXPERIMENTS

The experiments test whether frozen DeFM features transfer across tasks, domains, sensors, and model sizes. Evaluations combine qualitative PCA analysis with linear probing and comparisons against RGB-pretrained and distilled baselines.

  • Evaluation design: The evaluation targets transfer across different tasks, domains, and sensors using qualitative and quantitative protocols.The study includes PCA visualization and frozen-encoder linear probing.
  • Qualitative analysis: PCA features from cup depth images remain consistent across four sensors, clustering functional parts such as handles and rims.The first three PCA components are mapped to RGB channels for visualization.
  • Qualitative analysis: DeFM features highlight functional object parts across varied cups, drawers, and cabinets, supporting generalized semantic representations.The drawer and cabinet result comes from the qualitative manipulation visualization.
  • Classification: Classification uses the ImageNet-Depth-1K benchmark with frozen-encoder linear probing and top-1 and top-5 KNN evaluation.The benchmark consists of RGB images converted to depth using a monocular depth estimator.
  • Baselines: The comparisons include RGB-domain foundation models and distilled models representing different pretraining paradigms and size variants.RGB baselines receive min-max depth normalization, channel stacking, and ImageNet normalization.

2) Results:

DeFM delivers strong transfer across classification and segmentation while retaining useful performance in compact models. The results support its use across diverse robotic settings, although deployment latency depends on architecture and unoptimized runtimes.

  • Classification: 84.79% Top-5 KNN and 71.72% linear probing are achieved by DeFM-L/14 on ImageNet-1k-Depth.DeFM-L/14 obtains the highest reported scores on both metrics.
  • Classification: Up to 10% improvement across all three classification metrics is reported for DeFM-S/14 against existing models in its size category.The result motivates smaller DeFM variants for efficient robotic deployment.
  • Compact models: RegNetY-400MF, with 4.1 M parameters, can outperform 22.1 M-parameter ViT-S RGB baselines despite its smaller size.The comparison illustrates a favorable compact-model tradeoff in feature quality.
  • Segmentation: DeFM-L/14 achieves the highest mIoU on four of five segmentation datasets, demonstrating transfer across varied domains.The benchmarks span indoor, tabletop manipulation, and outdoor navigation environments.
  • Segmentation: Up to 30% higher mIoU than corresponding baselines is achieved by DeFM-S/14 on several segmentation benchmarks.Distilled ResNet models also outperform similar-sized baselines on almost all datasets.
  • Deployment: Orin inference speed depends strongly on architecture, with some RegNetY and EfficientNet operations poorly optimized in standard PyTorch.Parameter count alone does not determine edge-device latency.
  • Deployment: Reported timings omit TensorRT and ONNX runtime optimizations, which are expected to reduce latency, especially on Jetson AGX Orin.The measurements therefore describe standard PyTorch execution rather than optimized deployment.

VI. ROBOTIC EXPERIMENTS

This section evaluates DeFM representations on four robotic reinforcement-learning tasks spanning navigation, manipulation, and locomotion. Unless otherwise stated, the downstream policy uses a frozen DeFM encoder while only the policy network is trained.

  • Task selection: Four downstream tasks cover Habitat Point-Goal Navigation, embodiment-aware navigation, dexterous grasping, and quadrupedal ladder climbing.The tasks test navigation, manipulation, and locomotion under diverse geometric and embodiment constraints.
  • Navigation: Habitat Point-Goal Navigation requires an agent to reach target 3D points across multi-room, multi-floor indoor environments.The policy uses DD-PPO and an RNN to process visual-encoder features.
  • Navigation: Embodiment-aware navigation tests long-range obstacle avoidance while accounting for the Unitree B2W robot’s kinematics and physical footprint.The task uses a single front-facing depth camera across diverse terrains and complex environments.
  • Manipulation: Dexterous grasping uses a KUKA-Allegro arm-hand setup to grasp diverse objects, with depth features combined with proprioception.The student policy is trained through teacher-student distillation from privileged state information.
  • Locomotion: Quadrupedal ladder climbing requires ANYmal to climb ladders of varying sizes and angles using depth-camera features and proprioception.The student policy processes depth embeddings and proprioceptive measurements through recurrent control components.
  • Evaluation protocol: Across the evaluations, frozen DeFM encoders provide the representations while only downstream policy networks are trained.The benchmark setup compares DeFM models with scratch-trained and frozen RGB-foundation-model baselines in competitive size categories.

2) Results:

DeFM’s frozen features outperform competing frozen foundation-model baselines for Habitat navigation and improve generalization in embodiment-aware navigation. The model also transfers to diverse simulated and real-world environments, with particular gains for out-of-distribution obstacle handling.

  • Habitat navigation: Frozen DeFM models outperform all other frozen foundation-model baselines on Gibson and MatterPort3D Habitat validation sets measured by SPL.DINOv3 performs strongly on depth, but both DeFM models outperform it; DINOv2 and Theia perform less well.
  • Habitat navigation: DeFM-ResNet-50 achieves competitive navigation performance against a scratch-trained ResNet-50, showing the value of pretraining for avoiding task-specific encoder training.The result supports using distilled DeFM features directly in depth-based navigation policies.
  • Embodiment-aware navigation: DeFM improves testing performance in Abandoned Cable and Modern City Downtown relative to the VAE baseline for embodiment-aware navigation.The policy using DeFM performs better at avoiding thin and out-of-distribution obstacles such as fences, traffic signs, and lamp posts.
  • Embodiment-aware navigation: DeFM consistently produces fewer collision failures than the other compared encoders in embodiment-aware navigation.The analysis distinguishes collisions as primarily perception-related failures and timeouts as policy-local-minimum failures.
  • Representation transfer: DeFM’s metric-aware normalization is hypothesized to explain its stronger transfer than DINOv3, which treats depth as an intensity channel.The comparison emphasizes retaining metric depth understanding for embodiment-aware robotic tasks.
  • Real-world transfer: A DeFM policy generalizes across indoor, urban, park, and construction environments, navigating endpoints ranging from 5 m to 100 m.Deployments include pedestrians, scooters, barriers, furniture, and uneven terrain.

4) Discussions and Limitations:

DeFM operates out of the box for several downstream robotics settings and remains effective under realistic depth noise. Its distilled and frozen variants provide useful performance, although the policy can miss fine obstacles at low input resolution.

  • Practical deployment: Unlike the VAE baseline, DeFM works without a specific noise model, depth clipping, or additional task-specific heuristics.The comparison concerns real-world navigation transfer across diverse environments.
  • Limitations: The policy occasionally struggles with very small or thin obstacles such as wires, poles, and mesh structures.The authors attribute this limitation primarily to the 64 × 40 depth input resolution used during policy training.
  • Dexterous grasping: Fine-tuned DeFM achieves the highest dexterous-grasping performance across both evaluated noise scenarios.The frozen DeFM encoder outperforms scratch-trained and fine-tuned ImageNet encoders while remaining competitive with fine-tuned DINOv3.
  • Dexterous grasping: Under Kinect noise, frozen models lose substantial performance, but frozen DeFM shows a smaller decline than the other frozen models.Fine-tuned DeFM is least affected by the noise-model change.
  • Qualitative representation analysis: DeFM’s PCA embeddings cluster scene components such as countertops, background surfaces, robot arms, and manipulable objects in real-world DROID images.This provides qualitative evidence of structured semantic features in cluttered kitchen scenes.

D. Locomotion: Quadruped Ladder Climbing

DeFM is evaluated for perceptive quadrupedal ladder climbing using a teacher-student controller and depth-camera observations. Its features perform competitively in simulation and retain consistent ladder structure representations during real-world deployment, while broader scaling and hardware coverage remain future directions.

  • Method and setup: The ladder-climbing student combines CNN depth embeddings with proprioception and recurrent processing to mimic a privileged-information teacher.Evaluation varies rung radii, ladder angles, and random disturbances.
  • Simulation results: 90.14%: DeFM matches the CNN baseline’s ladder-climbing success rate across varying rung radii and ladder angles.Each configuration is evaluated over 750 episodes with 1024 environments, using success within 15 seconds as the criterion.
  • Real-world transfer: DeFM consistently assigns similar features to ladder structures in noisy real-world deployment images.The qualitative PCA analysis suggests robustness to noise from real-world stereo depth cameras.
  • Efficiency: Distilled DeFM variants retain most of the large model’s representational strength while improving efficiency for resource-constrained robotic systems.The compact models are intended to make strong depth representations practical to deploy.
  • Limitations and future work: The authors identify occasional representation artifacts, limited real-world task diversity, and future expansion to LiDAR depth representations as open directions.They also propose increasing dataset diversity, model capacity, and training iterations.
Loading 2601.18923v1…