PaperlayerView exact revision on arXiv

Source-linked AI summary

MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

Baorui Ma, Jiahui Yang, Donglin Di, Xuancheng Zhang, Jianxun Cui, Hao Li, Yan Xie, Wei Chen

arXiv:2601.22054v2cs.CVcs.AI

TL;DR

Metric depth estimation has lacked the scaling behavior of other foundation-model tracks because heterogeneous sensors, reconstruction methods, and camera intrinsics create noisy and ambiguous supervision. Metric Anything introduces Sparse Metric Prompts and pretrains on large heterogeneous 3D data, then distills the result into a prompt-free student. The models show clear scaling and state-of-the-art generalization across diverse metric-depth tasks.

  • Problem

    Metric depth estimation has not shown the scaling trends of relative depth because heterogeneous sources introduce domain gaps, noisy supervision, and metric ambiguity from varying camera intrinsics.

  • Method

    Metric Anything uses randomly masked Sparse Metric Prompts as a universal interface for pretraining on heterogeneous 3D data, followed by distillation into a prompt-free student.

  • Results

    The pretrained and distilled models achieve state-of-the-art performance across ten downstream tasks while demonstrating a clear scaling trend in metric depth estimation.

  • Takeaways & Limitations

    Metric Anything establishes prompt-based pretraining as a scalable path toward general-purpose metric depth perception from heterogeneous 3D sources.

  • Takeaways & Limitations

    The work assumes central-projection cameras and leaves specialized camera models and architectural scaling unexplored.

Abstract

from arXiv · show

Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging due to heterogeneous sensor noise, camera-dependent biases, and metric ambiguity in noisy cross-source 3D data. We introduce Metric Anything, a simple and scalable pretraining framework that learns metric depth from noisy, diverse 3D sources without manually engineered prompts, camera-specific modeling, or task-specific architectures. Central to our approach is the Sparse Metric Prompt, created by randomly masking depth maps, which serves as a universal interface that decouples spatial reasoning from sensor and camera biases. Using about 20M image-depth pairs spanning reconstructed, captured, and rendered 3D data across 10000 camera models, we demonstrate-for the first time-a clear scaling trend in the metric depth track. The pretrained model excels at prompt-driven tasks such as depth completion, super-resolution and Radar-camera fusion, while its distilled prompt-free student achieves state-of-the-art results on monocular depth estimation, camera intrinsics recovery, single/multi-view metric 3D reconstruction, and VLA planning. We also show that using pretrained ViT of Metric Anything as a visual encoder significantly boosts Multimodal Large Language Model capabilities in spatial intelligence. These results show that metric depth estimation can benefit from the same scaling laws that drive modern foundation models, establishing a new path toward scalable and efficient real-world metric perception. We open-source MetricAnything at http://metric-anything.github.io/metric-anything-io/ to support community research.

1 Introduction

Metric depth estimation has struggled to scale because heterogeneous sensors, reconstruction methods, and camera intrinsics introduce domain gaps, noise, and metric ambiguity. Metric Anything addresses this with Sparse Metric Prompts and large-scale pretraining, yielding scaling and broad downstream generalization.

  • Motivation: Metric depth lacks the scaling trends seen in relative depth because it must learn absolute distances from more difficult heterogeneous data.The challenge is compounded by source-specific noise, artifacts, and camera-dependent biases.
  • Approach: Sparse Metric Prompts randomly mask depth maps to provide a universal interface that decouples spatial understanding from sensor and camera biases.The paradigm avoids task-specific prompt engineering and specialized architectures.
  • Approach: Approximately 20M image–depth pairs spanning diverse sources and over 10,000 cameras support scalable metric-depth pretraining.The dataset is designed to address the heterogeneity that limits small, carefully curated datasets.
  • Results: The pretrained model and distilled prompt-free student achieve state-of-the-art performance across ten downstream tasks without task-specific engineering.Tasks include prompt-driven depth completion, super-resolution, radar-camera estimation, monocular depth, calibration, reconstruction, and VLA planning.
  • Results: MetricAnything exhibits a clear scaling trend and strong overall downstream performance.The result establishes prompt-based pretraining as a scalable path toward general-purpose metric depth perception.

2 Related Work

Prior metric-depth work introduced camera-aware, domain-specific, and handcrafted processing strategies, while sparse-prompt methods generally remained task-specific. Metric Anything instead emphasizes a data-driven pretraining paradigm for noisy, heterogeneous sources.

  • Relative Depth Estimation: Relative-depth methods scale effectively from heterogeneous weak supervision, but their ordinal formulation differs fundamentally from absolute metric depth.Relative-depth supervision includes pairwise orderings, stereo/MVS cues, and pseudo-depth.
  • Metric Depth Estimation: Metric-depth methods progressed from fixed-intrinsics settings toward camera-aware formulations using metric heads, canonicalization, and intrinsic-conditioned features.Representative systems include ZoeDepth, Metric3D, DepthPro, UniDepth, and MoGe-2.
  • Metric Depth Estimation: MoGe-2 and related approaches use complex non-learnable processing, whereas Metric Anything discards handcrafted processing in favor of data-driven learning from noisy sources.The proposed direction aims to learn robust metric-depth representations directly from heterogeneous 3D data.
  • Prompt-Based Methods: Prompt-based metric-depth methods use sparse LiDAR, RGB-D, or radar cues, but commonly rely on task-specific designs for densification and cross-dataset transfer.Examples include Marigold-DC, OMNI-DC, and PriorDA.

3 Method

Metric Anything builds a large heterogeneous metric-depth dataset, trains with randomly sampled Sparse Metric Prompts, and distills the pretrained model into a prompt-free student. Its distillation uses distance-balanced supervision and an inverted ViT–DPT skip-connection design.

  • 3.1 Multi-Source Data Collection: The training corpus combines reconstructed, captured, and rendered 3D data standardized as metric depth maps with validity masks.Reconstructed data contain artifacts and missing regions, captured data are noisy and sparse, and rendered data provide noise-free geometric supervision.
  • 3.1 Multi-Source Data Collection: Approximately 20 million image–depth pairs from over 10,000 camera models span diverse real and synthetic scenes.This aggregation provides the scale and camera diversity needed for unified pretraining.
  • 3.2 Pretraining via Sparse Metric Prompt: Sparse prompts sample 2,000–40,000 valid pixels, roughly 1% of each image, and the model propagates these metric constraints into dense depth.The model is trained to correct errors in noisy and incomplete prompts.
  • 3.2 Pretraining via Sparse Metric Prompt: Prompt preparation aligns sparse samples with a prior depth map using pixel-wise scale alignment and global metric-depth recovery.The final prompt combines the refined and corrected representations before injection into the model.
  • 3.3 Prompt-Free Model Distillation: Deep ViT features are injected into deeper decoding layers while shallow features feed shallower layers, reversing the conventional skip-connection scheme.This design emphasizes semantic reasoning near the final prediction stage.
  • 3.3 Prompt-Free Model Distillation: A prompt-free student is trained from teacher-generated pseudo-depth labels to preserve metric understanding using only RGB inputs.The teacher converts sparse, noisy prompts into dense supervision across the collected real-world images.
  • 3.3 Prompt-Free Model Distillation: The student uses a distance-balanced inverse-depth loss that retains near-field sensitivity while extending supervision to long-range regions.The log-space formulation is intended for the wide depth range of the teacher-generated pseudo labels.

4.1 Prompt-Based Downstream Task

Metric Anything is evaluated on zero-shot depth super-resolution, completion, and radar–camera depth estimation using sparse metric prompts. The pretrained model generalizes across prompt types and unseen sensing modalities without task-specific prompt alignment.

  • 4.1.1 Zero-Shot Depth Super-Resolution and Completion.: Zero-shot evaluation uses sparse or low-resolution depth maps as prompts without task-specific fine-tuning.The study compares post-aligned and prior-based monocular depth baselines across multiple prompt types.
  • 4.1.1 Zero-Shot Depth Super-Resolution and Completion.: The pretrained model consistently outperforms post-aligned and prior-based baselines across prompt types and datasets.It operates after a single pretraining stage with simple sparse prompts, without task-specific prompt alignment.
  • 4.1.1 Zero-Shot Depth Super-Resolution and Completion.: Figure 5 visualizes improved recovery of missing depth regions and structure for super-resolution and completion.The visualization complements the quantitative comparisons in Table 1.
  • 4.1.1 Zero-Shot Depth Super-Resolution and Completion.: Table 1 reports zero-shot depth super-resolution and completion using AbsRel %.The table summarizes the metric used for comparison in this subsection.
  • 4.1.2 Radar-Camera Depth Estimation.: Radar–camera depth estimation combines RGB frames with extremely sparse and noisy mmWave Radar measurements.Radar point clouds are substantially sparser than LiDAR and may be misaligned with other onboard sensors.
  • 4.1.2 Radar-Camera Depth Estimation.: The radar experiments evaluate transfer to an unseen sensor because Radar data are excluded from pretraining.They compare a teacher trained from scratch with the same teacher finetuned from pretrained weights on nuScenes.
  • 4.1.2 Radar-Camera Depth Estimation.: Table 2 compares radar–camera depth estimation on nuScenes using millimeters as the unit.The experiment includes scratch training, finetuning, and prior radar–image fusion methods.
  • 4.1.2 Radar-Camera Depth Estimation.: Finetuning with Radar prompts achieves state-of-the-art performance, nearly doubling the accuracy of the from-scratch counterpart and surpassing prior fusion methods.The result supports adaptation of randomly sampled sparse-prompt pretraining to a new sensing modality.

4.2 Prompt-Free Downstream Tasks

The prompt-free student generalizes across monocular depth, geometry, camera calibration, multi-view reconstruction, VLA planning, and spatial reasoning tasks. Results emphasize balanced performance across diverse domains, state-of-the-art geometry and calibration, and improved spatial understanding.

  • Monocular Depth Estimation.: Metric Anything distinguishes monocular metric depth maps from camera-coordinate 3D point maps as two evaluation tracks.
  • Monocular Depth Estimation.: The Student-DepthMap achieves Rank 1 on Sun-RGBD and Middlebury, with AbsRel 0.085 versus 0.156 for the comparison on Sun-RGBD.
  • Monocular Depth Estimation.: On nuScenes and ETH3D, the method ranks second in absolute error while often achieving the highest accuracy at the stricter δ3 threshold.
  • Monocular Depth Estimation.: On Booster, the method outperforms all baselines with AbsRel 0.282 and remains second-best in Log10 on synthetic Sintel.
  • Recovering Camera Intrinsics.: Student-PointMap pseudo-label distillation achieves state-of-the-art monocular geometry performance across prediction heads and initialization regimes.
  • Recovering Camera Intrinsics.: Camera focal length is recovered by optimizing a point-map projection under unit aspect ratio and an approximately centered principal point, achieving the best mean and median angular errors.
  • Multi-view Metric 3D Reconstruction.: The prompt-free model improves multi-view metric reconstruction, particularly metric-scale estimation, using single-feed-forward inference without cross-frame correction or bundle adjustment.
  • VLA Planning: The model achieves the best average LIBERO success rate and improves VLA spatial understanding without depth inputs during training or inference.

5 Ablation Study

Ablations show that performance improves with more training data and test-time resolution, while proposed architectural and loss choices improve utilization of learned representations and long-range depth. Prompt density exhibits diminishing returns, and balance weighting trades near- and far-field accuracy.

  • AbsRel on KITTI 8× depth super-resolution falls from 5.22 at 5% data to 2.34 at 100%, revealing a clear data-scaling trend.
  • Inverse Skip-Connection better leverages ViT semantic representations than classic U-Net skip connections when trained with consistent pseudo-labels.
  • Latency is evaluated at VGA, HD, and 4K resolutions under FP32 on an H200 GPU, alongside parameter counts and FLOPs.
  • Test Time Resolution Scaling: Test-time resolution scaling from 1× to 3× and 9× progressively recovers finer structures and high-frequency details without fine-tuning.
  • The distance-balanced inverse-depth loss matches standard inverse-depth performance at close range but performs better as depth range increases.
  • Increasing sampled prompt pixels from 500 to 64,000 yields diminishing accuracy gains while increasing computational complexity.
  • The ablation includes real-world night scenes with LiDAR prompts and a sensor setup whose limited vertical field of view leaves image regions without metric-depth cues.
  • A balance weight that is too large emphasizes distant regions at the expense of nearby detail, whereas a weight that is too small under-supervises distant regions.

6 Generalizability to Unseen Sensors, Scenarios, and Extreme Environmental Conditions

The pretrained model is evaluated without fine-tuning across unseen sensor configurations, degraded environmental conditions, and visual domains absent from training. Reported results indicate reliable depth estimation despite sensor artifacts, low light, adverse weather, distortion, and unusual imagery.

  • A real-world vehicle test uses three cameras and a 128-beam solid-state LiDAR with sensor type and spatial arrangement differing from training.
  • Night driving reduces signal quality through low signal-to-noise ratios, lost texture and color, artificial-light saturation, and high-contrast shadows.
  • Rain and fog produce LiDAR anomalies or artifacts through reflectivity problems, testing robustness under degraded sensor signals.
  • The deployment evaluates depth super-resolution and completion in an unseen sensor configuration without fine-tuning.
  • The model maintains reliable depth estimation in night-time and rainy or foggy conditions, with supplementary video showing stable long-term temporal predictions.
  • Zero-shot tests cover panoramic, fisheye, and diverse in-the-wild images, including cartoons, grayscale images, and artistic renderings.
  • Qualitative results report accurate metric depth across visual domains absent from training, including extreme distortion and novel viewpoints.

7 Training Details

Metric Anything uses large-scale training, prompt-free distillation, robust losses, and standard metrics to evaluate metric depth across diverse settings.

  • Training Setup: Training uses a zero-shot protocol, except ScanNet training and testing follow the Map Anything protocol for multi-view reconstruction.The exception preserves the protocol used by the comparison method.
  • Training Setup: The model is trained on 144 H200 GPUs for 100k steps, with separate learning rates for the ViT backbone, DPT head, and prompt layer.The ViT uses 1×10^-6, while the DPT head and prompt layer use 1×10^-5.
  • Distillation: The pretrained model is distilled into a prompt-free student using teacher-generated pseudo-labels and two training paradigms.The paradigms include from-scratch training and fine-tuning across depth-map and 3D-point-map outputs.
  • Loss Functions: The teacher uses Robust MAE on inverse depth for real-world data, discarding the top 20% largest-error pixels to reduce noisy-supervision effects.Synthetic data additionally uses scale-and-shift-invariant multi-scale gradient loss.
  • Loss Functions: The student combines distance-balanced inverse-depth supervision with SSI-MAGE loss to balance near- and far-range pseudo-label sensitivity.The distance-balanced loss operates in log space, while SSI-MAGE further targets boundary sharpness and geometry.

8 Limitations

The study assumes central-projection cameras and does not extend to specialized camera models; architectural scaling also remains unexplored.

  • Scope Boundary: The method has not been extended to non-central or non-pinhole camera configurations, and model-architecture scalability remains unexplored.The authors identify architectural expansion as a possible direction for more complex and diverse scenarios.

9 Conclusion

Metric Anything presents scalable metric-depth pretraining on diverse noisy 3D data, using Sparse Metric Prompts and prompt-free distillation to support broad downstream generalization.

  • Conclusion: Metric Anything learns metric depth from diverse noisy 3D sources without task-specific architectures or manually engineered prompts.Sparse Metric Prompts separate spatial reasoning from sensor and camera biases.
  • Conclusion: The pretrained model and distilled prompt-free student achieve state-of-the-art results across a wide range of downstream tasks.The conclusion frames these results as supporting more efficient general-purpose real-world depth perception.
Loading 2601.22054v2…