PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation

Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, Bingyi Kang

arXiv:2412.14015v4cs.CV

TL;DR

Depth foundation models produce strong relative depth but remain scale-ambiguous for metric estimation. Prompt Depth Anything uses low-cost LiDAR to guide Depth Anything through multi-scale decoder fusion, reporting state-of-the-art results on ARKitScenes and ScanNet++ and benefits for downstream applications.

  • Problem

    Depth foundation models estimate strong relative depth but suffer from scale ambiguity, while metric-depth training data can lack LiDAR or precise GT depth.

    Source: §1 · pp. 1–2

  • Method

    Prompt Depth Anything fuses low-cost LiDAR with a Depth Anything model at multiple decoder scales and uses simulated LiDAR and real-data pseudo GT depth for training.

    Source: §1 · p. 2

  • Results

    The method reports state-of-the-art performance across ARKitScenes and ScanNet++ and benefits 3D reconstruction and generalized robotic grasping.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that the foundation model and prompt can be replaced with DepthPro and vehicle LiDAR, respectively, while retaining downstream application benefits.

    Source: §1 · p. 2

    With iPhone LiDAR prompts, the method cannot handle long-range depth, and temporal LiDAR flicker leads to flickering depth predictions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks. For the first time, we introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution. Our approach centers on a concise prompt fusion design that integrates the LiDAR at multiple scales within the depth decoder. To address training challenges posed by limited datasets containin both LiDAR depth and precise GT depth, we propose a scalable data pipeline that includes synthetic data LiDAR simulation and real data pseudo GT depth generation. Our approach sets new state-of-the-arts on the ARKitScenes and ScanNet++ datasets and benefits downstream applications, including 3D reconstruction and generalized robotic grasping.

1. Introduction

Depth foundation models produce strong relative depth but remain scale-ambiguous, limiting accurate metric depth estimation. Prompt Depth Anything uses low-cost LiDAR to guide a depth foundation model, with a scalable training pipeline, and reports strong benchmark and downstream results.

1. Introduction

  • The paper frames metric depth estimation as a downstream task for depth foundation models, prompted with metric information such as low-cost LiDAR.The authors select LiDAR because it provides precise metric scale information and is widely available in devices such as smartphones.Source: §1 · p. 2
  • Prompt Depth Anything introduces low-cost LiDAR prompting for accurate metric depth estimation, integrating metric information into a depth foundation model.The approach is based on Depth Anything and is designed to achieve 4K-resolution output.Source: §1 · p. 2
  • The method addresses training-data gaps by simulating LiDAR for synthetic data and generating pseudo GT depth for real data, with an edge-aware loss.The loss uses pseudo-GT gradients to emphasize edges while retaining accurate depth supervision in textureless areas.Source: §1 · p. 2
  • Prompt Depth Anything reports state-of-the-art performance across ARKitScenes and ScanNet++ and benefits 3D reconstruction and generalized robotic grasping.The authors also report that a zero-shot model outperforms other methods in non-zero-shot testing, and that the foundation model and prompt can be replaced.Source: §1 · p. 2

2. Related Work

Depth foundation models generalize well for relative depth but remain scale-agnostic, so metric depth estimation remains challenging. Prior work uses metric-data fine-tuning or camera intrinsics, while auxiliary-sensor methods often complete sparse depth; Prompt Depth Anything instead uses sparse depth as a prompt.

Monocular depth estimation

  • Depth foundation models estimate relative depth effectively but remain scale-agnostic, leaving metric depth estimation challenging.Earlier approaches include global distribution classification, metric-depth fine-tuning, and incorporating camera intrinsic parameters.Source: §2 · p. 2

Depth estimation with auxiliary sensors

  • Auxiliary-sensor depth completion commonly fills sparse depth maps, but many methods are tested on simulated rather than real indoor LiDAR data.Real testing setups have required both low-power and high-power LiDAR sensors; datasets using iPhone and FARO LiDAR include ARKitScenes and ScanNet++.Source: §2 · pp. 2–3
  • Unlike Depth Prompting [43], which fuses sparse depth with foundation-model features to post-process its output, this approach uses sparse depth as a prompt to obtain a specific output.The authors describe the latter distinction as prompting the foundation model.Source: §2 · p. 3

3. Method

Prompt Depth Anything prompts a DPT-based depth foundation model with low-cost LiDAR to estimate metric depth. Its multi-scale fusion architecture combines image and depth features, while a training pipeline simulates synthetic LiDAR and generates real-data pseudo GT depth with edge-aware supervision.

3. Method

  • Low-cost LiDAR supplies metric scale information as a prompt to a depth foundation model, addressing the metric-depth ambiguity of monocular models.The paper focuses on DPT-based models for real-time performance, rather than diffusion-based depth models because of their high computational cost.Source: §3 · p. 3, §3.1 · p. 3

3.2. Prompt Depth Anything

  • At each DPT decoder scale, the LiDAR depth map is resized, encoded by shallow convolutions, projected to the image-feature dimension, and added to intermediate features.The fusion layers are zero-initialized, and the encoder and decoder are initialized from the foundation model; the design adds 5.7% computational overhead for a 756 × 1008 image.Source: §3.2 · pp. 3–4
  • Adaptive LayerNorm, CrossAttention, and ControlNet conditioning designs performed worse than the proposed fusion block in the reported experiments.The authors suggest these designs may not effectively use the pixel alignment between low-resolution LiDAR and output depth.Source: §3.2 · p. 4

3.3. Training Prompt Depth Anything

  • Synthetic LiDAR simulation downsamples GT depth to 192 × 256, samples a distorted grid with stride 7, and interpolates remaining values using RGB similarity with KNN.The sparse-anchor method is intended to simulate LiDAR noise; direct downsampling instead leads the model to learn depth super-resolution without correcting LiDAR noise.Source: §3.3 · p. 4
  • For ScanNet++, Zip-NeRF generates pseudo GT depth from unblurred iPhone frames and DSLR videos to address holes and poor edges in FARO-rendered depth.The authors note that reconstruction remains challenging in textureless and reflective regions, where FARO depth is comparatively good.Source: §3.3 · p. 4
  • The edge-aware loss uses FARO depth for overall depth supervision and pseudo-GT depth gradients to encourage accurate edges.The authors set λ = 0.5 and report that the loss improves depth prediction in experiments.Source: §3.3 · pp. 4–5

3.4. Implementation Details

  • Implementation uses a ViT-large backbone, normalizes LiDAR depth to [0, 1], and trains for 200K steps after a 10K-step warm-up.Training uses batch size 2 on 8 GPUs, with AdamW learning rates of 5e-6 for the ViT backbone and 5e-5 for other parameters.Source: §3.4 · p. 5

4. Experiments

Experiments compare Prompt Depth Anything across ARKitScenes and ScanNet++, then examine design choices, generalization, reconstruction, and robotic grasping. The reported results include state-of-the-art comparisons and benefits for both downstream applications.

4.1. Experimental Setup

  • Experiments use HyperSim, ARKitScenes, and ScanNet++; ARKitScenes training and evaluation use 40K and 5K images, respectively.For ScanNet++, training uses about 60K images from 230 scenes, with about 5K validation images from 20 scenes.Source: §4.1 · p. 5

4.2. Comparisons with the State of the Art

  • Prompt Depth Anything consistently outperforms existing methods, and its HyperSim-only model surpasses non-zero-shot models on ScanNet++.The comparisons cover monocular depth estimation and depth completion/upsampling methods.Source: §4.2 · pp. 5, 7, Table 2 · p. 7

4.3. Ablations and Analysis

  • Removing prompting or foundation-model initialization reduces performance, and ControlNet is the strongest tested alternative but remains below the proposed design.The tested alternatives also include AdaLN and cross-attention.Source: §4.3 · p. 7
  • Adding ARKitScenes training data helps only on that dataset, while adding ScanNet++ improves both; direct pseudo-GT supervision decreases performance.The edge-aware loss using pseudo-GT and FARO-annotated GT gives comparable results to the two-dataset setup but better thin-structure depth.Source: §4.3 · p. 7
  • At 2160 × 3840 resolution, ViT-L runs at 2.0 FPS and ViT-S at 10.3 FPS on an A100 GPU.At 768 × 1024, the corresponding speeds are 20.4 FPS and 80.0 FPS.Source: §4.3 · p. 7

4.4. Zero-shot Testing on Diverse Scenes

  • Despite indoor-scene training, the model generalizes to new rooms, gyms, poorly lit museums, human environments, and outdoor scenes.The diverse-scene results are presented as zero-shot testing.Source: §4.4 · p. 7

4.5. Application: 3D Reconstruction

  • Scale-accurate depth supports indoor 3D reconstruction, and replacing the prompt with vehicle LiDAR enables large-scale outdoor reconstruction.The outdoor reconstruction uses the vehicle LiDAR as the metric prompt.Source: §4.5 · pp. 7–8, Figure 6 · p. 7

4.6. Application: Generalized Robotic Grasping

  • In grasping, model depth outperforms RGB and LiDAR inputs on unseen objects, including transparent and specular objects, when training uses diffusive objects.The authors attribute RGB's weaker generalization to color-dominated input and note that iPhone LiDAR is noisy and cannot perceive transparent objects.Source: §4.6 · p. 8, Table 4 · p. 8

5. Conclusion and Discussions

The paper frames metric depth estimation as prompting a depth foundation model with metric information and reports benefits for 3D reconstruction and generalized robotic grasping. It also reports generalization to ARKit6 images and depth at resolutions differing from training.

5. Conclusion and Discussions

  • The proposed metric-prompting paradigm outperforms existing monocular depth estimation and depth completion/upsampling methods and benefits 3D reconstruction and generalized robotic grasping.Source: §5 · p. 8
  • With iPhone LiDAR prompts, the method cannot handle long-range depth, and noisy far-object measurements can produce flickering depth predictions.The authors suggest more advanced prompt learning and temporal prompt tuning as future directions.Source: §5 · p. 8

A.1. Generalizability to Different Resolutions

  • Although trained on ScanNet++ and ARKitScenes images at 1440 × 1920, the model generalizes to ARKit6 images and depth at different resolutions.ARKit6 images reach 3024×4032, while the example shown uses 2160×3840 images and 144×256 LiDAR depth.Source: §A.1 · p. 12

A.2. Why Do We Need Synthetic Data?

Synthetic data provides high-quality ground-truth depth for training, while its availability can add diversity beyond real data that is currently concentrated in indoor scenes. Experiments also found that including synthetic human data improves generalization to human subjects.

A.2. Why Do We Need Synthetic Data?

  • Synthetic data provides high-quality ground-truth depth, which has been crucial to recent depth-estimation work and is also used for high-quality results here.Source: §A.2 · p. 12
  • Real data with low-cost and high-power LiDAR is currently limited primarily to indoor scenes, while synthetic data can further enhance diversity.Source: §A.2 · p. 12
  • Including human synthetic data improved the method’s generalization to human subjects in the authors’ experiments.Source: §A.2 · p. 12

A.3. Why Do We Need Real Data?

The authors report that adding real data further enhances performance because synthetic LiDAR simulation cannot fully replicate real LiDAR noise. Additional qualitative results examine the effects of using real data.

A.3. Why Do We Need Real Data?

  • Training with real data further enhances performance beyond preliminary results achieved using synthetic data.Source: §A.3 · p. 12
  • Synthetic LiDAR simulation cannot fully replicate real LiDAR noise patterns, which the authors identify as a reason real data is beneficial.Source: §A.3 · p. 12
  • Figure 11 provides additional qualitative results about the effects of using real data.Source: Figure 11 · p. 13

A.4. Replacing Depth Foundation Models

The DPT-based design can be adapted to other depth foundation models such as Depth Pro. The authors report improved Depth Pro performance, though it does not surpass their Depth Anything choice.

A.4. Replacing Depth Foundation Models

  • Adapting the design to Depth Pro significantly enhances its performance, but does not outperform the Depth Anything choice.Source: §A.4 · p. 13
  • The authors describe the method as a general design for DPT that can be adapted to other DPT-structured depth foundation models.Source: §A.4 · p. 13

B. Additional Results

The additional results examine simulated LiDAR, ScanNet++ reconstruction and depth annotations, alongside model implementation details.

B. Additional Results

  • The additional results include visualizations of simulated LiDAR outputs in Fig. 12.Source: B. Additional Results · p. 13
  • Resampling ScanNet++ frames to remove blurred frames leads to better ZipNeRF reconstruction; using DSLR and iPhone data substantially improved results, and training on ScanNet++ data required approximately 280 × 2.5 × 8 GPU hours.Source: B. Additional Results · p. 14, Figure 13 · p. 14
  • ScanNet++ FARO-mesh GT depth has holes and poor edges, while NeRF pseudo GT has accurate edges but performs poorly in planar regions.The paper proposes an edge-aware loss to merge their complementary properties.Source: B. Additional Results · p. 14, Figure 14 · p. 14

C.1. Details about Our Model

  • The model uses Depth Anything v2 ViT-large, with two 3×3 stride-1 convolutional layers and a zero-initialized 1×1 projection layer.Training uses the main-paper loss on ScanNet++, L1 on ARKitScenes, and gradient plus pixel-wise L1 losses on synthetic data.Source: §C.1 · p. 14

C.2. Optional Design Details

The authors compare optional fusion designs with their proposed architecture and test whether combining ControlNet adds gains.

C.2. Optional Design Details

  • ControlNet performs best among the alternatives but remains less effective than the proposed design; combining them raises computational cost without improving ScanNet++ F-score (0.730 vs. 0.731).The authors suggest alternatives designed for cross-modal prompts may not exploit pixel alignment between low-resolution input depth and output depth.Source: §C.2 · p. 14

C.3. Evaluation Metrics

These sections define evaluation metrics, describe baseline alignment, and report vehicle-LiDAR and robotic-grasping evaluation procedures.

C.3. Evaluation Metrics

C.4. Baseline Details

  • For baseline comparisons, Metric3D v2 and Depth Pro receive ground-truth focal length, while monocular outputs are aligned to iPhone LiDAR depth.ZoeDepth* uses reproduced Depth Anything v1 code; MSPF results use the stated dataset-specific procedures.Source: §C.4 · p. 15, §C.5 · p. 15

C.5. Ransac Alignment Details

  • RANSAC alignment typically improves ScanNet++ F-score by 8–10% over commonly used polyfit alignment.It estimates scale and shift from sample groups, then votes using a threshold set to the median absolute deviation.Source: §C.5 · p. 15

D. Prompting with a vehicle LiDAR

E. Generalized Robotic Grasping Details

  • The robotic grasping study trains ACT policies on diffusive-object trajectories and evaluates visual inputs from the proposed depth, ARKit depth, and RGB.The setup tests objects at nearby, middle, and far positions; the policy crops inputs to 480×640 and uses a pretrained ResNet18 backbone.Source: E. Generalized Robotic Grasping Details · p. 16
Loading 2412.14015v4…