PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun

arXiv:2410.02073v2cs.CVcs.LG

TL;DR

Zero-shot monocular depth estimation must provide metric scale, sharp high-resolution boundaries, and low latency for broad interactive use. Depth Pro addresses these requirements with a foundation model that produces 2.25-megapixel depth maps in 0.3 seconds and outperforms prior work across multiple dimensions.

  • Problem

    Zero-shot monocular depth estimation lacks a broadly applicable combination of metric scale, fine boundary detail, high resolution, and sub-second inference.

  • Method

    Depth Pro combines an efficient multi-scale vision transformer with real-and-synthetic training and dedicated metrics for sharp depth-boundary evaluation.

  • Results

    2.25-megapixel depth maps are produced in 0.3 seconds, while Depth Pro outperforms prior work in metric accuracy, boundary sharpness, and runtime.

  • Takeaways & Limitations

    Depth Pro supports novel view synthesis from single images by providing sharp, metric depth maps at sub-second runtimes without camera-intrinsic metadata.

  • Takeaways & Limitations

    Comparisons with leading systems cannot control for differences in training data because some use proprietary, restricted, unlabeled, or externally pretrained resources.

Abstract

from arXiv · show

We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code and weights at https://github.com/apple/ml-depth-pro

1 INTRODUCTION

Depth Pro is a foundation model for zero-shot monocular depth estimation designed for arbitrary images, high-resolution fine-detail tracing, and interactive low-latency use. It produces metric depth with absolute scale without camera intrinsics, while achieving sharp boundaries and substantially faster inference than prior fine-grained methods.

  • Motivation: Novel view synthesis motivates depth estimators that work zero-shot on any image, operate at high resolution, preserve fine structures, and run in less than a second.Sharp depth maps tracing hair, fur, and other details can eliminate flying pixels that degrade view-synthesis quality.
  • Depth Pro: Depth Pro produces zero-shot metric depth maps with absolute scale at 2.25-megapixel native resolution in 0.3 seconds on a V100 GPU.The model operates on arbitrary images without requiring metadata such as camera intrinsics; its native output is 1536 × 1536 before optional upsampling.
  • Depth Pro: Depth Pro offers unparalleled boundary tracing, outperforming prior work by a multiplicative factor in boundary recall while remaining one to two orders of magnitude faster than prior state of the art.It also provides metric depth with absolute scale and more accurate boundaries than Ke et al., 2024 and Li et al., 2024a.
  • Technical contributions: Depth Pro introduces boundary-tracing metrics that leverage highly accurate matting datasets to quantify monocular depth-map boundary accuracy.The authors find that high output resolution is necessary but not sufficient to improve boundary accuracy.

2 RELATED WORK

Prior monocular depth work evolved from dataset-specific metric prediction toward zero-shot generalization, metric recovery, and sharper boundaries. Depth Pro combines multi-scale ViT fusion with metric, high-resolution prediction and introduces complementary boundary-tracing metrics for broader scene evaluation.

  • Zero-shot depth estimation: Metric monocular depth progressed from single-camera, single-dataset training to zero-shot approaches using diverse datasets and scale-and-shift-invariant objectives.Early methods enabled metric predictions but remained limited to narrow domains; MegaDepth and MiDaS broadened generalization, followed by transformer and self-supervised extensions.
  • Zero-shot metric depth: Zero-shot metric methods recover scale through global depth distributions, scene-type conditioning, or explicit camera-intrinsic modeling.Examples include distribution-based methods (Fu et al., 2018; Bhat et al., 2021; 2022; Li et al., 2024b), scene conditioning (Bhat et al., 2023), Cam-Convs (F´acil et al., 2019), and LeReS (Yin et al., 2021).
  • Sharp occluding contours: Depth Pro proposes a simpler architecture without task-specific modules or diffusion priors that produces sharper, more accurate metric depth while reducing runtime by more than two orders of magnitude.These alternatives predominantly predict relative depth, whereas the proposed approach targets metric depth.
  • Boundary evaluation: Depth Pro’s boundary evaluation complements iBims by using segmentation and matting datasets that cover complex dynamic environments and extremely fine detail.iBims relies on manual occluding-contour annotations and highly accurate ground truth, restricting it largely to a small set of indoor scenes.
  • Multi-scale vision transformers: Multi-scale ViT fusion combines global context and local detail, unlike methods that augment pretrained ViTs with convolutional networks or feature pyramids.The architecture applies a plain ViT backbone at multiple scales and fuses predictions into one high-resolution output, while allowing newer pretrained variants to be substituted easily.

3 METHOD

Depth Pro combines shared-weight multi-scale ViT patch encoders with a global image encoder and DPT decoder to produce sharp, high-resolution dense depth predictions. Its method further uses canonical inverse-depth objectives, a two-stage real/synthetic training curriculum, and boundary-focused evaluation metrics.

  • Network architecture: A fixed 1536×1536 operating resolution provides a large receptive field, constant runtimes, and protection against out-of-memory failures on large images.The resolution is selected as a multiple of the ViT’s 384×384 input size.
  • Network architecture: Shared-weight ViT encoders process patches at multiple scales, while a global image encoder anchors predictions before DPT fusion produces high-resolution dense depth.The architecture merges patch features into maps, upsamples them, and fuses them through a DPT decoder.
  • Training objectives: The network predicts canonical inverse depth and converts it to metric depth using the image’s focal length and width, while training combines metric and normalized non-metric losses with multi-scale derivative errors.The derivative objectives include gradient and Laplace errors at multiple scales.
  • Training curriculum: A two-stage curriculum first learns robust cross-domain features from mixed real and synthetic data, then sharpens boundaries using only synthetic data with pixel-accurate ground truth.The second stage reverses the common practice of synthetic pretraining followed by real-data fine-tuning.
  • Evaluation metrics for sharp boundaries: The method introduces scale-invariant boundary metrics based on neighboring-pixel depth ratios, reporting weighted F1 scores across thresholds from 5% to 25%.The metrics can also derive occluding contours from binary segmentation, saliency, and matting masks without manual edge annotations.

4 EXPERIMENTS

Experiments show that Depth Pro generalizes strongly for zero-shot metric depth, produces sharper boundaries than competing methods, and achieves the most accurate focal-length estimation across evaluated datasets.

  • Zero-shot metric depth: Metric-depth comparisons use δ1, the percentage of pixels whose predicted and ground-truth depths differ by no more than 25%, alongside average rank across datasets.Additional metrics, including AbsRel, Log10, δ2, δ3, and point-cloud metrics, are reported in the appendices.
  • Zero-shot metric depth: Depth Pro achieves the best average rank across datasets and consistently scores among the top approaches for zero-shot metric depth.Table 1 reports δ1 per dataset, with higher scores better and lower average rank better; evaluations use datasets not used to train the evaluated systems.
  • Zero-shot boundaries: Depth Pro produces significantly more accurate boundaries than all baselines on every evaluated dataset, including sharper thin structures such as hair and fur.The zero-shot boundary evaluation includes synthetic, low-resolution, matting, and high-resolution natural-image datasets; boundary metrics are scale-invariant.
  • Focal length estimation: Depth Pro is the most accurate focal-length estimator across all datasets, reaching 64.6% below 25% error on PPR10K versus 34.6% for SPEC.The evaluation reports the percentages of images with relative focal-length errors below 25% and 50% using datasets with intact EXIF data.

5 CONCLUSION & LIMITATIONS

Depth Pro delivers high-resolution, high-frequency metric depth at sub-second runtimes, achieving state-of-the-art zero-shot accuracy without camera-intrinsics metadata while tracing occlusion boundaries in unprecedented detail, though limitations remain.

  • Depth Pro produces high-resolution metric depth maps with high-frequency detail at sub-second runtimes.
  • Depth Pro achieves state-of-the-art zero-shot metric depth estimation accuracy without requiring metadata such as camera intrinsics.
  • Depth Pro traces occlusion boundaries in unprecedented detail, facilitating novel view synthesis from single images in the wild.
  • Despite outperforming prior work along multiple dimensions, Depth Pro has limitations.

SUPPLEMENTAL MATERIAL · A ADDITIONAL RESULTS · A.1 QUALITATIVE RESULTS

The supplemental material adds further results and experiments, including qualitative comparisons of Depth Pro with recent baselines on in-the-wild images. Depth Pro preserves fine details that Metric3D v2 and Depth Anything v2 often miss, while Marigold produces finer details but commonly introduces noise.

  • A ADDITIONAL RESULTS: Section A provides additional results and experiments beyond the main paper.
  • A ADDITIONAL RESULTS: Section A.1 presents further qualitative comparisons to baselines.
  • SUPPLEMENTAL MATERIAL: The supplemental material also includes detailed zero-shot evaluation, runtime measurements, boundary-accuracy experiments, and controlled studies of architectural and training choices.These materials are organized across Sections A.2–A.4 and B.2, B.3, B.5, and B.7.
  • A.1 QUALITATIVE RESULTS: Depth Pro is qualitatively compared with Marigold, Metric3D v2, and Depth Anything v2 on in-the-wild images.The comparisons use AM-2k, DIS-5k, and Unsplash images across Figs. 5–7.
  • A.1 QUALITATIVE RESULTS: Metric3D v2 and Depth Anything v2 repeatedly miss fine details in the evaluated images.
  • A.1 QUALITATIVE RESULTS: Marigold reproduces finer details than Metric3D v2 and Depth Anything v2 but commonly yields noisy predictions.

A.2 ZERO-SHOT METRIC DEPTH · A.3 RUNTIME

Depth Pro is evaluated zero-shot across six diverse benchmarks with standard depth and point-cloud metrics, alongside qualitative comparisons against leading methods. Its runtime evaluation shows high native output resolution with substantially lower computational cost and latency than comparable approaches.

  • A.2 ZERO-SHOT METRIC DEPTH: Depth Pro’s zero-shot metric-depth evaluation spans Booster, Middlebury, Sun-RGBD, ETH3D, nuScenes, and Sintel against Depth Anything, Metric3D, and other baselines.The evaluation reports AbsRel, Log10, δ2, δ3, SI-Log, and point-cloud metrics.
  • A.2 ZERO-SHOT METRIC DEPTH: Qualitative zero-shot comparisons visualize Depth Pro alongside Marigold, Metric3D v2, and Depth Anything v2 on Unsplash, AM-2k, and DIS-5k images.The figures provide visual comparisons across varied image sources but do not supply numerical values in the passages.
  • A.3 RUNTIME: Runtime measurements cover VGA, HD, and 4K inputs, including preprocessing, resizing, and inference under a reproduced evaluation environment.Results are reported for multiple image sizes and include parameter counts and FLOPs.
  • A.3 RUNTIME: Depth Pro processes more than 3 times as many pixels as Metric3D v2 while using less than half its parameters and one-third its runtime.This comparison applies among approaches with fixed output resolution.
  • A.3 RUNTIME: Depth Pro has the highest native output resolution among the evaluated fixed-output-resolution approaches.The result is reported alongside its parameter and runtime advantages over Metric3D v2.
  • A.3 RUNTIME: ZeroDepth takes almost 4 times as long as Depth Pro even on small VGA images, while variable-resolution approaches have considerably larger runtimes.The comparison concerns PatchFusion and ZeroDepth, with ZeroDepth identified as the faster variable-resolution model.

A.4 BOUNDARY EXPERIMENTS … B.2 NETWORK BACKBONE

Depth Pro’s boundary metrics are validated qualitatively and empirically, while controlled experiments show that native output resolution strongly affects boundary accuracy and motivate systematic architectural comparisons. On iBims, Depth Pro matches state-of-the-art DDE and PE performance while outperforming prior work on boundary metrics.

  • A.4 BOUNDARY EXPERIMENTS: Depth Pro is on par with the state of the art on iBims DDE and PE, while significantly outperforming all prior work on boundary metrics.The iBims benchmark evaluates metric depth, depth boundaries, and planar accuracy using laser-scanned indoor scenes with annotated occlusion boundaries.
  • A.4 BOUNDARY EXPERIMENTS: The proposed boundary metrics compare ground-truth and predicted occluding contours to expose how incorrect depth boundaries affect metric measurements.Figure 8 visualizes sample images, boundary sources, predictions, and derived occluding contours across ground-truth depth and binary maps from matting or segmentation datasets.
  • A.4 BOUNDARY EXPERIMENTS: Boundary precision and recall are empirically tested under image shifts, downsampling followed by upsampling, and Gaussian blurring, linking perturbed contours to boundary F1 scores.The study reports both quantitative and qualitative results for ground-truth perturbations and their resulting occluding contours.
  • B CONTROLLED EXPERIMENTS: Controlled experiments examine native resolution, network architecture, depth representation, training objectives, curriculum, and focal-length estimation.These experiments are designed to isolate the contribution of major Depth Pro components and design decisions.
  • B.1 NATIVE OUTPUT RESOLUTION: Native output resolution is evaluated using Log10, AbsRel, and F1 metrics after bilinear downsampling and upsampling across several candidate resolutions.The experiment compares metric-depth prediction and boundary accuracy at the original input resolution.
  • B.1 NATIVE OUTPUT RESOLUTION: 3-fold increase per doubling of native output resolution shows that high-resolution prediction strongly improves boundary accuracy, making native resolution necessary but not necessarily sufficient.The UnrealStereo4K experiment downsamples ground-truth depth to common literature resolutions, upsamples it to the input resolution, and evaluates depth and boundary metrics.
  • B.2 NETWORK BACKBONE: Candidate image-encoder backbones are compared using pretrained TIMM models trained at 384 × 384 across five RGBD datasets.The datasets are Keystone, HRWSI, RedWeb, TartanAir, and Hypersim; evaluation measures metric-depth performance.

B.3 HIGH-RESOLUTION ALTERNATIVES · B.4 DEPTH REPRESENTATION

Depth Pro’s high-resolution architecture outperforms alternative backbones and scaling strategies on boundary quality while maintaining strong metric depth accuracy and competitive latency. Among tested depth representations, inverse-depth supervision performs best, especially for near-camera predictions.

  • B.3 HIGH-RESOLUTION ALTERNATIVES: The high-resolution alternatives study evaluates 1536×1536 network structures and pretrained weights under identical Stage 1 training, with latency measured on a single V100 GPU.All ViT models use 16×16 patches, while Depth Pro uses ViT-L DINOv2 for both image and patch encoders.
  • B.3 HIGH-RESOLUTION ALTERNATIVES: The image-encoder comparison identifies ViT-L DINOv2 as the optimal backbone among candidates trained at 384×384 resolution across five RGB-D datasets.Candidates were selected with comparable computational complexity and parameter counts.
  • B.3 HIGH-RESOLUTION ALTERNATIVES: Depth Pro is faster and more accurate at object boundaries than a plain ViT, while achieving comparable metric depth accuracy.The comparison evaluates high-resolution alternatives under the Stage 1 training protocol.
  • B.3 HIGH-RESOLUTION ALTERNATIVES: Compared with transformer and convolutional alternatives, Depth Pro has comparable latency, lower metric depth error, and higher object-boundary recall.The reported differences are described as several times lower metric depth error and several times higher boundary-recall accuracy.
  • B.3 HIGH-RESOLUTION ALTERNATIVES: On DIS5K, Depth Pro improves boundary recall by 23% relative to a straightforwardly scaled ViT with DINOv2 pretraining.This comparison isolates the benefit of the proposed architecture over simply scaling the ViT architecture.
  • B.4 DEPTH REPRESENTATION: Inverse-depth supervision yields the most accurate depth predictions overall, outperforming direct depth supervision and log-depth under mean absolute error training.The experiment uses a ViT encoder and DPT decoder trained on Hypersim, evaluating delta1 error across depth ranges.
  • B.4 DEPTH REPRESENTATION: Optimizing inverse depth is especially accurate near the camera, a property highlighted as important for novel-view synthesis.Table 10 reports delta1 error across several depth ranges.

B.5 TRAINING OBJECTIVES · B.6 FULL CURRICULA

The study evaluates alternative objectives for each training stage and compares complete curricula. Scale-and-shift-invariant gradient supervision on synthetic data is best for stage 1, while the full-curriculum comparison finds single-stage training a reasonable default.

  • B.5 TRAINING OBJECTIVES: The curriculum evaluation first examines individual training stages and then compares complete training schedules.The full-curriculum alternatives include single-stage training and reversing the proposed stage order.
  • B.5 TRAINING OBJECTIVES: Stage 1 ablations compare mean absolute error and gradient-loss assignments across metric, non-metric, and synthetic datasets.Condition 1A uses LMAE for metric data and scale-and-shift-invariant LSSI-MAE for non-metric data; 1B adds gradients only on synthetic data, while 1D applies gradients everywhere.
  • B.5 TRAINING OBJECTIVES: Stage 1 performs best with a scale-and-shift-invariant gradient loss on synthetic datasets, improving convergence and overall performance.This setting is condition 1C; the alternatives vary whether metric and non-metric losses or gradient losses are applied across datasets.
  • B.5 TRAINING OBJECTIVES: Using both first- and second-order derivative losses in stage 2 yields the best balance across metrics by average rank.This is condition 2A in the stage-2 objective comparison.
  • B.5 TRAINING OBJECTIVES: Stage 2 targets sharper depth boundaries while retaining metric accuracy and uses only synthetic datasets because their ground truth is higher quality.The evaluated sharpening strategy combines multi-scale derivative-based losses, including LMAE, LMSE, LMAGE, LMALE, and LMSGE.
  • B.6 FULL CURRICULA: Single-stage training is a reasonable default compared with the proposed two-stage curriculum and reversed synthetic-pretraining curriculum.Condition 3A is the two-stage curriculum, 3B applies all second-stage gradient losses throughout training, and 3C pretrains synthetically before real-data fine-tuning.

B.7 FOCAL LENGTH ESTIMATION … C.2 FOCAL LENGTH HEAD

Depth Pro achieves the strongest zero-shot focal-length estimation across all evaluated datasets, with gains from task-specific image encoding alongside frozen depth features. The merged implementation sections also specify Voronoi-based patch merging and evaluation details.

  • B.7 FOCAL LENGTH ESTIMATION: Depth Pro outperforms all compared approaches on every dataset in the zero-shot focal-length evaluation.Figure 10 reports the percentage of samples below each AbsRel error threshold for each method and dataset.
  • B.7 FOCAL LENGTH ESTIMATION: The controlled focal-length-head experiment uses 500 images randomly sampled from Flickr to compare alternative network structures.The reported variants include a small convolutional head on frozen depth features and separate or depth-derived ViT encoders.
  • C.2 FOCAL LENGTH HEAD: The selected focal-length architecture combines frozen depth-network features with task-specific features from a separate ViT encoder in parallel.The controlled comparison evaluates frozen depth features with a convolutional head, a separate ViT encoder, and a ViT encoder trained on extracted depth features.
  • B.7 FOCAL LENGTH ESTIMATION: Training a separate image encoder improves focal-length performance by 14.6 percentage points, showing that task-specific knowledge complements depth features.Refining depth features performs on par with using frozen depth features alone, despite the added computation.
  • C IMPLEMENTATION, TRAINING AND EVALUATION DETAILS: The implementation and evaluation details section covers training and evaluation datasets, hyperparameters, and the evaluation setup.These details provide the procedural scope for the method’s experiments.
  • C.1 MERGE OPERATION: Overlapping feature patches are merged into feature maps by assigning each patch center a Voronoi cell and copying the cell-covered patch area.The remaining feature-map area is discarded, while overlap gives the patch encoder receptive-field coverage of neighboring patches.

C.3 DATASETS … C.6 EVALUATION SETUP

The paper documents its datasets, training configuration, baseline implementations, and evaluation protocol. It emphasizes controlled comparison because depth ranges, resolutions, resizing, intrinsics, and other setup choices can materially affect results.

  • C.3 DATASETS: The datasets, training hyperparameters, and stage-specific loss functions are specified in Tables 15–17.These tables summarize the datasets used, training settings, and losses applied across datasets and training stages.
  • C.5 BASELINES: Depth Anything metric models are matched to dataset domains, while relative models are used for qualitative and zero-shot boundary evaluations.For mixed indoor–outdoor datasets, the best-performing domain model is selected; relative models provide sharper qualitative boundaries.
  • C.6 EVALUATION SETUP: Table 18 defines each evaluation dataset’s valid-depth range, sample count, and ground-truth resolution, with NuScenes evaluated on a random subset of its approximately 35K validation samples.Predictions at different resolutions are resized bilinearly to the ground-truth resolution before evaluation.
  • C.5 BASELINES: Metric3D evaluation uses separate crop sizes for indoor (512, 1088) and outdoor (512, 992) datasets because fixed cropping can severely harm metric-scale accuracy.These results are marked as not strictly zero-shot, and Metric3D v2 uses its largest, “giant” model.
  • C.5 BASELINES: UniDepth uses the ViT-L variant, while ZoeDepth uses the model fine-tuned on both indoor and outdoor data, ZoeD NK.The ViT-L UniDepth model performs best on average among its variants.
  • C.6 EVALUATION SETUP: The evaluation controls depth ranges, metric resolution, resizing, and intrinsics in a common environment to prioritize fair baseline comparison.The authors note that baseline reports often omit setup details, making exact reproduction and guaranteed fairness difficult; most reported results were matched, with three notable differences.
  • C.6 EVALUATION SETUP: Boundary metrics average F1 or recall across thresholds from tmin = 5 to tmax = 25 using normalized range-based weights that emphasize higher thresholds.The same weighted-averaging strategy is applied to depth-based and mask-based boundary metrics.

D APPLICATIONS

Depth Pro’s metric, sharp, and fast monocular depth estimation supports downstream applications in depth-conditioned image synthesis and synthetic depth of field. These applications use predicted depth to preserve scene structure during ControlNet stylization and to delineate objects for subject-preserving blur.

  • Depth-conditioned image synthesis: Depth Pro enables depth-conditioned image synthesis by using its predicted depth map to condition a pretrained depth-to-image ControlNet SD 1.5 model while stylizing an input through a text prompt.Figure 12 compares Depth Pro with Depth Anything v2, Marigold, and Metric3D v2.
  • Synthetic depth of field: Synthetic depth of field uses a single image and depth map with BokehMe’s hybrid neural and physically motivated renderer, making object delineation essential for keeping the subject correctly in focus.Figure 13 compares BokehMe results using depth maps from Depth Pro, Marigold, and Depth Anything v2.
  • Depth-conditioned image synthesis: Missing cables in Depth Anything v2 and Metric3D v2, or a spurious sky gradient in Marigold, alter the synthesized scene structure compared with Depth Pro’s result.The comparison is shown in Figure 12.
Loading 2410.02073v2…