PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Anything V2

Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao

arXiv:2406.09414v2cs.CV

TL;DR

Monocular depth estimation needs models that remain robust and fine-grained across complex scenes while supporting varied applications. Depth Anything V2 addresses this through data redesign and pseudo-labeled real images, producing a more capable foundation model and a diverse, precisely annotated evaluation benchmark.

  • Problem

    Monocular depth estimation lacks a foundation model that jointly provides robust complex-scene predictions, fine details, varied scales, and downstream transferability.

    Source: §1 · p. 2

  • Method

    Depth Anything V2 replaces labeled real images with precise synthetic data, scales its teacher, trains students with pseudo-labeled real images, and constructs the DA-2K benchmark.

    Source: §1 · p. 3, §6.2 · p. 8, §7.4 · p. 9

  • Results

    Depth Anything V2 provides robust, fine-grained predictions, supports 25M–1.3B-parameter models, and serves as an initialization for downstream fine-tuning.

    Source: §9 · p. 10

  • Takeaways & Limitations

    The work shows that efficient discriminative models can produce extremely fine details without heavy diffusion-based modeling and offers a versatile benchmark for evaluation.

    Source: §1 · p. 3

    Existing depth benchmarks contain frequent label noise and limited scene diversity, making reported metrics unreliable and potentially unrepresentative of real-world reliability.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.

1 Introduction

Depth Anything V2 seeks a more capable monocular depth foundation model by preserving robust scene predictions while adding fine details, efficiency, varied scales, and transferability. Its central findings identify data design and scaling—not sophisticated techniques—as the key to achieving these goals, alongside a new precise and diverse evaluation benchmark.

Motivation

  • The proposed foundation model targets robust predictions for complex scenes, fine depth details, varied efficient model scales, and strong transferability to downstream tasks.These goals include handling transparent and reflective surfaces, thin objects and small holes, and supporting fine-tuning for downstream applications.Source: §1 · p. 2

Section 2

  • Efficient discriminative models can produce extremely fine details when all labeled real images are replaced with precise synthetic images.This finding challenges the assumption that fine details require heavy diffusion-based modeling.Source: §1 · p. 3

Section 3

  • Synthetic images are clearly superior to real images for labeled training, but their drawbacks are substantial and were not trivial to address in previous paradigms.The introduction identifies these drawbacks as the central limitation motivating the next stage of the method.Source: §1 · p. 3

Section 4

  • Scaling the teacher trained solely on synthetic images and teaching smaller students through large-scale pseudo-labeled real images avoids synthetic-data drawbacks while amplifying its advantages.The pseudo-labeled real images serve as a bridge between the high-capacity teacher and student models.Source: §1 · p. 3

Section 6

  • The work constructs a versatile evaluation benchmark with precise annotations and diverse scenes because current test sets [70] are too noisy to reflect models’ true strengths.The benchmark is introduced in Section 6 to support more reliable evaluation.Source: §1 · p. 3

2 Revisiting the Labeled Data Design of Depth Anything V1

Depth Anything V1’s large real-image training set is undermined by noisy and coarse depth labels, especially for transparent objects, reflective surfaces, and fine structures. The section motivates replacing real labeled data with precise synthetic images that preserve fine details, capture challenging surfaces, and scale efficiently.

Drawbacks of real labeled data

  • Real labeled depth maps contain inaccuracies from sensor capture, stereo matching, and SfM, particularly around transparent objects, textureless or repetitive patterns, dynamic objects, and outliers.These limitations make real labels unreliable supervision for monocular depth estimation.Source: §2 · p. 3
  • Real datasets omit fine boundaries, thin holes, and small structures, producing over-smoothed predictions and similar errors in trained models.The coarse depth representation of objects such as trees and chairs illustrates this loss of detail.Source: §2 · p. 4, Figure 4 · p. 4
  • 25.9% and 53.5% are the Transparent Surface Challenge scores of MiDaS and Depth Anything V1, respectively, whereas Depth Anything V2 reaches 83.6% zero-shot.These results demonstrate the practical impact of unreliable real labels and the later model’s improved handling of transparent surfaces.Source: §2 · p. 4

Advantages of synthetic images

  • Synthetic images provide precise ground-truth depth for fine details, transparent objects, and reflective surfaces, while enabling rapid scaling from graphics engines without privacy or ethical concerns.Synthetic labels capture boundaries, thin holes, small objects, thin mesh structures, leaves, and challenging surfaces such as a transparent vase.Source: §2 · p. 4, Figure 4 · p. 4

3 Challenges in Using Synthetic Data

Synthetic data face two central obstacles for monocular depth estimation: distribution shift from real images and restricted scene coverage. Pilot results further show that synthetic-only training transfers poorly except with a large encoder, which still fails on unseen patterns and is inefficient.

Limitation 1: Distribution shift

  • Synthetic images differ from real images in style, color distributions, cleanliness, and layout randomness, hindering synthetic-to-real transfer even when layouts are similar [57] [9].Synthetic images appear too clean and ordered, whereas real images contain more randomness.Source: §3 · p. 4

Limitation 2: Restricted scene coverage

  • Fixed scene types restrict synthetic coverage, limiting generalization to real-world settings such as crowded people despite precise datasets like Hypersim [58] and Virtual KITTI [9].Web stereo and monocular-video datasets such as HRWSI and MegaDepth [37] cover more extensive real-world scenes.Source: §3 · p. 4

Pilot study

  • Only DINOv2-G achieves satisfying synthetic-to-real transfer in the pilot study, while other encoders and smaller DINOv2 models suffer severe generalization issues.The comparison evaluates BEiT [4], SAM, SynCLR [75], and DINOv2 [50] encoders trained solely on synthetic images.Source: Figure 5 · p. 5, §3 · p. 5
  • DINOv2-G still predicts incorrect depths for rarely represented patterns such as cloudy skies and human heads, while its 1.3B parameters make storage and inference costly.These failures arise because synthetic training lacks diverse sky patterns and humans; the smallest Depth Anything V1 model is more widely used for real-time speed.Source: Figure 6 · p. 5, §3 · p. 5

Why existing remedies fall short

  • Combining real and synthetic data can make coarse real-image depth destructive to fine-grained prediction, while expanding synthetic scenes is unsustainable because graphics engines cannot mimic every scenario [7] [28].The paper therefore motivates a reliable solution that resolves the precision–robustness dilemma across model scales without trade-offs.Source: §3 · p. 5

4 Key Role of Large-Scale Unlabeled Real Images

Large-scale unlabeled real images are indispensable for Depth Anything V2: the strongest synthetic-trained teacher pseudo-labels real images, and students train solely on those labels. This bridges the synthetic-to-real gap, expands scene coverage, and transfers the teacher’s knowledge to smaller models through label-level distillation.

Overall pipeline

  • The training pipeline first fits a DINOv2-G-based teacher on precise synthetic images, then pseudo-labels unlabeled real images, and finally trains students on those high-quality labels.This procedure uses the most capable MDE model as the teacher and trains new models solely with large-scale, precisely pseudo-labeled real images.Source: §4 · p. 5, Figure 7 · p. 6

Bridge the domain gap

  • Large-scale unlabeled real images bridge the synthetic-to-real domain gap by providing an intermediate learning target with fine-grained, complete pseudo labels.The teacher generates pseudo depth labels for unlabeled real images, making models more familiar with real-world distributions than direct synthetic-to-real transfer; Figure 17 visualizes the label quality.Source: §4 · p. 5

Enhance the scene coverage

  • Unlabeled real images expand scene coverage beyond limited and redundant synthetic data, strengthening zero-shot MDE and pretraining for downstream tasks [72].Public real-image datasets provide numerous distinct, informative scenes that synthetic data lacks.Source: §4 · p. 6

Transfer knowledge from the most capable model to smaller ones

  • Large-scale pseudo-labeled real images transfer the strongest model’s predictions to smaller models through safer label-level distillation rather than feature- or logit-level distillation [27] [48].Smaller models cannot directly exploit synthetic-to-real transfer, but can mimic the capable teacher on extra unlabeled real images; this substantially improves their robustness.Source: §4 · p. 6

5 Depth Anything V2

Depth Anything V2 trains a reliable synthetic-image teacher, transfers its predictions through large-scale pseudo-labeled real images, and releases four student scales for robust generalization. Its training details address pseudo-label noise and depth sharpness, while DA-2K provides a broader, more precise evaluation benchmark than noisy existing tests.

5.1 Overall Framework

  • Depth Anything V2 uses a three-step pipeline: train a DINOv2-G teacher on high-quality synthetic images, pseudo-label unlabeled real images, and train students on those labels.The student stage does not require synthetic images, supporting robust generalization from pseudo-labeled real data.Source: §5.1 · p. 6
  • Four student models based on DINOv2 small, base, large, and giant are released.The released models span four backbone scales.Source: §5.1 · p. 6

5.2 Details

  • Training uses 595K precise synthetic images and 62M pseudo-labeled real images, ignores the top 10% largest-loss regions, and adds feature alignment on pseudo-labeled data.The method combines scale- and shift-invariant and gradient-matching losses, while following V1 for noisy-region filtering and feature alignment.Source: §5.2 · p. 6
  • DA-2K evaluates relative depth with annotated point pairs sampled from SAM [33] masks and expands scenario coverage, addressing noise in widely used benchmarks [70].Figure 8 shows incorrect depths for mirrors and thin structures in an adopted test benchmark, whereas Figure 9 presents the DA-2K annotation pipeline and coverage.Source: Figure 9 · p. 7, Figure 8 · p. 7

6 A New Evaluation Benchmark: DA-2K

Existing depth benchmarks suffer from noisy annotations, narrow scene coverage, and low resolution, limiting reliable evaluation of monocular depth estimation. DA-2K addresses these issues with precise sparse depth pairs across diverse, mostly high-resolution scenes and supplements rather than replaces existing benchmarks.

6.1 Limitations in Existing Benchmarks

  • Existing benchmarks contain noisy depth annotations, including errors around mirrors and thin structures in NYU-D, making reported metrics unreliable.The benchmark authors identify frequent annotation noise despite specialized depth sensors.Source: §6.1 · p. 7
  • Their limited diversity—such as NYU-D indoor rooms and KITTI street scenes—and low resolution around 500×500 weaken conclusions about unseen and high-resolution scenes.Modern applications may require images around 1000×2000, whose performance is not established by these benchmarks.Source: §6.1 · p. 7

6.2 DA-2K

  • DA-2K provides precise relative depth through sparse pixel pairs, extensive scene coverage, and mostly high-resolution images, avoiding impractical dense pixel annotation.The benchmark uses sparse depth relationships, where each pair identifies which of two pixels is closer.Source: §6.2 · p. 7
  • DA-2K combines model-voted and human-verified pixel pairs with manual analysis of challenging cases, then triple-checks annotations across 1K images and 2K pairs.Its images span eight application scenarios selected using scenario-specific keywords and Flickr retrieval.Source: §6.2 · pp. 7–8
  • Even the most lightweight Depth Anything V2 model surpasses all other community models on DA-2K’s eight representative scenarios.Table 3 evaluates performance on the proposed DA-2K benchmark.Source: Table 3 · p. 8
  • DA-2K supplements rather than replaces existing benchmarks because sparse depth is insufficient for precise dense reconstruction, while supporting prior model validation and future 3D-awareness testing.The authors also position it as a potential testbed for multimodal LLMs [41] [21] [3].Source: §6.2 · p. 8

7 Experiment

Depth Anything V2 outperforms prior methods on the proposed DA-2K benchmark and improves downstream metric depth estimation, while pseudo-labeled real images substantially strengthen training. Ablations further show that pseudo labels outperform noisy manual labels on real images.

7.2 Zero-Shot Relative Depth Estimation

  • On five unseen conventional benchmarks, Depth Anything V2 surpasses MiDaS V3.1 [7] and is comparable to Depth Anything V1, though slightly lower on two datasets.These metrics do not fully reflect the model’s intended improvements for thin structures, complex scenes, and transparent objects.Source: §7.2 · p. 8
  • 10.6% higher relative-depth discrimination accuracy than Margold is achieved by the most capable model on the diverse DA-2K benchmark.Even the smallest model significantly outperforms other heavy Stable Diffusion-based models, including Marigold [31] and Geowizard [20].Source: §7.2 · pp. 8–9

7.3 Fine-tuned to Metric Depth Estimation

  • Significant improvements over previous methods are obtained on both NYU-D and KITTI, with the ViT-S model surpassing other ViT-L-based models.The encoder is transferred through the ZoeDepth [6] pipeline, replacing its MiDaS [7] encoder.Source: §7.3 · p. 9
  • Metric-depth models are fine-tuned on Hypersim [58] and Virtual KITTI [9] synthetic data for indoor and outdoor estimation because NYUv2 and KITTI training sets limit fine-grained and transparent-object robustness.The resulting models are intended for real-world applications such as multi-view synthesis.Source: §7.3 · p. 9

7.4 Ablation Study

  • Pseudo-labeled real images greatly enhance models over synthetic-only training, and removing synthetic images can slightly improve smaller ViT-S and ViT-B models.Student models are therefore trained purely on pseudo-labeled images.Source: §7.4 · p. 9, Table 5 · p. 9
  • Models trained with produced pseudo labels significantly outperform models trained with DIML’s original manual labels, indicating higher pseudo-label quality and substantial manual-label noise.The comparison uses real images from the DIML [14] dataset and evaluates transfer performance.Source: §7.4 · p. 10, Table 6 · p. 10

8 Related Work

Prior monocular depth work spans in-domain metric estimation, zero-shot relative estimation, large-scale data use, and knowledge distillation. Depth Anything V2 emphasizes precise synthetic data, pseudo-labeled real images, and prediction-level distillation from a large teacher.

Monocular depth estimation

  • Depth Anything V2 argues for replacing labeled real images with precise synthetic images, then restoring generalization through 62M pseudo-labeled real images and a scaled teacher model.This contrasts with prior approaches using Stable Diffusion [31] [25] [20], large labeled datasets such as MiDaS [55] [7] and Metric3D, or unlabeled data in Depth Anything V1.Source: §8 · p. 10

Learning from unlabeled real images

  • “Precise synthetic data + pseudo-labeled real data” is presented as more promising than labeled real data, especially when replacing all labeled real images with synthetic images.The setting boosts a baseline trained with 0.6M labeled images using 62M unlabeled images, unlike semi-supervised benchmarks limited to small-scale labeled and unlabeled data.Source: §8 · p. 10

Knowledge distillation

  • Unlike typical knowledge distillation [73], Depth Anything V2 distills predictions through extra unlabeled real images rather than features or logits through labeled images.The approach supports distillation from a 1.3B-parameter teacher to a 25M-parameter student without relying primarily on delicate loss designs [69] or distillation pipelines [10].Source: §8 · p. 10

9 Conclusion … B.2 Transferring performance of each labeled dataset

Depth Anything V2 is presented as a robust, fine-grained monocular depth foundation model supporting multiple scales, downstream fine-tuning, and evaluation through the DA-2K benchmark. Experiments show strong transfer to semantic segmentation and complementary benefits from the synthetic datasets used for training.

9 Conclusion

  • Depth Anything V2 provides robust, fine-grained depth prediction, model sizes from 25M to 1.3B parameters, and easy fine-tuning for downstream tasks.It is presented as a more powerful foundation model for monocular depth estimation and a promising initialization.Source: §9 · p. 10
  • The DA-2K benchmark addresses poor diversity and rich noise in existing test sets with diverse high-resolution images and precise, challenging sparse depth labels.The benchmark is intended to support evaluation of future monocular depth estimation research.Source: §9 · p. 10

Appendix

  • The appendix compiles a comprehensive presentation of Depth Anything V2 and directs readers to detailed sections for further understanding and visualization.Its contents cover training data, transfer experiments, and dataset-specific analyses.Source: Appendix · p. 11

A Sources of Training Data

  • Depth Anything V2 replaces labeled real datasets from Depth Anything V1 with five synthetic datasets to improve label preciseness.The synthetic datasets train the teacher model used for pseudo-labeling.Source: A Sources of Training Data · p. 11, Table 7 · p. 12
  • Eight large-scale public datasets provide 62M diverse raw real images, which the strongest teacher pseudo-labels for purely pseudo-labeled student training.This design mitigates distribution shift and limited diversity from synthetic images.Source: A Sources of Training Data · p. 11

B Experiments

  • The experiments collectively assess Depth Anything V2’s transferability through semantic segmentation and dataset-specific analyses of generalization, sharpness, and viewpoint handling.These evaluations cover the model’s downstream initialization potential and the distinct contributions of its synthetic training datasets.Source: §B.1 · pp. 11–12, §B.2 · p. 12

B.1 Fine-tuned to semantic segmentation

  • Depth Anything V2 encoders consistently achieve the best semantic-segmentation performance across model scales, without Mapillary [1] or COCO [40] pre-training.The transfer uses Mask2Former and indicates potential as initialization for diverse downstream semantic-related tasks.Source: Table 8 · p. 12, §B.1 · p. 12

B.2 Transferring performance of each labeled dataset

  • Hypersim [58] and IRS provide the strongest generalization benefits, while VKITTI 2 [9] improves prediction sharpness and BlendedMVS supports bird’s-eye-view handling.Each synthetic dataset contributes a distinct property to the combined model performance.Source: §B.2 · p. 12, Table 9 · p. 12

B.3 Transferring performance of each unlabeled dataset … C.1 Per-scenario accuracy

Depth Anything V2 gains robustness and fine-grained accuracy from diverse unlabeled data, precise synthetic supervision, suitable encoders, and pseudo-label training. Across specialized surfaces, qualitative comparisons, and DA-2K scenarios, the results support these design choices and strong generalization.

B.3 Transferring performance of each unlabeled dataset

  • Unlabeled real images benefit nearly all test sets, with same-domain transfer strongest and even contradictory-domain transfer improving KITTI from δ1 0.889 to 0.913.LSUN improves NYU-D from δ1 0.928 to 0.970, while Sintel is the exception.Source: §B.3 · p. 13, Table 10 · p. 13

B.4 Are such large-scale unlabeled images really necessary?

  • Data diversity remains essential: iterating solely on SA-1B cannot replace the 62M-image multi-dataset training needed for open-world generalization.The comparison uses the same training iterations as all unlabeled sets, but more cycles on one dataset remain inferior.Source: §B.4 · p. 13, Table 11 · p. 13

B.5 Performance on transparent or reflective surfaces

  • On transparent surfaces, V2 reaches 0.912 after fine-tuning versus 0.917 for the challenge first-place score, while its zero-shot performance strongly exceeds MiDaS and Depth Anything V1.The pre-trained model also outperforms the DINOv2 [50] encoder, 0.912 versus 0.758.Source: §B.5 · p. 13, Table 12 · p. 13

B.6 Comparison among various pre-trained encoders

  • DINOv2 encoders [50] [16] are strongest at ViT-large scale, but DINOv2-G Reg [16] is inferior to the original encoder at ViT-giant scale, motivating the original DINOv2 design.The encoder comparison covers BEiT [4], SAM, SynCLR [75], DINOv2 [50], and DINOv2 with registers [16].Source: §B.6 · p. 14, Table 13 · p. 14

B.7 Benefit of gradient matching loss to fine-grained predictions

  • On precise synthetic labels, increasing gradient-matching loss weight from 0.5 to 4.0 steadily improves sharpness, while weight 2.0 balances metric results and detail.The loss brings no evident improvement on labeled real datasets, whose depth labels are sparse and coarse.Source: §B.7 · p. 14, Figure 10 · p. 14

B.8 Test-time resolution scaling up; B.9 Harm of real labeled images to fine-grained predictions

  • Test-time resolution scaling from the 518 base to 2× and 4× progressively improves depth sharpness, whereas adding only 5% HRWSI causes coarse labels to harm fine-grained predictions.These findings support training the largest teacher purely on synthetic images while increasing resolution at inference when finer maps are desired.Source: §B.8 · p. 14, Figure 11 · p. 15, §B.9 · p. 15

B.10 Qualitative comparison between Depth Anything V1 and V2; B.11 Qualitative comparison between Marigold and Depth Anything V2; B.12 Qualitative comparison between our metric depth models and ZoeDepth

  • V2 is finer and more robust than V1 and Marigold [31], while metric models trained on Hypersim [58] and Virtual KITTI [9] are compared with ZoeDepth trained on NYUv2.V2 is also described as robust to transparent objects, and the metric models use purely synthetic fine-tuning.Source: §B.10 · p. 15, §B.11 · p. 16, §B.12 · p. 16

B.13 Qualitative comparison between w/ and w/o pseudo-labeled real images; B.14 Qualitative results of produced pseudo labels; B.15 Qualitative results on test benchmarks; C.1 Per-scenario accuracy

  • Pseudo-labeled real images give the 25M-parameter model powerful complex-scene generalization, while the teacher produces precise labels and V2 is consistently better than V1 on standard benchmarks.The DA-2K per-scenario comparison between labeled synthetic images Dl and pseudo-labeled real images Du demonstrates the value of large-scale unlabeled data and pseudo-label precision.Source: §B.13 · p. 16, §B.14 · p. 16, §B.15 · p. 16

C.2 Comparison with the DIW dataset · C.3 Annotation details · C.4 Visualization

DA-2K uses the same sparse pixel-pair format as DIW [11] but improves precision, organization, diversity, and resolution. Its annotations are selected through ratio-based and manual procedures, covering eight scenarios with representative visualized samples.

C.2 Comparison with the DIW dataset

  • DA-2K improves on DIW [11] through more precise annotations, scenario-based organization, broader image diversity, and mostly higher-resolution images.The dataset excludes controversial pairs, organizes images by application scenario, includes non-real images, and primarily uses 1500×2000 images rather than DIW’s roughly 300×500 images.Source: §C.2 · pp. 16–17
  • DA-2K is more precise because it excludes many hard-to-decide or controversial pixel pairs, whereas DIW is described as very noisy.MiDaS [7] results also support the concern that better and larger models perform worse on DIW.Source: §C.2 · p. 16
  • DA-2K organizes images by application scenarios, enabling evaluation results to provide straightforward insights for each individual scenario.DIW instead randomly downloads Flickr images without careful organization.Source: §C.2 · p. 16
  • DA-2K expands beyond DIW’s real-life imagery with AI-generated and cartoon images, while providing mostly 1500×2000 images instead of DIW’s roughly 300×500 resolution.These additions target the widespread use of monocular depth estimation in AIGC [39].Source: §C.2 · p. 17

C.3 Annotation details

  • Annotation candidates are restricted to pixel pairs whose predicted depth ratio exceeds 3, while transparent and object scenarios use manual selection of challenging pairs.Other scenarios combine automatic disagreement-based selection with manual selection.Source: §C.3 · p. 17
  • DA-2K covers eight evaluation scenarios using GPT-4-generated Flickr-search keywords listed in Table 15.The keywords are used to download images corresponding to each scenario.Source: Table 15 · p. 17, §C.3 · p. 17

C.4 Visualization

  • Figure 19 visualizes DA-2K samples spanning diverse representative scenarios with precise sparse annotations.The visualization provides examples of the benchmark’s scenario coverage and annotation format.Source: §C.4 · p. 17

D Limitations

The current training pipeline relies on 62M unlabeled images with heavy computational cost, while its synthetic training sets lack sufficient diversity. Future work will improve data efficiency and broaden synthetic-image sources for stronger teacher pseudo-labeling.

D Limitations

  • Using 62M unlabeled images imposes a very heavy computational burden, motivating more efficient use of large-scale visual data.The authors identify data-efficiency improvements as a future research direction.Source: D Limitations · p. 17
  • The current synthetic training sets are not diverse enough, so future work will collect synthetic images from more sources to train a more capable teacher for better pseudo labeling.The limitation directly concerns synthetic-data diversity and teacher-model supervision.Source: D Limitations · p. 17

Related papers

Loading 2406.09414v2…