PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Anything V2

Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao

arXiv:2406.09414v1cs.CV

TL;DR

Depth Anything V2 addresses noisy real depth labels, synthetic-to-real transfer challenges, and limited benchmark diversity in monocular depth estimation. It uses precise synthetic supervision, a scaled teacher, and large-scale pseudo-labeled real images to train robust models across sizes, while introducing a more diverse evaluation benchmark.

  • Problem

    Monocular depth estimation lacks sufficiently precise and diverse training and evaluation data because real labels and current test benchmarks contain noise and limited scene coverage.

  • Method

    The paper replaces labeled real images with synthetic images, scales the synthetic-trained teacher, and trains students through large-scale pseudo-labeled real images.

  • Results

    The work builds a more capable monocular depth foundation model with robust, fine-grained predictions, model sizes from 25M to 1.3B parameters, and downstream fine-tuning capability.

  • Takeaways & Limitations

    Depth Anything V2 provides a foundation model family and DA-2K benchmark for broad monocular depth applications and future evaluation.

  • Takeaways & Limitations

    Existing depth benchmarks remain limited by frequent annotation noise and narrow scene diversity, weakening their reliability as measures of real-world performance.

Abstract

from arXiv · show

This work presents Depth Anything V2. Without pursuing fancy techniques, we aim to reveal crucial findings to pave the way towards building a powerful monocular depth estimation model. Notably, compared with V1, this version produces much finer and more robust depth predictions through three key practices: 1) replacing all labeled real images with synthetic images, 2) scaling up the capacity of our teacher model, and 3) teaching student models via the bridge of large-scale pseudo-labeled real images. Compared with the latest models built on Stable Diffusion, our models are significantly more efficient (more than 10x faster) and more accurate. We offer models of different scales (ranging from 25M to 1.3B params) to support extensive scenarios. Benefiting from their strong generalization capability, we fine-tune them with metric depth labels to obtain our metric depth models. In addition to our models, considering the limited diversity and frequent noise in current test sets, we construct a versatile evaluation benchmark with precise annotations and diverse scenes to facilitate future research.

1 Introduction

Depth Anything V2 targets a monocular depth foundation model that combines robust complex-scene predictions, fine details, varied scales, and downstream transfer. Its central findings are that precise synthetic labels and large-scale pseudo-labeled real images address key data challenges without requiring sophisticated techniques.

  • Monocular depth estimation supports applications including 3D reconstruction, navigation, autonomous driving, and AI-generated content.
  • The target model should produce robust complex-scene predictions, fine details, varied scales, efficient inference, and downstream transferability.
  • Replacing all labeled real images with precise synthetic images is the most critical modification for obtaining fine details with efficient discriminative models.
  • Scaling the synthetic-trained teacher and teaching smaller students through large-scale pseudo-labeled real images addresses synthetic-data drawbacks.
  • The work constructs a versatile evaluation benchmark with precise annotations and diverse scenes because current test sets are too noisy to reflect model strengths.

2 Revisiting the Labeled Data Design of Depth Anything V1

The paper revisits the assumption that more labeled real images necessarily improve monocular depth estimation. It identifies label noise and missing fine detail in real datasets, motivating precise synthetic supervision.

  • Prior work accumulated up to 16M labeled images, but the benefit of such large labeled datasets remains insufficiently examined.
  • Real labeled depth data contain noise from transparent objects, textureless or repetitive patterns, dynamic objects, and outliers.
  • Real datasets provide coarse supervision around object boundaries and thin holes, producing over-smoothed predictions and similar learned errors.
  • The method therefore replaces labeled real images with synthetic images whose depth labels are treated as truly ground truth.
  • Synthetic images provide precise depth for fine structures, thin holes, small objects, transparent objects, and reflective surfaces.

3 Challenges in Using Synthetic Data

Synthetic images offer precise depth supervision but do not transfer directly to real-world monocular depth estimation. Their distribution shift and restricted scene coverage create complementary generalization problems.

  • Synthetic images differ from real images in style, color distributions, layout order, and visual randomness, creating a synthetic-to-real distribution shift.
  • Synthetic datasets use predefined scene types, limiting coverage of real-world settings such as crowded people.
  • Only DINOv2-G produces satisfying synthetic-to-real predictions among the compared pretrained encoders, while smaller models suffer severe generalization issues.
  • Purely synthetic training still fails on cases such as assigning ultra-far depth to the sky and maintaining consistent head-to-body depth.
  • Mixing real images with synthetic images can make coarse real depth destructive to fine-grained prediction, while expanding synthetic coverage is unsustainable.

4 Key Role of Large-Scale Unlabeled Real Images

Depth Anything V2 uses unlabeled real images as a bridge between precise synthetic supervision and real-world deployment. A synthetic-trained teacher labels real images, and students learn from those large-scale pseudo-labeled images.

  • The pipeline trains the most capable model on high-quality synthetic images, generates pseudo-depth labels for unlabeled real images, and trains new models on them.
  • Pseudo-labeled real images make transfer more reliable by exposing models to the real-world data distribution.
  • Large-scale unlabeled real images expand scene coverage and provide more informative, less redundant training examples than repeatedly sampled synthetic data.
  • The approach transfers the strongest model’s knowledge to smaller models through label-level distillation on extra unlabeled real data.

5 Depth Anything V2

Depth Anything V2 uses a three-stage pipeline: train a synthetic-data teacher, generate pseudo-depth on unlabeled real images, and train student models for robust generalization. It also provides multiple model scales and introduces the DA-2K evaluation benchmark.

  • Overall Framework: The pipeline trains a reliable DINOv2-G teacher purely on high-quality synthetic images, generates pseudo-depth for unlabeled real images, then trains student models on those pseudo-labels.The student stage does not require synthetic images.
  • Overall Framework: Four student models are released using DINOv2 small, base, large, and giant encoders.
  • Training Data: The training uses 595K synthetic images and 62M pseudo-labeled real images, while ignoring the top 10% largest-loss regions of each pseudo-labeled sample.
  • Evaluation: DA-2K provides an evaluation benchmark with precise annotations and diverse scenes to address noise in existing test sets.Its annotation pipeline is illustrated through relative-depth pairs and scenario coverage.

6 A New Evaluation Benchmark: DA-2K

The paper argues that common depth benchmarks are noisy, narrow in scene coverage, and low resolution. DA-2K addresses these issues with precise sparse depth annotations across diverse, mostly high-resolution images, while supplementing rather than replacing existing benchmarks.

  • Limitations in Existing Benchmarks: Existing benchmarks contain noisy annotations, limited scene diversity, and mostly low-resolution images, weakening their ability to reflect real-world reliability.NYU-D focuses on a few indoor rooms, while KITTI contains several street scenes.
  • DA-2K: DA-2K targets precise depth relationships, extensive scene coverage, and mostly high-resolution images for relative monocular depth estimation.
  • DA-2K: DA-2K samples key-point pairs from SAM-predicted masks and sends pairs with disagreement among four expert models to human annotators.
  • DA-2K: 1K images with 2K pixel pairs are annotated, with every annotation triple-checked by two additional annotators.
  • Position of DA-2K: DA-2K complements rather than replaces existing benchmarks because sparse accurate depth remains insufficient for precise dense scene reconstruction.It can also provide quick prior validation for models in covered scenarios.

7 Experiment

Depth Anything V2 is comparable to or better than prior methods on conventional benchmarks, while DA-2K reveals stronger relative-depth performance and pseudo-labeled real images substantially improve training. Its encoder also transfers effectively to metric depth estimation.

  • Zero-Shot Relative Depth Estimation: On conventional benchmarks, Depth Anything V2 surpasses MiDaS and is comparable to Depth Anything V1, although it is slightly inferior on two datasets.The paper notes that these metrics do not capture fine details or robustness to complex and transparent scenes.
  • Zero-Shot Relative Depth Estimation: Even the smallest model significantly outperforms heavy Stable Diffusion-based models on the diverse DA-2K benchmark.
  • Zero-Shot Relative Depth Estimation: 10.6% higher accuracy is achieved by the most capable model than Marigold on relative depth discrimination.
  • Fine-tuned to Metric Depth Estimation: The transferred encoder significantly improves metric depth estimation on both NYU-D and KITTI, with even the ViT-S model surpassing models built on ViT-L.
  • Fine-tuned to Metric Depth Estimation: The paper fine-tunes its encoder on Hypersim and Virtual KITTI synthetic datasets to obtain indoor and outdoor metric depth models.This responds to fine-grained prediction and transparency-robustness problems associated with noisy NYUv2 and KITTI training sets.
  • Ablations: Pseudo-labeled real images greatly enhance models compared with training solely on synthetic images, and can slightly improve smaller models when synthetic images are removed.

8 Related Work

Related work spans discriminative and generative approaches to monocular depth estimation, alongside large-scale data-driven learning, unlabeled-image learning, and knowledge distillation. Depth Anything V2 emphasizes precise synthetic data plus pseudo-labeled real data as an alternative training roadmap.

  • Monocular Depth Estimation: Recent monocular depth models use either discriminative encoders or generative Stable Diffusion-based modeling, with different strengths in detail and robustness.
  • Learning from Unlabeled Real Images: The paper studies scaling a baseline trained on 0.6M labeled images with 62M unlabeled images, beyond small-scale semi-supervised benchmarks.
  • Knowledge Distillation: Its approach performs prediction-level distillation through extra unlabeled real images, unlike typical feature- or logit-level distillation through labeled images.

9 Conclusion

Depth Anything V2 combines precise synthetic supervision, pseudo-labeled real images, and scalable encoders to improve monocular depth estimation and downstream transfer. It also provides broad training-data analyses and varied model sizes.

  • Depth Anything V2 provides robust, fine-grained predictions, model sizes from 25M to 1.3B parameters, and initialization for downstream fine-tuning.
  • The training data combine five precise synthetic datasets with 62M pseudo-labeled real images from eight public datasets.
  • The released encoders consistently achieve the best performance when transferred to semantic segmentation.
  • Hypersim and IRS provide the strongest generalization among the examined synthetic datasets, while VKITTI 2 improves prediction sharpness.

B.3 Transferring performance of each unlabeled dataset

The analyses show that diverse pseudo-labeled real images, suitable encoder choices, and several inference or training decisions materially affect generalization and depth quality. Depth Anything V2 also improves comparisons on transparent surfaces and standard benchmarks.

  • Transferring performance of each unlabeled dataset: 0.928 → 0.970: adding LSUN improves the δ1 metric on NYU-D when unlabeled and test images share an indoor domain.
  • Are such large-scale unlabeled images really necessary?: Data diversity across multiple unlabeled datasets remains important; repeating training on SA-1B alone cannot replace broader dataset coverage.
  • Performance on transparent or reflective surfaces: 0.912 vs. 0.917: fine-tuning Depth Anything V2 nearly reaches the first-place score in the NTIRE 2024 Transparent Surface Challenge.
  • Comparison among various pre-trained encoders: DINOv2 encoders outperform the compared pre-trained encoders at ViT-large scale, while DINOv2-G Reg is inferior to the original version at ViT-giant scale.
  • Test-time resolution scaling up: 2× and 4× test-time resolution scaling progressively improves depth-map sharpness beyond the 518 base resolution.
  • Harm of real labeled images to fine-grained predictions: Adding HRWSI real images, despite comprising only 5% of training images, harms the original fine-grained predictions because of coarse labels.
  • Qualitative comparisons: Depth Anything V2 produces finer and more robust predictions than V1 and Marigold, and pseudo-labeled real images improve generalization even for the 25M model.

C.2 Comparison with the DIW dataset

DA-2K is designed as a more precise, organized, diverse, and higher-resolution alternative to DIW. Its annotations and scenario structure support more interpretable evaluation across applications.

  • DA-2K excludes hard-to-decide or controversial pixel pairs, making its sparse relative-depth annotations more precise than DIW’s noisy labels.
  • DA-2K organizes images by application scenario, enabling evaluation results for individual scenarios rather than an unstructured image collection.
  • DA-2K adds non-real images, including AI-generated and cartoon images, to broaden coverage beyond real-life imagery.
  • DA-2K primarily provides high-resolution images around 1500×2000, compared with DIW images around 300×500.
  • The benchmark selects pixel pairs using predicted depth-ratio disagreement, manual analysis for transparent and object scenarios, or both pipelines for other scenarios.
  • DA-2K samples cover diverse representative scenarios and use precise sparse annotations.

D Limitations

The remaining limitations concern computational cost and insufficient diversity in the synthetic training data. The qualitative comparisons also document model behavior across open-world, benchmark, and pseudo-labeling settings.

  • Limitations: Training with 62M unlabeled images imposes a heavy computational burden, motivating more efficient use of large-scale visual data.
  • Limitations: The current synthetic training sets are not diverse enough, limiting the teacher’s potential for better pseudo-labeling.
  • Qualitative comparisons: The figures compare Depth Anything V2 with V1, Marigold, and ZoeDepth across open-world and metric-depth settings.
  • Pseudo-labeled real images: A DINOv2-small model trained only on synthetic images has severe generalization problems, while pseudo-labeled real images substantially improve robustness.
  • Pseudo-label visualization: Pseudo-depth labels are visualized on diverse image sources, including BDD100K, ImageNet-21K, Places365, and SA-1B.
  • Evaluation coverage: Qualitative results are reported on KITTI, NYU, and DIODE, while DA-2K includes diverse scenarios with precise sparse annotations.
Loading 2406.09414v1…