PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao

arXiv:2401.10891v2cs.CV

TL;DR

Monocular depth estimation lacks broad depth-labeled datasets for foundation-model-level generalization. Depth Anything scales training with automatically annotated unlabeled images, a harder optimization target, and semantic-prior supervision, achieving stronger zero-shot capability than MiDaS-BEiTL-512 and serving as a strong downstream initialization. Its scope remains bounded by ViT-Large models and 512×512 training resolution.

  • Problem

    Monocular depth estimation lacks depth datasets with tens of millions of labels, limiting data coverage and generalization across scenes.

  • Method

    Depth Anything collects and automatically annotates 62M unlabeled images, uses a harder optimization target, and transfers semantic priors from pretrained encoders.

  • Results

    Depth Anything exhibits stronger zero-shot capability than MiDaS-BEiTL-512 and provides a promising initialization for downstream metric depth estimation and semantic segmentation.

  • Takeaways & Limitations

    Large-scale unlabeled images can support robust zero-shot depth estimation and useful downstream initialization when paired with the proposed training strategies.

  • Takeaways & Limitations

    The current model is constrained to ViT-Large and 512×512 training resolution, motivating larger models and higher-resolution training.

Abstract

from arXiv · show

This work presents Depth Anything, a highly practical solution for robust monocular depth estimation. Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet. Our models are released at https://github.com/LiheYoung/Depth-Anything.

1. Introduction

Depth Anything targets robust monocular depth estimation by scaling training with cheap, diverse unlabeled images and adding two forms of supervision. It reports stronger zero-shot capability than MiDaS-BEiTL-512 and better fine-tuned performance than ZoeDepth.

  • Motivation: MDE needs foundation-model capabilities, but existing datasets lack tens of millions of depth labels and MiDaS suffers poor coverage in some scenarios.The paper motivates broader data coverage for generalization across robotics, autonomous driving, and other applications.
  • Data scaling: Monocular unlabeled images are cheap, diverse, and easy to annotate using a pretrained MDE model, producing dense depth maps without stereo matching.The proposed data source avoids specialized devices and can cover broader scene distributions.
  • Data scaling: A data engine collects 62M unlabeled images from eight public datasets and automatically annotates them using an initial MDE model trained on 1.5M labeled images.The unlabeled images are jointly learned with labeled images after automatic annotation.
  • Training strategies: The method challenges the model with a harder optimization target when learning unlabeled images, rather than learning raw unlabeled images directly.This strategy is intended to obtain additional visual knowledge from large-scale unlabeled data.
  • Training strategies: Semantic priors from pretrained encoders provide auxiliary supervision for scene understanding instead of using an auxiliary semantic segmentation task.The paper motivates preserving semantic information because decoding into discrete classes can lose information.
  • Results: Depth Anything shows stronger zero-shot capability than MiDaS-BEiTL-512 and significantly outperforms ZoeDepth after fine-tuning with metric depth.These are the paper’s headline comparisons for zero-shot and downstream metric depth estimation.

2. Related Work

Related work spans monocular depth estimation, zero-shot depth estimation, and semi-supervised learning. Depth Anything focuses on the underexplored setting of abundant labeled data alongside much larger unlabeled image collections.

  • Monocular depth estimation: Traditional monocular depth methods relied on handcrafted features and explicit depth cues, struggling with occlusions and textureless regions.Deep learning methods later learned depth representations from annotated datasets.
  • Monocular depth estimation: Deep learning MDE methods improve accuracy through task reformulation, additional priors, and improved objectives but remain difficult to generalize to unseen domains.The cited approaches include classification-based regression and stronger objective functions.
  • Zero-shot depth estimation: Zero-shot depth estimation seeks models trained on diverse data that predict depth for arbitrary images, while pioneering methods used sparse supervision on limited point pairs.Depth Anything belongs to this research direction.
  • Zero-shot depth estimation: MiDaS enables multi-dataset training with an affine-invariant loss that ignores dataset-specific depth scales and shifts, producing relative depth.Later methods estimate metric depth, but the paper notes poorer generalization than MiDaS in practice.
  • Leveraging unlabeled data: Existing semi-supervised learning typically assumes limited images and rarely studies sufficient labeled data combined with larger-scale unlabeled data.The paper reports that unlabeled images can enhance coverage and improve generalization and robustness in zero-shot MDE.

3. Depth Anything

Depth Anything scales monocular depth estimation with jointly trained labeled and pseudo-labeled images, challenging unlabeled-image learning with strong perturbations and transferring semantic features from a frozen encoder. The resulting pipeline combines large-scale data coverage with robust and semantically informed representations.

  • 3.1. Learning Labeled Images: A teacher trained on labeled images assigns pseudo-depth labels to unlabeled images, after which a reinitialized student trains on both sets.The method uses a teacher–student pipeline and combines labeled and pseudo-labeled data for student training.
  • 3.1. Learning Labeled Images: 1.5M labeled and 62M unlabeled images are jointly used to train Depth Anything.The labeled data come from six public datasets, while the unlabeled data are automatically annotated for training.
  • 3.1. Learning Labeled Images: Monocular unlabeled images expand data coverage because they are easy to collect, diverse, and can be densely annotated by a pretrained depth model.The authors contrast this approach with costly sensor acquisition, stereo matching, and SfM reconstruction.
  • 3.2. Unleashing the Power of Unlabeled Images: Strong color and spatial perturbations make unlabeled-image supervision harder, encouraging invariant representations and improving the labeled-image baseline.The perturbations include color jittering, Gaussian blurring, and CutMix; CutMix is applied by spatially combining two unlabeled images.
  • 3.3. Semantic-Assisted Perception: Feature alignment with a frozen DINOv2 encoder transfers semantic information while a tolerance margin preserves part-level depth discrimination.The aligned features come from the depth model and the frozen semantic encoder; pixels already exceeding the cosine-similarity margin are excluded from the feature loss.

4. Experiment

Depth Anything is evaluated for zero-shot relative and metric depth estimation across indoor, outdoor, and unseen domains. It also uses its pre-trained encoder for downstream metric-depth fine-tuning, achieving improvements over prior methods.

  • Zero-Shot Relative Depth Estimation: Across six unseen datasets, Depth Anything surpasses MiDaS in both AbsRel (↓) and δ1 (↑) for zero-shot relative depth estimation.The evaluation covers KITTI, NYUv2, Sintel, DDAD, ETH3D, and DIODE.
  • Zero-Shot Relative Depth Estimation: 0.230 AbsRel (↓) and 0.789 δ1 (↑) improve on MiDaS's 0.251 and 0.766, respectively, on DDAD.Both models use a ViT-L encoder in this comparison.
  • Zero-Shot Relative Depth Estimation: Depth Anything remains superior to MiDaS on KITTI and NYUv2 despite MiDaS using corresponding training images while Depth Anything uses neither dataset for training.On KITTI, AbsRel is 0.076 versus 0.127, and δ1 is 0.947 versus 0.850.
  • Metric Depth Estimation: Fine-tuning with metric depth information uses the pre-trained encoder for downstream metric depth estimation, with experiments covering in-domain and zero-shot metric depth settings.The encoder is initialized from Depth Anything, while the decoder is randomly initialized.
  • Metric Depth Estimation: On NYUv2, Depth Anything improves over VPD from 0.964 to 0.984 δ1 (↑) and from 0.069 to 0.056 AbsRel (↓).Similar improvements are reported on KITTI, where δ1 (↑) rises from 0.978 to 0.982.
  • Metric Depth Estimation: Replacing ZoeDepth's MiDaS encoder with the Depth Anything encoder improves metric depth estimation across unseen indoor and outdoor datasets.The remaining ZoeDepth components are left unchanged.

4.4. Fine-tuned to Semantic Segmentation

Depth Anything’s encoder transfers effectively to semantic segmentation and other downstream tasks, while ablations show that perturbations and semantic constraints strengthen the use of unlabeled data.

  • Semantic segmentation: The MDE encoder achieves 86.2 mIoU on Cityscapes after fine-tuning for semantic segmentation.
  • Semantic segmentation: The encoder is proposed as a generic multi-task representation for middle-level and high-level visual perception.
  • Ablations: Strong perturbations make the student seek additional visual knowledge, while the semantic constraint further amplifies unlabeled-data benefits.
  • Downstream encoder comparisons: 0.951 vs. 0.984 in NYUv2 δ1 and 52.4 vs. 59.4 in ADE20K mIoU show stronger downstream performance than the MiDaS encoder.
  • Downstream encoder comparisons: The produced encoder outperforms the original DINOv2 encoder on downstream metric depth estimation and semantic segmentation.

4.6. Qualitative Results

Qualitative evaluations show robust predictions across six unseen datasets and improved depth-conditioned image synthesis compared with MiDaS.

  • Unseen-domain predictions: Depth Anything produces robust predictions across test images from six unseen datasets and various domains.
  • Depth-conditioned synthesis: Compared with MiDaS, it produces more accurate depth estimates and better ControlNet synthesis results.The authors retrain a depth-conditioned ControlNet using Depth Anything for image synthesis and video editing.

5. Conclusion

Depth Anything uses large-scale unlabeled images with challenging perturbations and semantic-prior preservation to build a robust monocular depth foundation model. It shows strong zero-shot depth capability and transfers to metric depth estimation and semantic segmentation.

  • Depth Anything targets robust monocular depth estimation as a practical foundation model for images under varied circumstances.
  • The method exploits cheap, diverse unlabeled images through stronger optimization targets and preserved semantic priors from pre-trained models.
  • The resulting model exhibits excellent zero-shot depth estimation and provides a promising initialization for metric depth estimation and semantic segmentation.

6. More Implementation Details

Implementation uses resolution and patch-size choices tailored to DINOv2, then fine-tunes the encoder with task-specific codebases, resolutions, and learning rates.

  • Depth estimation: Images are resized to a shorter side of 518, cropped to 518×518 for training, and kept uncropped during inference with dimensions divisible by 14.Predictions are interpolated to the original resolution, with scale and shift alignment used for zero-shot evaluation.
  • Metric depth: Metric-depth fine-tuning replaces the MiDaS encoder in ZoeDepth with Depth Anything and uses 392×518 for NYUv2 and 384×768 for KITTI.The encoder learning rate is set to 1/50 of the randomly initialized decoder’s learning rate.
  • Semantic segmentation: Semantic-segmentation fine-tuning uses MMSegmentation, 896×896 inputs, Mask2Former, and separate 160K-iteration ADE20K and 80K-iteration Cityscapes schedules.The encoder learning rate is 3e-6, while the decoder learning rate is 10× larger.

7. More Ablation Studies

The ablations examine feature-alignment design choices, showing that tolerance margins matter and that applying the auxiliary constraint to unlabeled rather than labeled data is beneficial.

  • Data selection: Applying feature alignment to labeled data is not beneficial because its mean AbsRel is almost unchanged from the baseline at 0.180.The authors attribute this to higher-quality labeled annotations and possible interference from semantic loss.
  • Data selection: Applying the auxiliary constraint to unlabeled data can combat noise in pseudo depth labels while adding semantic capability.The comparison motivates enforcing Lfeat on unlabeled rather than labeled data.

8. Limitations and Future Works

The current model scale and training resolution remain constrained, motivating future expansion to larger encoders, improved teachers, and higher-resolution training.

  • Model scale: The largest model is currently limited to ViT-Large, with future work targeting ViT-Giant to produce more accurate pseudo labels.The larger teacher is intended to support smaller models such as ViT-L and ViT-B.
  • Training resolution: The 512×512 training resolution is considered insufficient for real-world applications, motivating retraining at 700+ or even 1000+ resolution.The proposed higher resolutions are framed as future work to facilitate real-world applications.

9. More Qualitative Results

Qualitative results compare Depth Anything with MiDaS across six unseen test sets, reporting higher accuracy and stronger robustness across indoor and outdoor scenes.

  • Cross-dataset comparison: Depth Anything is compared with DPT-BEiTL-512, the strongest MiDaS v3.1 model, across KITTI, NYUv2, Sintel, DDAD, ETH3D, and DIODE.The six datasets span representative unseen indoor and outdoor scenes.
  • Cross-dataset comparison: The qualitative results report higher depth estimation accuracy and stronger robustness for Depth Anything across the six unseen datasets.The accompanying figures provide results for KITTI, NYUv2, Sintel, DDAD, ETH3D, and DIODE.
  • KITTI: The KITTI visualization compares predictions with MiDaS v3.1 because KITTI ground truth is extremely sparse and difficult to visualize.Brighter colors denote closer distance.
  • NYUv2: The NYUv2 comparison notes that MiDaS uses NYUv2 training data, whereas Depth Anything does not.This distinction is relevant when interpreting the qualitative comparison.
Loading 2401.10891v2…