PaperlayerView exact revision on arXiv

Source-linked AI summary

ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller

arXiv:2302.12288v1cs.CV

TL;DR

Single-image depth estimation must balance metric scale with generalization across datasets and environments. ZoeDepth combines relative-depth pre-training with lightweight domain-specific metric heads and automatic routing, achieving strong NYU Depth v2 results and zero-shot transfer across unseen indoor and outdoor datasets.

  • Problem

    Metric-depth models often overfit specific datasets, while relative-depth models generalize better but lack metric meaning across varied environments.

  • Method

    ZoeDepth pre-trains a shared encoder-decoder for relative depth, then fine-tunes lightweight domain-specific metric heads with automatic routing for metric prediction.

  • Results

    ZoeDepth improves NYU Depth v2 SOTA by 21% in REL and achieves zero-shot improvements of up to 976.4% on unseen metric datasets.

  • Takeaways & Limitations

    The framework bridges relative and metric depth estimation while improving metric depth performance and zero-shot transfer across indoor and outdoor domains.

  • Takeaways & Limitations

    Metric fine-tuning across datasets with different cameras, settings, and scene-scale ranges remains difficult, motivating separate metric bins modules for scene types.

Abstract

from arXiv · show

This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains. The code and pre-trained models are publicly available at https://github.com/isl-org/ZoeDepth .

1. Introduction

ZoeDepth combines relative-depth generalization with metric-scale prediction through staged pre-training, domain-specific metric heads, and automatic routing. The framework improves metric depth results on NYU Depth v2 and supports zero-shot transfer across indoor and outdoor datasets.

  • Motivation: Metric depth models provide absolute physical units but often overfit specific datasets and generalize poorly across differing depth scales.Relative-depth models generalize across diverse scenes by factoring out scale, but their predictions lack metric meaning.
  • Approach: ZoeDepth first pre-trains a shared encoder-decoder for relative depth, then fine-tunes lightweight domain-specific metric heads and routes images automatically using encoder features.Each metric head has less than 1% of the backbone's parameters, and the metric bins module estimates depth bins that are transformed into metric depth.
  • Contributions: The framework offers multiple configurations for improving metric single-image depth estimation across different categories.The paper identifies three configurations as its main contributions.
  • Results: 13.7% improvement over NeWCRFs on indoor depth estimation is achieved by ZoeD-X-N without relative-depth pre-training.This configuration validates the metric bins module using only NYU Depth v2 training.
  • Results: 21% improvement over current published SOTA is achieved by ZoeD-M12-N after relative pre-training on 12 datasets and metric fine-tuning on NYU Depth v2.ZoeD-M12-N improves on ZoeD-X-N by 8.5%.
  • Results: 24.3% overall REL improvement is achieved on jointly trained NYU Depth v2 and KITTI, while unseen metric datasets show up to 976.4% improvement.The jointly trained ZoeD-M12-NK also demonstrates zero-shot performance on seven metric datasets not used for training.

2. Related Work

Related work separates monocular depth estimation into metric and relative approaches, while newer methods seek to recover metric information or reason over depth distributions. These directions address generalization, scale, and structured depth prediction through different architectural strategies.

  • Metric and relative depth: Metric depth methods predict absolute depth but commonly train on single datasets, overfit, and generalize poorly across unseen environments or varying depth ranges.Relative-depth methods generally generalize better because they can use more diverse datasets, but they do not directly preserve metric scale.
  • Recovering metric information: Two-stage methods combine monocular depth estimation with 3D point-cloud encoders or decompose metric depth into normalized depth and scale features.The latter approach uses a multidecoder network whose metric decoder leverages relative-depth features.
  • Distribution-based prediction: Depth estimation is increasingly formulated as classification-regression over depth-value distributions rather than only per-pixel regression.AdaBins is cited as extending encoder-decoder backbones with a transformer-based module.

3. Methodology

The method combines relative-depth pre-training with metric-depth fine-tuning through lightweight metric heads and a metric bins module that refines ordered bin centers using attractors. It supports multi-dataset training and routing across metric heads.

  • Metric bins module: The metric bins module predicts all bin centers at the bottleneck and adjusts them through decoder layers instead of progressively splitting bins.This design uses multi-scale decoder features to refine the metric depth representation.
  • Metric bins module: Attractor layers predict points on the depth interval and move bin centers toward them during multi-scale refinement.The inverse attractor uses hyperparameters α and γ to determine attractor strength.
  • Metric bins module: The implementation uses 64 total bins with decoder-layer attractor counts of {16, 8, 4, 1}.The authors report that the inverse attractor performs better than the exponential variant.
  • Depth prediction: Final metric depth is a linear combination of bin centers weighted by probability scores, using a log-binomial formulation rather than softmax over ordered bins.The resulting scores are normalized with temperature-scaled softmax for numerical stability before depth prediction.
  • Training strategies: Training compares relative-depth pre-training choices and single- versus multiple-head fine-tuning for single- and multi-dataset metric depth.Mixed indoor and outdoor metric training is difficult because cameras and scene-scale ranges differ substantially.

4. Experimental Setup

The experiments use NYU Depth v2 and KITTI as primary indoor and outdoor datasets, with a 12-dataset mixture for relative-depth pre-training and eight unseen datasets for zero-shot evaluation. Metric performance is measured with standard depth errors and threshold accuracies under capped depth ranges.

  • Datasets: The primary metric-depth datasets are NYU Depth v2 for indoor scenes and KITTI for outdoor scenes.The joint configuration uses separate heads fine-tuned on both datasets.
  • Datasets: Relative-depth pre-training uses 12 datasets: the 10-dataset mixture from prior work plus KITTI and NYU Depth v2.The listed mixture includes indoor, outdoor, synthetic, and movie-derived datasets.
  • Datasets: Zero-shot evaluation covers eight unseen datasets, split across four indoor and four outdoor benchmarks.The indoor set is SUN RGB-D, iBims, DIODE Indoor, and HyperSim; the outdoor set is DDAD, DIML Outdoor, DIODE Outdoor, and Virtual KITTI 2.
  • Models: Models follow the naming scheme ZoeD-{RDPT}-{MFT}, where RDPT identifies relative-depth pre-training datasets and MFT identifies metric fine-tuning datasets.The evaluated configurations include no-pre-training, M12 pre-training, and joint NYU Depth v2/KITTI fine-tuning.
  • Evaluation metrics: Metric evaluation reports absolute relative error, log10 error, and threshold accuracy δn for n = 1, 2, 3.Ground-truth and predicted depths are compared per pixel, with indoor depths capped at 10m and outdoor depths at 80m.

5. Results

ZoeDepth improves metric depth estimation on NYU Depth v2, supports joint indoor–outdoor training, and transfers strongly to unseen datasets. Ablations show that backbone size, metric-bin design, routing, and log-binomial probabilities materially affect performance.

  • Comparison to SOTA on NYU Depth V2: 13.7% improvement over NeWCRFs (REL = 0.082) is achieved on NYU Depth v2 without relative-depth pre-training.ZoeD-X-N uses standard ImageNet initialization, demonstrating the architecture’s standalone contribution.
  • Universal Metric SIDE: 25.2% mean relative improvement over NeWCRFs is obtained by the two-head ZoeD-M12-NK model when trained jointly on NYU and KITTI.The two-head model reduces the NYU performance gap from 8% with a shared head to 2.6%.
  • Zero-shot Generalization: 7 out of 8 unseen indoor and outdoor datasets are led by ZoeD-M12-NK in zero-shot transfer, with indoor mRIθ ranging from 5.3% to 46.3%.The model is evaluated without fine-tuning on four unseen indoor and four unseen outdoor datasets.
  • Ablation Studies: Larger backbones improve performance, while the model still outperforms prior state of the art with the same backbone.Backbone image-classification performance is highly correlated with lower depth-estimation REL.
  • Ablation Studies: The best attractor metric-bin variant performs about 21% better than the naive head and 11.7% better than the LocalBins splitter variant.Using log binomial instead of softmax changes REL from 0.077 to 0.075, an improvement of about 2%.

6. Conclusion

ZoeDepth bridges relative and metric depth estimation through relative-depth pre-training and domain-specific metric fine-tuning. It improves NYU Depth v2 performance and zero-shot transfer, while identifying finer domains and additional metric datasets as future directions.

  • ZoeDepth pre-trains an encoder-decoder with relative depth, then adds domain-specific metric-bins heads for metric-depth fine-tuning.The heads are fine-tuned on one or more metric datasets.
  • 21% in REL improvement over state-of-the-art is reported on NYU Depth v2.
  • ZoeDepth significantly improves state-of-the-art zero-shot transfer across unseen datasets.
  • More granular domains and additional metric fine-tuning datasets are proposed as ways to improve results further.The paper also identifies mobile and stereo-image extensions as future work.

A.1. Datasets Overview

The evaluation uses NYU Depth v2 and KITTI for metric fine-tuning, plus four indoor and four outdoor unseen datasets for generalization testing. Models are evaluated after resizing inputs to their training resolutions and predictions back to ground-truth resolution.

  • NYU Depth v2 and KITTI provide metric-depth fine-tuning data, while four indoor and four outdoor datasets test generalization.
  • Models use training resolutions of 384 × 512 for Zoe-*-N and Zoe-*-NK, and 384 × 768 for Zoe-*-K.Predictions are resized to the original ground-truth resolution before evaluation.

A.2. Training Details

ZoeDepth configurations vary by relative-depth pre-training datasets, metric fine-tuning datasets, and the number of metric heads. Each combination defines a distinct model.

  • ZoeDepth training strategies vary the relative pre-training datasets, metric fine-tuning datasets, and number of metric heads.
  • Each combination of these training options defines a different ZoeDepth model.Results for the combinations are reported in Appendix A.3.

A.3. Detailed Results

The detailed results supplement zero-shot evaluations with additional metrics and qualitative depth-map examples. Across indoor and outdoor examples, ZoeDepth predictions are generally closer to ground truth, sharper, and advantageous relative to NeWCRF, with a noted exception.

  • Zero-shot transfer results on four unseen indoor and outdoor datasets are supplemented with δ2, δ3, and average log10 error.Additional ZoeDepth configurations are also included in the appendix tables.
  • Figures 6 to 13 provide metric depth maps for example images from the evaluated indoor and outdoor datasets.
  • Zoe-M12-NK and Zoe-M12-N are generally closer to indoor ground truth than NeWCRF, which tends to underestimate depth.
  • ZoeD-M12-NK shows an outdoor advantage over NeWCRF, while improved prediction sharpness persists across all eight indoor and outdoor datasets.Its joint fine-tuning on KITTI and NYU Depth v2 is noted in relation to low depth values in Figure 12.

A.4. ZoeDepth with different backbones

ZoeDepth’s performance and parameter count depend primarily on the MiDaS encoder backbone, while its modular architecture permits backbone replacement. The appendix also documents the datasets and zero-shot evaluations used to compare model variants.

  • Backbone comparison: 345M parameters characterize the best-performing ZoeDepth configuration with the BEiT384-L backbone, including 344M in MiDaS.The BEiT384-L backbone accounts for 305M of MiDaS’s 344M parameters.
  • Backbone comparison: 112M parameters result when ZoeDepth replaces BEiT384-L with the base BEiT384-B transformer.The architecture also supports alternative backbones such as Swin2.
  • Evaluation setup: Zero-shot evaluation covers 13165 indoor samples and 6597 outdoor samples across the listed datasets.HyperSim evaluation permits depths up to 80 m because some samples exceed 10 m; specified crops are used for comparison with prior work.
  • Backbone comparison: ZoeDepth parameter count varies with the backbone, not with the relative-depth pre-training or metric-depth fine-tuning dataset combination.This applies across variants including ZoeD-X-N, ZoeD-M12-N, and ZoeD-M12-NK.
Loading 2302.12288v1…