PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Zero-Shot Scale-Aware Monocular Depth Estimation

Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, Adrien Gaidon

arXiv:2306.17253v1cs.CVcs.LG

TL;DR

Monocular depth estimation needs metric scale, but learned scales are tied to camera geometry and relative-depth systems still require test-time scaling. ZeroDepth uses geometric embeddings and a conditioned variational latent representation to transfer metric depth across domains, achieving state-of-the-art results across outdoor and indoor benchmarks with one pretrained model.

  • Problem

    Learned metric scales are geometry-specific, while relative-depth methods sacrifice scale and require groundtruth-based test-time alignment for metric predictions.

  • Method

    ZeroDepth combines input-level geometric embeddings with a decoupled encoder-decoder architecture using a conditioned variational latent representation.

  • Results

    ZeroDepth achieves state-of-the-art zero-shot transfer on outdoor and indoor benchmarks using the same pretrained model, outperforming methods requiring in-domain training or test-time scale alignment.

  • Takeaways & Limitations

    Metric depth can be transferred across datasets with different camera geometries through scale priors anchored in image and geometric information.

Abstract

from arXiv · show

Monocular depth estimation is scale-ambiguous, and thus requires scale supervision to produce metric predictions. Even so, the resulting models will be geometry-specific, with learned scales that cannot be directly transferred across domains. Because of that, recent works focus instead on relative depth, eschewing scale in favor of improved up-to-scale zero-shot transfer. In this work we introduce ZeroDepth, a novel monocular depth estimation framework capable of predicting metric scale for arbitrary test images from different domains and camera parameters. This is achieved by (i) the use of input-level geometric embeddings that enable the network to learn a scale prior over objects; and (ii) decoupling the encoder and decoder stages, via a variational latent representation that is conditioned on single frame information. We evaluated ZeroDepth targeting both outdoor (KITTI, DDAD, nuScenes) and indoor (NYUv2) benchmarks, and achieved a new state-of-the-art in both settings using the same pre-trained model, outperforming methods that train on in-domain data and require test-time scaling to produce metric estimates.

1. Introduction

ZeroDepth addresses the geometric domain gap that prevents metric depth scales from transferring across cameras and datasets. It combines geometric embeddings, a variational latent representation, and encoder-level augmentation to achieve zero-shot metric depth transfer across indoor and outdoor benchmarks.

  • Problem: Metric depth models learn camera-specific scales that fail to transfer across datasets because training cameras differ geometrically.Relative-depth methods improve unscaled transfer but still require groundtruth information at test time for metric scaling.
  • Method: Input-level geometric embeddings jointly encode camera parameters and image features, enabling scale priors based on objects’ physical size.The approach is designed to address the geometric rather than only appearance domain gap.
  • Method: A learned variational global latent representation decouples encoding from decoding and supports multiple probabilistic depth predictions after conditioning on a single frame.The latent representation can be sampled and decoded to generate predictions.
  • Method: Encoder-level data augmentation improves robustness to both appearance and geometric domain gaps.These augmentations are part of the framework’s proposed contributions.
  • Results: ZeroDepth achieves state-of-the-art zero-shot transfer on outdoor KITTI, DDAD, and nuScenes and indoor NYUv2, outperforming methods requiring in-domain images or test-time scale alignment.The same framework is evaluated across both settings.

2. Related Work

Prior monocular depth methods address scale ambiguity through metric cues, zero-shot transfer, or test-time alignment, but camera geometry remains a central challenge. ZeroDepth instead targets direct metric transfer across domains without adaptive range prediction.

  • Monocular Depth Estimation: Self-supervised monocular depth is scale-ambiguous, so predictions are commonly median-scaled against groundtruth at test time, reducing practicality.Supervised approaches produce metric predictions but depend on costly range-sensor data.
  • Metric Scale Supervision: Metric cues from velocity, camera height, extrinsics, and surrounding views have been used to inject scale information into self-supervised depth estimation.These approaches rely on additional measurements or multi-view information.
  • Zero-Shot Transfer: Zero-shot methods use diverse multi-source training data and geometry encoding to resist camera-geometry shifts when evaluating without fine-tuning.This focus arose because in-domain training can overfit to camera geometry.
  • Concurrent Work: ZoeDepth predicts domain-specific adaptive ranges and can specialize to its training domains and camera geometries, whereas ZeroDepth directly decodes metric depth without domain-specific bounds.ZeroDepth is therefore not bounded or conditioned to a specific domain in the described comparison.

3. Zero-Shot Scale-Aware Monocular Depth

ZeroDepth combines camera-aware geometric embeddings with a decoupled variational encoder–decoder to transfer metric depth across camera geometries and datasets. Its training uses geometric augmentation and multiple regularized losses, while probabilistic decoding provides uncertainty estimates.

  • 3.1. Perceiver IO Overview: ZeroDepth conditions a variational latent representation on image and geometric embeddings, then decodes predictions using only geometric embeddings.This decouples encoding from decoding and permits predictions from embeddings that were not encoded.
  • 3.3. Input-Level Embeddings: Camera-aware geometric embeddings encode viewing directions from pixel coordinates and intrinsics, enabling features that reason about object shape and physical properties.The embeddings are Fourier-encoded, and camera centers or poses are unnecessary in the single-frame setting.
  • 3.5. Encoder-Level Data Augmentation: Encoder-level augmentation resizes images with adjusted intrinsics, jitters pixel rays, and discards embeddings to improve coverage and robustness to geometric variation.Resizing preserves 3D scene structure and metric scale, while ray perturbation expands the distribution of viewing directions.
  • 3.4. Variational Latent Representation: The variational latent representation samples multiple decoded depth maps whose per-pixel mean and standard deviation approximate depth uncertainty.High-uncertainty pixels can be selectively removed to improve performance.
  • 3.6. Training Losses: Training combines depth supervision, surface-normal regularization, and KL divergence, with smooth L1 depth loss and additional normal-based geometric constraints.The KL term promotes learning a Gaussian distribution for sampling during decoding.
  • 3.7. Results: Using one pretrained model, ZeroDepth qualitatively transfers metric depth across datasets with different camera geometries and depth ranges.The predictions are shown as colored monocular point clouds overlaid with ground-truth point clouds represented as height maps.

4. Experiments

ZeroDepth is evaluated with one model trained on mixed indoor and outdoor data, then tested without fine-tuning across multiple benchmarks. It achieves strong zero-shot metric transfer, while ablations examine dataset composition, latent complexity, uncertainty, and fine-tuning.

  • Experimental Setup: The model is trained on combined synthetic and real-world outdoor data plus Omnidata indoor data, then evaluated without fine-tuning using each benchmark’s standard protocol.Training uses 3,159,424 outdoor samples before repetition and 14,340,580 indoor samples.
  • Scale-Aware Monocular Depth Estimation: ZeroDepth outperforms published methods across KITTI, DDAD, nuScenes, and NYUv2 without seeing target data, including methods using in-domain supervision or test-time scaling.The same pretrained model transfers metric depth across outdoor and indoor settings.
  • Ablative Analysis: Reducing latent representation capacity steadily harms performance, with latent dimension affecting metric results more sharply than latent-vector count.At Dl = 32, metric RMSE reaches 6.421 versus 4.904 for median-scaled results; Nl = 32 still gives RMSE 4.881, close to monodepth2’s 4.863.
  • Ablative Analysis: Removing any outdoor training dataset degrades performance, with LSD and Waymo having the largest overall impact because of their size and reduced appearance gap.The supplementary results report consistent degradation across evaluation datasets as fewer training datasets are used.
  • Ablative Analysis: Filtering predictions by lower estimated standard deviation improves depth performance, while increasing the number of samples helps until saturation at roughly 50% valid pixels.The uncertainty analysis uses multiple variational samples and retains pixels with lower standard deviation.
  • Fine-Tuned Depth Estimation: Fine-tuning the pretrained model on each evaluation dataset significantly improves metric depth and surpasses methods trained specifically for those datasets.Fine-tuning uses five additional epochs on each dataset’s training split.

5. Conclusion

The conclusion presents ZeroDepth as a framework for transferring metric scale across datasets through geometric embeddings, a variational latent representation, and encoder-level augmentation. A single pretrained model achieves state-of-the-art results indoors and outdoors.

  • 5. Conclusion: ZeroDepth transfers metric scale across datasets by learning scale priors from image and geometric embeddings.The framework targets datasets collected with different cameras and locations.
  • 5. Conclusion: Its conditioned variational latent representation can be sampled and decoded into multiple probabilistic depth maps.The model conditions a global latent representation using a single frame during encoding.
  • 5. Conclusion: The same pretrained model achieves state-of-the-art results across indoor and outdoor benchmarks, outperforming methods using in-domain self-supervision or test-time median-scaling.The conclusion also highlights encoder-level augmentations for appearance and geometric domain gaps.

A. Training Details

Training uses distributed AdamW optimization with warmup and scheduled decay, together with resolution jittering and embedding dropout. Evaluation omits augmentation.

  • A. Training Details: Models are trained with AdamW across 8 A100 GPUs using TensorFloat-32 precision, batch size 16 per GPU, and an initial learning rate of 10^-4.The optimizer uses β1 = 0.9, β2 = 0.999, and weight decay 10^-4.
  • A. Training Details: The learning rate warms from 10^-5 during the first epoch and then decays by 0.8 every 5 epochs outdoors and every 2 epochs indoors.
  • A. Training Details: Training applies independent height and width resolution jittering from 25% to 150% and randomly drops 0%–50% of encoder embeddings.Sampled resolutions are rounded to multiples of 32.
  • A. Training Details: No data augmentation is applied during evaluation.

B. Network Architecture

The architecture uses a ResNet18 encoder, Fourier-style geometric embeddings, and a large self-attention latent representation conditioned through attention layers.

  • B. Network Architecture: A ResNet18 backbone produces 960-dimensional image embeddings, while geometric embeddings use 16 frequency bands and maximum resolution 64 to form 51-dimensional vectors.
  • B. Network Architecture: The latent representation has dimensionality 1024 × 1024 and is conditioned with 8 self-attention heads across 8 layers.The conditioning stack uses GeLU activations and dropout of 0.1.
  • B. Network Architecture: The architecture uses one cross-attention layer for conditioning and another single cross-attention stage in the decoder.

C. Extended Depth Estimation Tables

Extended evaluations cover individual cameras and training-dataset ablations, showing that broader training data consistently improves performance across evaluation datasets.

  • The extended tables report per-camera DDAD and nuScenes results produced by ZeroDepth’s outdoor variant.
  • Fewer training datasets consistently degrade performance across all evaluation datasets for both metric and median-scaled predictions.
  • Performance improvements correlate with the number of training tokens available in each dataset.At 384 × 640 resolution with encoding downsample ratio 4, datasets contain approximately 8.5B to 1.6T tokens.

D. Variational Uncertainty Sampling

Variational uncertainty identifies unreliable depth predictions and enables confidence-based pixel selection to improve reconstruction accuracy while preserving overall pointcloud structure.

  • Removing 10% of valid depth pixels improves RMSE by 30%, from 4.044 to 2.859, mainly by removing bleeding artifacts.
  • Uncertainty increases at longer ranges and near sudden depth discontinuities, where predictions are commonly smoothed across object boundaries.
  • Removing 50% of valid depth pixels improves RMSE by 63% relative to the full pointcloud while preserving cars, ground plane, and walls.

E. Full Surround Pointclouds

ZeroDepth supports full-surround pointcloud reconstruction on DDAD and nuScenes by combining predictions from each dataset’s multiple cameras.

  • Multiple cameras per DDAD and nuScenes sample enable full-surround pointclouds assembled from individual-camera reconstructions.
  • ZeroDepth pointcloud examples for both datasets are formed by overlaying the individual camera pointclouds.

F. Additional Ablative Analysis

Ablations show that geometric embeddings and an intermediate latent representation are important for transferring metric scale across datasets, beyond the choice of backbone.

  • Network Architecture: Monodepth2 models struggle with both median-scaled and metric predictions regardless of network complexity.
  • Network Architecture: ZeroDepth achieves 0.100 AbsRel metric error versus 0.144 for a DPT variant with geometric embeddings, supporting the benefit of its intermediate latent representation.
  • Network Architecture: Adding geometric embeddings to DPT improves metric performance from 0.240 to 0.144 AbsRel and median-scaled performance from 0.126 to 0.119 AbsRel.
  • Design Choices: Replacing ZeroDepth’s 3D geometric embeddings with 2D positional embeddings causes a large degradation in metric performance.Even with this replacement, ZeroDepth achieves 0.175 versus DPT’s 0.240 AbsRel.
Loading 2306.17253v1…