PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDepth: Universal Monocular Metric Depth Estimation

Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, Fisher Yu

arXiv:2403.18913v1cs.CV

TL;DR

Existing MMDE methods achieve strong accuracy but generalize poorly beyond training domains and often require camera information at test time. UniDepth predicts metric 3D points from a single image using self-prompted camera representations, pseudo-spherical outputs, and geometric invariance. Across ten datasets in zero-shot evaluation, it consistently outperforms competing methods, including methods trained on the test domains.

  • Problem

    Existing MMDE methods have limited out-of-domain generalization and commonly require camera intrinsics, restricting reliable metric estimation across diverse real-world scenes.

  • Method

    UniDepth predicts per-pixel metric 3D points from one image using a dense self-promptable camera module, pseudo-spherical output representation, and geometric invariance loss.

  • Results

    UniDepth consistently sets the state of the art across ten datasets in a zero-shot regime, including the official KITTI Depth Prediction Benchmark.

  • Takeaways & Limitations

    UniDepth provides a universal MMDE approach that operates without external camera or scene information while supporting additional camera information at test time.

Abstract

from arXiv · show

Accurate monocular metric depth estimation (MMDE) is crucial to solving downstream tasks in 3D perception and modeling. However, the remarkable accuracy of recent MMDE methods is confined to their training domains. These methods fail to generalize to unseen domains even in the presence of moderate domain gaps, which hinders their practical applicability. We propose a new model, UniDepth, capable of reconstructing metric 3D scenes from solely single images across domains. Departing from the existing MMDE methods, UniDepth directly predicts metric 3D points from the input image at inference time without any additional information, striving for a universal and flexible MMDE solution. In particular, UniDepth implements a self-promptable camera module predicting dense camera representation to condition depth features. Our model exploits a pseudo-spherical output representation, which disentangles camera and depth representations. In addition, we propose a geometric invariance loss that promotes the invariance of camera-prompted depth features. Thorough evaluations on ten datasets in a zero-shot regime consistently demonstrate the superior performance of UniDepth, even when compared with methods directly trained on the testing domains. Code and models are available at: https://github.com/lpiccinelli-eth/unidepth

1. Introduction

Existing monocular metric depth methods often depend on matched camera intrinsics and scene scales, limiting generalization to uncontrolled domains. UniDepth addresses universal MMDE by predicting metric 3D points from a single image while using camera self-prompting and a disentangled output representation.

  • MMDE methods often require training and testing data with similar camera intrinsics and scene scales.
  • Limited scene and camera diversity in training datasets contributes to poor generalization in uncontrolled environments.
  • Generalizable MMDE methods commonly assume camera intrinsics at test time, limiting in-the-wild use and exposing predictions to camera-parameter noise.
  • UniDepth predicts metric 3D points for each pixel from only a single image, without requiring external scene or camera information.
  • Its camera module produces a dense representation that prompts the depth module, while pseudo-spherical coordinates separate camera angles from radial depth.
  • UniDepth combines camera self-prompting, pseudo-spherical output, geometric invariance loss, and zero-shot evaluation across ten datasets.

2. Related Work

Related MMDE work improves generalization but typically relies on specialized priors, noiseless camera intrinsics, or predefined backprojection. These assumptions limit deployment across diverse camera settings and blur the distinction between depth estimation and full 3D estimation.

  • Scale-agnostic depth methods generalize robustly through extensive training data but do not provide metric outputs for downstream applications.
  • Generalizable MMDE methods use strategies such as geometric pretraining or dataset-specific priors, while assuming noiseless camera intrinsics during training and testing.
  • These methods may be limited to pinhole cameras and rely on predefined backprojection, blurring the distinction between learned depth and 3D scene estimation.

3. UniDepth

UniDepth estimates metric 3D points from a single image by jointly modeling camera geometry and depth in a disentangled pseudo-spherical space. A self-promptable camera module conditions depth prediction, while geometric consistency supports robustness across camera variations and noisy intrinsics.

  • UniDepth predicts metric 3D points from a single image without external camera information, while allowing additional camera information at test time.
  • 3D Representation: The pseudo-spherical output represents each point with azimuth, elevation, and log-depth, separating camera angles from depth.This contrasts with Cartesian coordinates and provides separate camera and depth optimization dimensions.
  • Self-Promptable Camera: Camera prompting supplies scene-scale prior information, reducing the burden of predicting depth scale from scratch and helping avoid scene-scale mode collapse.
  • Self-Promptable Camera: The Camera Module predicts a dense angular representation, embeds it with spherical harmonics, and prompts Depth Module features through cross-attention.The dense camera tensor contains azimuth and elevation values for each image location; the resulting embeddings condition depth features.
  • Geometric Invariance Loss: UniDepth uses bootstrapped camera prediction when intrinsics are unknown or noisy, while externally supplied low-noise camera information can improve peak reconstruction performance.The model therefore operates in both camera-agnostic and externally camera-conditioned regimes.
  • Geometric Invariance Loss: Geometric invariance loss enforces consistency between camera-prompted depth features from geometrically augmented views of the same scene.The training procedure uses two augmentations involving rescaling and horizontal translation, whose predicted camera representations may differ.

4. Experiments

UniDepth is evaluated across diverse zero-shot and standard benchmarks, where it generally outperforms prior methods, while ablations clarify the roles of camera prompting, output representation, and geometric invariance.

  • Comparison with the State of the Art: With self-prompted camera prediction, performance is preserved for most test sets and can exceed using ground-truth cameras on SIlog in ETH3D and IBims-1.For strongly out-of-domain camera types such as ETH3D and HAMMER, bootstrapping adds noise and worsens δ1.
  • Comparison with the State of the Art: 15.5% improvement in SIlog over the second-best method places UniDepth first on the KITTI Depth Prediction Benchmark.On NYU and KITTI, UniDepth also improves δ1 over Metric3D and ZeroDepth by 5.8% and 7.3%, and by 1.1% and 9.4%, respectively.
  • Ablation Study: 95.2% in δ1 on KITTI with the full model falls to 58.9% when the camera module is removed, despite only a 2% difference on nuScenes and DDAD.This ablation indicates that camera information is especially important for some out-of-distribution datasets.
  • Ablation Study: The pseudo-spherical representation improves optimization by separating camera and depth dimensions, while geometric invariance improves in-domain and out-of-domain performance.The geometric invariance loss promotes depth-feature invariance to appearance changes caused by different camera intrinsics and stabilizes training.

5. Conclusion

UniDepth predicts metric 3D points from single images across diverse scenes without external camera information. Its components support camera-free application, disentangled optimization, camera-aware consistency, and strong zero-shot benchmark performance.

  • UniDepth predicts metric 3D points from diverse scenes using only a single input image.
  • The self-prompting camera module enables camera-free test-time application and improves robustness against camera noise.
  • The pseudo-spherical output representation disentangles camera and depth during optimization.
  • The geometric invariance loss enforces camera-aware depth consistency.
  • UniDepth achieves state-of-the-art performance across multiple zero-shot benchmarks, surpassing even methods trained in-domain.

Supplementary Material

The supplementary material adds benchmark results, standard metric evaluations, backbone and representation ablations, and dataset and architecture comparisons.

  • The supplement reports official KITTI results and standard metric evaluations on KITTI and NYU validation sets.
  • It includes additional ViT-backbone ablations and comparisons of alternative pseudo-spherical representations.
  • The supplement discusses generalization differences between convolutional and ViT-based backbones.

A. Results

The supplementary results document UniDepth’s performance on official and standard KITTI and NYU evaluations, including zero-shot comparisons and metric-reporting considerations.

  • Official KITTI benchmark: UniDepth ranks first on the official KITTI private test benchmark among published and unpublished methods at submission time.
  • Official KITTI benchmark: Table 6 compares methods trained on KITTI and evaluated on the official KITTI private test set.
  • KITTI Eigen-split and NYUv2: The supplement notes that δ2 and especially δ3 are saturated, motivating the use of δ0.5 instead in the main paper.
  • KITTI Eigen-split and NYUv2: Tables 7 and 8 report standard metrics on the KITTI Eigen-split test set and NYU validation set, including zero-shot methods.

B.1. Ablations with ViT backbone

ViT ablations show component trends consistent with convolutional backbones but greater out-of-domain variability, while pseudo-spherical alternatives perform similarly and backbone resilience depends on domain-shift type.

  • Ablations with ViT backbone: ViT and ConvNext backbones show similar contributions from the ablated components.
  • Ablations with ViT backbone: ViT exhibits greater out-of-domain variability and a stronger pseudo-spherical representation effect for both Baseline and Full models.
  • Backbone comparison: Zero-shot results indicate convolutional architectures are more resilient to scale-related domain shifts, while ViT performs better under appearance-related shifts on SIlog.
  • Pseudo-spherical representation: UniDepth uses an angular pseudo-spherical representation based on azimuth, elevation angle, and log-depth to disentangle calibration and depth estimation.
  • Alternative pseudo-spherical representation: The alternative ray-and-log-depth representation differs marginally from the angular representation, usually within uncertainty, while using a different output-space dimensionality.
  • Alternative pseudo-spherical representation: Bearing-vector outputs are constrained to the unit-sphere manifold through L2 normalization.

C.1. Datasets details

The paper details dataset preparation, validation handling, and ablations of camera conditioning and pseudo-spherical representation across in-domain and out-of-domain settings.

  • Dataset preparation: 3’743’000 training samples are assembled after subsampling RGB–depth frames so consecutive pairs are at least one second apart.No postprocessing beyond this subsampling is applied.
  • Dataset preparation: NYU test samples are removed from SUN-RGBD validation to prevent overlap between validation and testing data.
  • Validation handling: Diode boundary artifacts are detected and masked because inaccurate near-zero ground-truth depths affect validation results.The corresponding neighborhoods are ignored during validation.
  • Ablations: Table 10 evaluates pseudo-spherical representation ablations using in-domain and out-of-domain unions, with ARelC measuring camera-intrinsic error.Medians and median average deviations are reported over three runs.
  • Ablations: Table 11 compares complete UniDepth against replacing its prompting mechanism with CAMConvs conditioning.
  • Dataset composition: Table 12 lists training and testing datasets by image count, scene type, and acquisition method.The caption defines SfM as Structure-from-Motion and MVS as Multi-View Stereo.

D. Model Complexity

UniDepth’s ViT-L configuration is reported as efficient relative to comparable methods, while its depth module is the main latency contributor because of decoder self-attention.

  • Overall comparison: UniDepth with a ViT-L backbone has efficiency and parameter counts comparable to ZoeDepth while achieving higher performance.
  • Overall comparison: Metric3D is more efficient because its decoder is fully convolutional and relatively low-dimensional.
  • Overall comparison: ZeroDepth’s low efficiency despite using ResNet-18 is attributed to expensive full-resolution cross-attention in its decoder.
  • Module analysis: The Camera Module has 13.4M parameters, whereas the Depth Module accounts for more than half of total latency.
  • Module analysis: Six self-attention layers in the decoder are identified as the cause of the Depth Module’s high latency.The module is described as having limited memory consumption despite its latency.

E. Network Architecture

The architecture extracts multiscale encoder features, predicts dense camera representations, and conditions depth estimation on camera-aware features for zero-shot qualitative evaluation.

  • Encoder: ConvNext and ViT encoders extract feature maps at each layer, with each scale formed by pixel-wise averaging.The backbones are originally classification models, with modifications described after the supplied passage.
  • Encoder: The encoder uses independent projections for feature maps and class tokens across four scales, with C set to 512.Class tokens are fed only to the Camera Module.
  • Camera Module: Camera parameters are initialized from four encoder class tokens, while flattened encoder features provide keys and values for cross-attention.
  • Camera Module: The dense camera representation is obtained by backprojecting predicted camera parameters and embedding azimuth and elevation angles with spherical harmonics.The embedding produces 81 channels.
  • Depth Module: Depth latents are initialized by averaging encoder features across scales and then conditioned on encoder features through cross-attention.The depth pathway also applies an MLP before producing its output tensor.
  • Depth Module: The final log-depth tensor is upsampled to the input resolution, and depth is obtained by element-wise exponentiation.
  • Qualitative evaluation: The qualitative comparisons cover KITTI, NYU, Diode, ETH3D, DDAD, NuScenes, SUN-RGBD, and IBims-1 among the listed zero-shot test sets.
  • Qualitative evaluation: Figures 5–7 show zero-shot samples using RGB images, absolute relative error maps, ground-truth depth, and predicted depth.The final columns specify depth and error colormap ranges; sparse DDAD and NuScenes maps are dilated fivefold for visibility.
Loading 2403.18913v1…