PaperlayerView exact revision on arXiv

Source-linked AI summary

UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler

Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, Luc Van Gool

arXiv:2502.20110v2cs.CV

TL;DR

Existing monocular metric depth estimators generalize poorly beyond their training domains and often depend on camera information. UniDepthV2 predicts metric 3D points from a single image using camera self-prompting, pseudo-spherical representations, and edge-aware training, achieving strong zero-shot generalization across ten datasets while improving efficiency. Its practical scope remains constrained by failures on mirrors, unrealistic paintings, distorted geometry, and non-pinhole projections.

  • Problem

    Existing MMDE methods generalize poorly across domain gaps and commonly rely on camera intrinsics, while imperfect edge ground truth causes blurry geometric details.

  • Method

    UniDepthV2 predicts metric 3D points from one image using a dense self-promptable camera module, pseudo-spherical output representation, geometric invariance, and edge-guided losses.

  • Results

    UniDepthV2 achieves state-of-the-art performance with improved computational efficiency in extensive zero-shot and fine-tuning evaluations.

  • Takeaways & Limitations

    Variable test-time input resolution enables a trade-off between inference speed and detail preservation while maintaining global scale consistency.

  • Takeaways & Limitations

    UniDepthV2 struggles with mirrors, unrealistic paintings, distorted geometry, and non-pinhole projections under challenging conditions.

Abstract

from arXiv · show

Accurate monocular metric depth estimation (MMDE) is crucial to solving downstream tasks in 3D perception and modeling. However, the remarkable accuracy of recent MMDE methods is confined to their training domains. These methods fail to generalize to unseen domains even in the presence of moderate domain gaps, which hinders their practical applicability. We propose a new model, UniDepthV2, capable of reconstructing metric 3D scenes from solely single images across domains. Departing from the existing MMDE paradigm, UniDepthV2 directly predicts metric 3D points from the input image at inference time without any additional information, striving for a universal and flexible MMDE solution. In particular, UniDepthV2 implements a self-promptable camera module predicting a dense camera representation to condition depth features. Our model exploits a pseudo-spherical output representation, which disentangles the camera and depth representations. In addition, we propose a geometric invariance loss that promotes the invariance of camera-prompted depth features. UniDepthV2 improves its predecessor UniDepth model via a new edge-guided loss which enhances the localization and sharpness of edges in the metric depth outputs, a revisited, simplified and more efficient architectural design, and an additional uncertainty-level output which enables downstream tasks requiring confidence. Thorough evaluations on ten depth datasets in a zero-shot regime consistently demonstrate the superior performance and generalization of UniDepthV2. Code and models are available at https://github.com/lpiccinelli-eth/UniDepth

I. INTRODUCTION

UniDepthV2 targets universal monocular metric depth estimation from a single image, without external camera information. Its camera-prompted, pseudo-spherical design and edge-aware training aim to separate camera and depth estimation while improving generalization and depth-edge sharpness.

  • Existing MMDE methods often require similar camera intrinsics and scene scales during training and testing, limiting real-world generalization.
  • Imperfect, sparse ground-truth depth near edges produces blurry predictions and inaccurate fine-grained geometric details.
  • UniDepthV2 predicts metric 3D points for every pixel from a single image without requiring camera parameters or other external information.
  • A dense camera representation prompts the depth module, while pseudo-spherical azimuth, elevation, and radial-depth components disentangle camera and depth dimensions.
  • The revised architecture improves efficiency and accuracy, while the edge-guided scale-shift-invariant loss preserves local depth structure and sharpens discontinuities.

II. RELATED WORK

Related work distinguishes metric depth estimation from scale-agnostic depth and surveys efforts toward generalizable monocular metric prediction. UniDepthV2 belongs to a newer camera-free paradigm while complementing robustness methods for challenging image-formation conditions.

  • Scale-agnostic methods generalize across dense prediction settings but lack metric outputs needed for practical downstream use.
  • Conventional MMDE achieves strong benchmark performance but faces substantial degradation in zero-shot scenarios and domain gaps.
  • Generalizable MMDE methods commonly incorporate camera awareness through external intrinsics or intrinsic-based normalization.
  • UniDepthV2 directly estimates the 3D scene from RGB input without additional information, unlike approaches relying on camera parameters.
  • Methods for adverse lighting and non-Lambertian surfaces address challenging conditions, while UniDepthV2 pursues camera-free universal MMDE across such domains.

III. UNIDEPTHV2

UniDepthV2 estimates metric 3D points from a single image without external camera information by disentangling camera and depth through a pseudo-spherical representation and camera-conditioned depth features.

  • UniDepthV2 targets generalizable MMDE by estimating 3D points without camera intrinsics or other external information.
  • A. 3D Representation: Its pseudo-spherical output uses azimuth, elevation, and log-depth to decouple camera angles from depth.
  • A. 3D Representation: The dense camera representation is a tensor C ∈ R^H×W×2 whose final dimension stores azimuth and elevation values.
  • B. Self-Promptable Camera: The camera module internally uses pinhole parameters and residuals to produce the dense angular representation.
  • B. Self-Promptable Camera: A self-promptable camera module converts backprojected rays into sine-encoded embeddings that condition the depth module.

C. Geometric Invariance Loss

The geometric invariance loss makes camera-prompted depth outputs consistent across geometrically augmented views of the same scene while accounting for their differing camera representations.

  • The loss enforces consistency between camera-prompted depth features corresponding to identical 3D locations across acquisition sensors.
  • Training creates two augmented views using random rescaling, translation, and cropping to simulate different cameras observing the same scene.
  • Because augmentations produce different camera representations, the consistency loss compares transformed depth outputs rather than raw camera prompts.
  • The bidirectional loss uses stop-gradient on one decoded depth output as pseudo ground truth and applies consistency to camera-aware outputs.

D. Edge-Guided Normalized Loss

LEG-SSI improves local depth precision by focusing normalized supervision on high-gradient image patches, encouraging true geometric boundaries while discouraging texture-induced discontinuities.

  • LEG-SSI samples patches from the top 5%-quantile RGB-gradient regions to emphasize likely depth discontinuities.Patch sizes are sampled between 4% and 8% of the input image’s smallest dimension.
  • The loss oversamples thin high-frequency boundaries and avoids superpixels because region uniformity can suppress boundary detail.
  • Predicted and ground-truth depth are independently normalized per patch so supervision measures local shape consistency rather than absolute depth scale.
  • LEG-SSI penalizes both missed depth discontinuities and hallucinated discontinuities caused by appearance changes without geometric edges.
  • A custom CUDA kernel accelerates the random-patch loss computation by 20x.

E. Network Design

UniDepthV2 combines a ViT encoder, camera-prompted multiscale depth decoding, and modular 3D output construction with a composite optimization objective.

  • Architecture: The architecture contains a ViT-based encoder, a Camera Module, and a Depth Module.
  • Architecture: The Camera Module initializes four tokens, processes them with self-attention, and embeds the resulting pinhole representation.
  • Architecture: Four encoder feature maps are separately conditioned on camera prompts and decoded with an FPN-style multiscale decoder.
  • Architecture: The final 3D output concatenates predicted rays C with exponentiated depth Z, and the ray bundle can alternatively be supplied by another camera model.
  • Optimization: The output-space objective uses weighted variance and mean errors over azimuth, elevation, and log-depth dimensions.
  • Optimization: The final loss combines weighted MSE, geometric consistency, edge-guided scale-shift invariant, and uncertainty terms with weights (0.1, 1.0, 0.1).

A. Experimental Setup

UniDepthV2 is trained on diverse data and evaluated zero-shot across indoor, outdoor, and challenging domains using consistent depth, 3D, camera, and uncertainty metrics.

  • Data: Training combines 23 public datasets totaling 16M images, while evaluation uses ten unseen datasets spanning indoor, outdoor, and challenging domains.The challenging group includes underwater and transparent-object settings.
  • Evaluation protocol: All methods are re-evaluated with the same weights and pipeline without test-time augmentation or in-domain fine-tuning.Zero-shot checkpoints are used rather than models fine-tuned on KITTI or NYU.
  • Metrics: δ1 measures depth estimation, FA measures 3D estimation accuracy, and ρA measures camera performance through angular-error AUC.ρA integrates average camera-ray angular error up to 15°.
  • Metrics: Confidence quality is assessed with AUSE, normalized AUSE, and Spearman’s ρ by comparing uncertainty rankings with prediction errors.nAUSE equal to 1 corresponds to random ranking, while oracle AUSE is 0.
  • Implementation: Training uses AdamW, cosine scheduling, 300k optimization iterations, batch size 128, and geometric and photometric augmentation.Image ratios are randomly sampled between 2:1 and 1:2; training takes six days on 16 NVIDIA 4090 GPUs.

B. Comparison with The State of The Art

Across ten zero-shot datasets, UniDepthV2 generally matches or outperforms baselines, with strong 3D accuracy and improved efficiency. It is also robust to image shape and adapts effectively after fine-tuning, while camera estimation remains a challenge.

  • Zero-shot comparison: UniDepthV2 performs better than or at least on par with all baselines across ten zero-shot validation sets, including methods requiring ground-truth cameras.The evaluation covers indoor, outdoor, and challenging domains.
  • Efficiency: UniDepthV2 is among the fastest and most efficient models despite including an additional camera-prediction module.The model outperforms UniDepth in nearly all reported cases except ρA on IBims-1, DDAD, and NuScenes.
  • Fine-tuning: Fine-tuning on NYU and KITTI shows that UniDepthV2 adapts effectively and outperforms similarly sized models pretrained on large, diverse datasets.The comparisons follow standard validation protocols for each target domain.
  • Robustness: UniDepthV2 is almost invariant to image shape, allowing resolution to trade off against speed without sacrificing accuracy.Figure 4 compares normalized δ1 across varying image shapes.
  • Efficiency: FPS decreases roughly inversely with resolution and peak GPU memory grows near-quadratically; UniDepthV2 is around 2× faster than baselines up to approximately 2 MP.Beyond approximately 5 MP, all methods become memory-bound and reach sub-FPS throughput.
  • Edge evaluation: UniDepthV2-Large achieves the best result on Sintel and remains competitive on ETH3D and IBims-1, outperforming the original UniDepth.The edge evaluation follows the scale-invariant boundary F1 protocol.

C. Ablation Studies

Ablations show that UniDepthV2’s architectural, loss, representation, conditioning, and confidence components jointly improve accuracy, efficiency, and uncertainty estimation. The experiments also examine qualitative edge refinement and challenging-domain behavior.

  • Ablation scope: The ablation suite evaluates architectural changes, losses, camera conditioning, output representation, efficiency, uncertainty, and qualitative edge predictions.Tables IX–XII and VII, together with Fig. 6, isolate these design choices and evaluation dimensions.
  • Architecture: The final multi-resolution architecture significantly reduces computational cost while preserving overall performance.It combines ResNet decoder blocks with FPN-style multi-resolution feature fusion; the final row adds the proposed losses.
  • Losses: 4.7% improvement in δ1 and 1.8% improvement in FA result from introducing the LEG−SSI loss.The loss also improves ρA and visibly refines depth discontinuities in qualitative comparisons.
  • Conditioning: Camera conditioning has a stronger positive effect for UniDepthV2 than for UniDepth under the ablation settings.The comparison removes LEG−SSI and deep-feature invariance to isolate conditioning impact.
  • Representation: The pseudo-spherical representation benefits depth-specific metrics while also substantially improving 3D and camera accuracy.This comparison is made under otherwise matched camera settings against direct Cartesian regression.
  • Confidence: Zero-shot uncertainty remains informative but becomes less calibrated under domain shift, with nAUSE rising to 0.54–0.65 and ρ falling to 0.29.In-domain nAUSE is 0.199–0.221 and ρ is 0.68–0.74; the output supports reliability-aware masking despite degraded calibration.

D. Failure Cases and Limitations

UniDepthV2 is tested on internet images containing mirrors, paintings, optical illusions, and non-pinhole projections. It handles some unusual cases but remains challenged by ambiguous mirrors, distorted geometry, and non-pinhole cameras.

  • Test conditions: The failure-case evaluation covers mirrors, paintings, optical illusions, and non-pinhole projections collected from the internet.These examples represent unusual conditions for depth models.
  • Observed successes: UniDepthV2 can resolve certain human-related optical illusions and recognize paintings when frames and contextual objects are visible.The passage gives “Liberty Leading the People” as an example.
  • Observed failures: Mirrors remain particularly challenging because insufficient context can make them appear as cavities while inducing strong deformation.The model’s interpretation is therefore ambiguous in these cases.
  • Observed failures: Non-realistic paintings and distorted geometry remain difficult, although the model can still separate foreground from background.“Starry Night” is cited as an example of a difficult painting.
  • Camera-model limitation: Non-pinhole projections are difficult because the implicit camera representation is pinhole-based and cannot rectify the resulting deformation.This limitation directly constrains performance on non-pinhole imagery.

V. CONCLUSION

UniDepthV2 targets universal monocular metric depth estimation without camera parameters at test time. Its architectural and loss improvements yield strong generalization and efficiency, while variable input resolution enables a speed–detail trade-off with global scale consistency.

  • Universal estimation: UniDepthV2 enhances generalization across diverse domains without requiring camera parameters at test time.The conclusion identifies it as a universal monocular metric depth estimation model.
  • Performance and efficiency: Architectural improvements and new training losses produce state-of-the-art performance while improving computational efficiency.The conclusion reports this outcome from extensive zero-shot and fine-tuning evaluations.
  • Inference flexibility: Variable input resolutions at test time provide a flexible inference-speed and detail-preservation trade-off while maintaining global scale consistency.This flexibility is enabled by the training strategy.
Loading 2502.20110v2…