UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler

• Existing MMDE methods often require similar camera intrinsics and scene scales during training and testing, limiting real-world generalization.Source: §I · p. 1
• Imperfect, sparse ground-truth depth near edges produces blurry predictions and inaccurate fine-grained geometric details.Source: §I · p. 1
• UniDepthV2 predicts metric 3D points for every pixel from a single image without requiring camera parameters or other external information.Source: §I · pp. 1–2
• A dense camera representation prompts the depth module, while pseudo-spherical azimuth, elevation, and radial-depth components disentangle camera and depth dimensions.Source: Figure 1 · p. 1, §I · p. 2
• The revised architecture improves efficiency and accuracy, while the edge-guided scale-shift-invariant loss preserves local depth structure and sharpens discontinuities.Source: §I · p. 2

• Scale-agnostic methods generalize across dense prediction settings but lack metric outputs needed for practical downstream use.Source: §II · pp. 2–3
• Conventional MMDE achieves strong benchmark performance but faces substantial degradation in zero-shot scenarios and domain gaps.Source: §II · p. 3
• Generalizable MMDE methods commonly incorporate camera awareness through external intrinsics or intrinsic-based normalization.Source: §II · p. 3
• UniDepthV2 directly estimates the 3D scene from RGB input without additional information, unlike approaches relying on camera parameters.Source: §II · p. 3, Figure 2 · p. 3
• Methods for adverse lighting and non-Lambertian surfaces address challenging conditions, while UniDepthV2 pursues camera-free universal MMDE across such domains.Source: §II · p. 3

• UniDepthV2 targets generalizable MMDE by estimating 3D points without camera intrinsics or other external information.Source: §III · p. 3

A. 3D Representation

• Its pseudo-spherical output uses azimuth, elevation, and log-depth to decouple camera angles from depth.Source: A. 3D Representation · p. 4
• The dense camera representation is a tensor C ∈ R^H×W×2 whose final dimension stores azimuth and elevation values.Source: A. 3D Representation · p. 4

B. Self-Promptable Camera

• The camera module internally uses pinhole parameters and residuals to produce the dense angular representation.Source: B. Self-Promptable Camera · p. 4
• A self-promptable camera module converts backprojected rays into sine-encoded embeddings that condition the depth module.Source: B. Self-Promptable Camera · p. 4

• The loss enforces consistency between camera-prompted depth features corresponding to identical 3D locations across acquisition sensors.Source: C. Geometric Invariance Loss · p. 4
• Training creates two augmented views using random rescaling, translation, and cropping to simulate different cameras observing the same scene.Source: C. Geometric Invariance Loss · p. 4
• Because augmentations produce different camera representations, the consistency loss compares transformed depth outputs rather than raw camera prompts.Source: C. Geometric Invariance Loss · p. 4
• The bidirectional loss uses stop-gradient on one decoded depth output as pseudo ground truth and applies consistency to camera-aware outputs.Source: C. Geometric Invariance Loss · p. 4

• LEG-SSI samples patches from the top 5%-quantile RGB-gradient regions to emphasize likely depth discontinuities.Patch sizes are sampled between 4% and 8% of the input image’s smallest dimension.Source: D. Edge-Guided Normalized Loss · p. 4
• The loss oversamples thin high-frequency boundaries and avoids superpixels because region uniformity can suppress boundary detail.Source: D. Edge-Guided Normalized Loss · p. 5
• Predicted and ground-truth depth are independently normalized per patch so supervision measures local shape consistency rather than absolute depth scale.Source: D. Edge-Guided Normalized Loss · p. 5
• LEG-SSI penalizes both missed depth discontinuities and hallucinated discontinuities caused by appearance changes without geometric edges.Source: D. Edge-Guided Normalized Loss · p. 5
• A custom CUDA kernel accelerates the random-patch loss computation by 20x.Source: D. Edge-Guided Normalized Loss · p. 5

Architecture

• The architecture contains a ViT-based encoder, a Camera Module, and a Depth Module.Source: E. Network Design · p. 5
• The Camera Module initializes four tokens, processes them with self-attention, and embeds the resulting pinhole representation.Source: E. Network Design · p. 5
• Four encoder feature maps are separately conditioned on camera prompts and decoded with an FPN-style multiscale decoder.Source: E. Network Design · p. 5
• The final 3D output concatenates predicted rays C with exponentiated depth Z, and the ray bundle can alternatively be supplied by another camera model.Source: E. Network Design · p. 5

Optimization

• The output-space objective uses weighted variance and mean errors over azimuth, elevation, and log-depth dimensions.Source: E. Network Design · p. 5
• The final loss combines weighted MSE, geometric consistency, edge-guided scale-shift invariant, and uncertainty terms with weights (0.1, 1.0, 0.1).Source: E. Network Design · p. 5

Data

• Training combines 23 public datasets totaling 16M images, while evaluation uses ten unseen datasets spanning indoor, outdoor, and challenging domains.The challenging group includes underwater and transparent-object settings.Source: A. Experimental Setup · p. 6

Evaluation protocol

• All methods are re-evaluated with the same weights and pipeline without test-time augmentation or in-domain fine-tuning.Zero-shot checkpoints are used rather than models fine-tuned on KITTI or NYU.Source: A. Experimental Setup · p. 6

Metrics

• δ1 measures depth estimation, FA measures 3D estimation accuracy, and ρA measures camera performance through angular-error AUC.ρA integrates average camera-ray angular error up to 15°.Source: A. Experimental Setup · pp. 6–7
• Confidence quality is assessed with AUSE, normalized AUSE, and Spearman’s ρ by comparing uncertainty rankings with prediction errors.nAUSE equal to 1 corresponds to random ranking, while oracle AUSE is 0.Source: A. Experimental Setup · p. 7

Implementation

• Training uses AdamW, cosine scheduling, 300k optimization iterations, batch size 128, and geometric and photometric augmentation.Image ratios are randomly sampled between 2:1 and 1:2; training takes six days on 16 NVIDIA 4090 GPUs.Source: A. Experimental Setup · p. 7

Zero-shot comparison

• UniDepthV2 performs better than or at least on par with all baselines across ten zero-shot validation sets, including methods requiring ground-truth cameras.The evaluation covers indoor, outdoor, and challenging domains.Source: B. Comparison with The State of The Art · p. 7, Table III · p. 8

Efficiency

• UniDepthV2 is among the fastest and most efficient models despite including an additional camera-prediction module.The model outperforms UniDepth in nearly all reported cases except ρA on IBims-1, DDAD, and NuScenes.Source: B. Comparison with The State of The Art · p. 8

Fine-tuning

• Fine-tuning on NYU and KITTI shows that UniDepthV2 adapts effectively and outperforms similarly sized models pretrained on large, diverse datasets.The comparisons follow standard validation protocols for each target domain.Source: B. Comparison with The State of The Art · p. 8, Table V · p. 9

Robustness

• UniDepthV2 is almost invariant to image shape, allowing resolution to trade off against speed without sacrificing accuracy.Figure 4 compares normalized δ1 across varying image shapes.Source: B. Comparison with The State of The Art · p. 8, Figure 4 · p. 8

Efficiency

• FPS decreases roughly inversely with resolution and peak GPU memory grows near-quadratically; UniDepthV2 is around 2× faster than baselines up to approximately 2 MP.Beyond approximately 5 MP, all methods become memory-bound and reach sub-FPS throughput.Source: B. Comparison with The State of The Art · p. 8, Figure 5 · p. 9

Edge evaluation

• UniDepthV2-Large achieves the best result on Sintel and remains competitive on ETH3D and IBims-1, outperforming the original UniDepth.The edge evaluation follows the scale-invariant boundary F1 protocol.Source: Table IV · p. 8, B. Comparison with The State of The Art · p. 9

Ablation scope

• The ablation suite evaluates architectural changes, losses, camera conditioning, output representation, efficiency, uncertainty, and qualitative edge predictions.Tables IX–XII and VII, together with Fig. 6, isolate these design choices and evaluation dimensions.Source: C. Ablation Studies · p. 9, Table VII · p. 9, Table X · p. 10

Architecture

• The final multi-resolution architecture significantly reduces computational cost while preserving overall performance.It combines ResNet decoder blocks with FPN-style multi-resolution feature fusion; the final row adds the proposed losses.Source: C. Ablation Studies · p. 9

Losses

• 4.7% improvement in δ1 and 1.8% improvement in FA result from introducing the LEG−SSI loss.The loss also improves ρA and visibly refines depth discontinuities in qualitative comparisons.Source: C. Ablation Studies · pp. 9–10

Conditioning

• Camera conditioning has a stronger positive effect for UniDepthV2 than for UniDepth under the ablation settings.The comparison removes LEG−SSI and deep-feature invariance to isolate conditioning impact.Source: C. Ablation Studies · p. 10

Representation

• The pseudo-spherical representation benefits depth-specific metrics while also substantially improving 3D and camera accuracy.This comparison is made under otherwise matched camera settings against direct Cartesian regression.Source: C. Ablation Studies · p. 11, Table XII · p. 11

Confidence

• Zero-shot uncertainty remains informative but becomes less calibrated under domain shift, with nAUSE rising to 0.54–0.65 and ρ falling to 0.29.In-domain nAUSE is 0.199–0.221 and ρ is 0.68–0.74; the output supports reliability-aware masking despite degraded calibration.Source: Table VII · p. 9, C. Ablation Studies · p. 11

Test conditions

• The failure-case evaluation covers mirrors, paintings, optical illusions, and non-pinhole projections collected from the internet.These examples represent unusual conditions for depth models.Source: D. Failure Cases and Limitations · p. 11

Observed successes

• UniDepthV2 can resolve certain human-related optical illusions and recognize paintings when frames and contextual objects are visible.The passage gives “Liberty Leading the People” as an example.Source: D. Failure Cases and Limitations · p. 11

Observed failures

• Mirrors remain particularly challenging because insufficient context can make them appear as cavities while inducing strong deformation.The model’s interpretation is therefore ambiguous in these cases.Source: D. Failure Cases and Limitations · p. 11
• Non-realistic paintings and distorted geometry remain difficult, although the model can still separate foreground from background.“Starry Night” is cited as an example of a difficult painting.Source: D. Failure Cases and Limitations · p. 11

Camera-model limitation

• Non-pinhole projections are difficult because the implicit camera representation is pinhole-based and cannot rectify the resulting deformation.This limitation directly constrains performance on non-pinhole imagery.Source: D. Failure Cases and Limitations · p. 11

Universal estimation

• UniDepthV2 enhances generalization across diverse domains without requiring camera parameters at test time.The conclusion identifies it as a universal monocular metric depth estimation model.Source: §V · p. 11

Performance and efficiency

• Architectural improvements and new training losses produce state-of-the-art performance while improving computational efficiency.The conclusion reports this outcome from extensive zero-shot and fine-tuning evaluations.Source: §V · p. 11

Inference flexibility

• Variable input resolutions at test time provide a flexible inference-speed and detail-preservation trade-off while maintaining global scale consistency.This flexibility is enabled by the training strategy.Source: §V · p. 11