PaperlayerView exact revision on arXiv

Source-linked AI summary

UniK3D: Universal Camera Monocular 3D Estimation

Luigi Piccinelli, Christos Sakaridis, Mattia Segu, Yung-Hsu Yang, Siyuan Li, Wim Abbeloos, Luc Van Gool

arXiv:2503.16591v1cs.CV

TL;DR

Monocular metric 3D estimation remains limited by methods that produce only relative-scale outputs, despite the need for metric estimates in 3D reconstruction and geometric understanding. UniK3D uses a spherical representation for diverse camera models and achieves state-of-the-art metric-depth and 3D estimation across 13 datasets.

  • Problem

    Recent generalizable monocular depth methods are constrained to relative scale, while practical 3D reconstruction and geometric scene understanding require metric-scaled estimates.

    Source: §1 · p. 1

  • Method

    UniK3D uses a spherical 3D representation to separate camera projection from scene geometry and support monocular metric estimation across diverse camera models.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across 13 metric-depth datasets, UniK3D achieves state-of-the-art monocular metric depth and 3D estimation and generalizes across camera models without preprocessing or camera-specific training domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UniK3D generalizes to unseen cameras and 3D scene domains, while broader training-data coverage could further improve its robustness and applicability.

    Source: §5 · p. 8

Abstract

from arXiv · show

Monocular 3D estimation is crucial for visual perception. However, current methods fall short by relying on oversimplified assumptions, such as pinhole camera models or rectified images. These limitations severely restrict their general applicability, causing poor performance in real-world scenarios with fisheye or panoramic images and resulting in substantial context loss. To address this, we present UniK3D, the first generalizable method for monocular 3D estimation able to model any camera. Our method introduces a spherical 3D representation which allows for better disentanglement of camera and scene geometry and enables accurate metric 3D reconstruction for unconstrained camera models. Our camera component features a novel, model-independent representation of the pencil of rays, achieved through a learned superposition of spherical harmonics. We also introduce an angular loss, which, together with the camera module design, prevents the contraction of the 3D outputs for wide-view cameras. A comprehensive zero-shot evaluation on 13 diverse datasets demonstrates the state-of-the-art performance of UniK3D across 3D, depth, and camera metrics, with substantial gains in challenging large-field-of-view and panoramic settings, while maintaining top accuracy in conventional pinhole small-field-of-view domains. Code and models are available at github.com/lpiccinelli-eth/unik3d .

1. Introduction

UniK3D introduces a camera-universal framework for monocular metric 3D estimation, using a spherical representation to model scene geometry and camera rays without assuming a specific camera model. The paper reports zero-shot state-of-the-art performance across 13 metric depth datasets and diverse camera types.

Introduction

  • Prior metric-estimation methods assume known intrinsics, pinhole cameras, or ground-truth rectification [24, 85, 9, 60, 61], limiting their applicability to non-linear fisheye and panoramic projections.Disparity and log-depth outputs are mathematically ill-posed above 180 degrees field of view, and estimating full metric 3D geometry depends more heavily on camera estimation than depth alone.Source: §1 · pp. 1–2
  • UniK3D models 3D outputs in a fully spherical space using radial distance, supporting extreme fields of view and disentangling projective effects from scene geometry.This is intended to improve reconstruction consistency and stability near the xy-plane, where depth approaches zero.Source: §1 · p. 2
  • Its camera module directly represents pencils of rays with a finite spherical harmonics basis, avoiding explicit camera-model assumptions while promoting compactness, continuity, and differentiability.The method also uses asymmetric angular loss based on quantile regression, static encoding, and curriculum learning to condition radial prediction.Source: §1 · p. 2
  • Across 13 widely used metric depth datasets, UniK3D achieves state-of-the-art monocular metric depth and 3D estimation and generalizes across camera models without preprocessing or camera-domain-specific training.The introduction describes these as zero-shot experiments.Source: §1 · p. 2

2. Related Work

Prior monocular depth methods struggle to generalize zero-shot across camera types, while UniK3D targets camera-agnostic metric 3D reconstruction through a spherical-basis camera representation.

Monocular Depth Estimation

  • Modern MDE models, from convolutional to transformer architectures, advance controlled benchmarks but often fail zero-shot across camera, scene, geometric, and visual domains.The field's central challenge is robust generalization across varying cameras and scenes.Source: §2 · p. 2

Generalizable Monocular Depth Estimation

  • Scale-agnostic methods [32] [63] [73] [81] [82] need extra scale information, while metric models [7] [9] [24] [28] [60] [61] [85] generally assume known or pinhole cameras.Prior zero-shot methods [9] [60] [85] remain limited by pinhole assumptions or test-time image rectification.Source: §2 · p. 2
  • UniK3D reconstructs a coherent 3D point cloud from one image for any inverse projection, without requiring camera intrinsics or rectification at test time.This is presented as a unified approach to monocular metric 3D estimation for diverse real-world applications.Source: §2 · pp. 2–3

Camera Calibration

  • The architecture predicts ray azimuth and polar angles, conditions radial features on camera geometry, and supervises angular and radial coordinates.The camera and radial outputs are concatenated before a closed-form transform produces Cartesian 3D points.Source: Figure 2 · p. 3, §2 · p. 3
  • Traditional parametric calibration models cover narrow- and wide-angle lenses but require controlled calibration, and individual models have limits such as UCM's inability to represent tangential distortion.Kannala-Brandt is reported to struggle beyond a 210° field of view.Source: §2 · p. 3
  • UniK3D models camera backprojection as a linear combination of spherical basis functions, inferring scalar expansion coefficients and spherical-domain boundaries.Its camera representation uses an inverse spherical harmonics transformation.Source: §2 · p. 3

3. UniK3D

UniK3D targets monocular 3D estimation across camera setups that challenge camera-specific or rectification-based methods. It combines spherical scene outputs, a flexible camera representation, and interventions against wide-field-of-view contraction.

3.1. Representations

  • UniK3D represents metric scene geometry with spherical coordinates, separating camera parameters from scene geometry and using Euclidean radius rather than perpendicular depth.The spherical-to-Cartesian transformation produces a 3D point cloud; the radius representation makes projected object dimensions vary univocally with range and improves stability near the xy-plane.Source: §3.1 · pp. 3–4
  • A learned superposition of spherical harmonics represents each pixel’s viewing ray, compactly modeling complex ray distributions with continuous, differentiable outputs.The basis is defined by a generalized principal point and horizontal and vertical fields of view; 15 harmonics plus four domain parameters describe virtually any camera geometry with 19 parameters.Source: §3.1 · p. 4

3.2. Preventing Distribution Contraction

  • An asymmetric angular loss based on quantile regression counters underprediction of wide-field-of-view angles without rebalancing the training data.Tuning α adjusts the relative weighting of polar-angle over- and underestimation; α = 0.5 recovers mean absolute error. The approach is inspired by type-I and type-II errors.Source: §3.2 · p. 4
  • To strengthen camera conditioning, UniK3D uses static camera-ray encoding, gradually switches from ground-truth to predicted camera inputs, and blocks feedback shortcuts.Ground-truth camera inputs are supplied with probability 1 − tanh(s/105); gradients from camera outputs fed to the Radial Module are detached, and learnable gains in its cross-attention layers are disabled.Source: §3.2 · pp. 4–5

3.3. Network Design

  • The network combines a ViT-based encoder, an Angular Module, and a camera-conditioned Radial Module to produce 3D points and radial confidence.It optimizes angular, log-radius, and confidence losses in a linear combination, with η = 2 and γ = 0.1; the radial loss uses ground-truth camera and depth.Source: §3.3 · pp. 5–6

4. Experiments

UniK3D achieves strong zero-shot 3D reconstruction across camera domains, with especially large gains for large-FoV and panoramic images while retaining top small-FoV accuracy. Ablations show that diverse camera training data, an SH-based camera model, radial output, and combined angular-loss and conditioning strategies contribute to performance.

4. Experiments

  • The zero-shot evaluation tests 13 unseen datasets across four camera domains, using a consistent pipeline without test-time augmentation or domain-specific fine-tuning.The domains are S.FoV, distorted S.FoV, L.FoV, and 360° panoramic; metrics cover scale- and shift-invariant depth, monocular 3D reconstruction, and camera-ray accuracy.Source: §4 · pp. 6–7

4.1. Comparison with The State of The Art

  • In L.FoV, UniK3D achieves δ1 SSI of 91.2% and FA of 71.6%, surpassing the second-best method by more than 20% and 40%, respectively.In Pano, its δ1 SSI and FA are 71.2% and 66.1%, setting new state-of-the-art results; in S.FoV, its δ1 SSI is 94.3.Source: §4.1 · p. 7

4.2. Ablation Studies

  • Training with strongly distorted-camera images generally improves performance across domains, especially distorted S.FoV and L.FoV, while Pano gains are limited with log-depth output.The authors attribute the limited panoramic improvement to the difficulty of representing panoramic images using log-depth.Source: Table 3 · p. 7, §4.2 · p. 8
  • The SH camera model performs best overall, particularly in L.FoV and Pano, while non-parametric rays underperform in FA and ρA and Zernike bases struggle with spherical geometries.The passage attributes the non-parametric model's weakness to its data-driven formulation and the Zernike basis's weakness to its planar structure.Source: Table 4 · p. 7, §4.2 · p. 8
  • Spherical-radius output improves reconstruction in L.FoV and Pano only when paired with a camera model such as SH, but degrades accuracy in distorted S.FoV.The passage attributes the degradation to radius sensitivity to minor angular variations in small, highly distorted views.Source: Table 5 · p. 7, §4.2 · p. 8
  • Combining asymmetric angular loss with enhanced camera conditioning outperforms configurations without both strategies, especially in distorted and L.FoV domains.The strategies mitigate contraction in backprojection and improve angular prediction; their effects are also visible in the qualitative examples in Figure 4.Source: §4.2 · p. 8, Table 6 · p. 8, Figure 4 · p. 8

5. Conclusion

UniK3D is a universal monocular 3D estimation framework that generalizes across pinhole, fisheye, and panoramic cameras. Its flexible backprojection and FOV-contraction prevention support metric 3D estimation and strong generalization to unseen cameras and scene domains.

5. Conclusion

  • UniK3D achieves compelling generalization to unseen cameras and 3D scene domains, surpassing previous state of the art with a fair quantity of data.Expanding training-data diversity and coverage could further improve its robustness and applicability.Source: §5 · p. 8
  • The framework generalizes across diverse camera models, from pinhole to fisheye and panoramic, and supports accurate metric 3D estimation.Its flexible, robust backprojection design works with any generic camera model.Source: §5 · p. 8
  • UniK3D introduces strategies to prevent field-of-view contraction.Source: §5 · p. 8

A. Architecture · A.1. Complexity · A.2. Architectural Alternatives

UniK3D separates camera modeling from dense scene features through angular and radial modules. Its attention-based camera conditioning outperforms simpler alternatives, while the model remains flexible across image resolutions and computationally competitive.

A. Architecture

  • The encoder extracts feature maps and class tokens from a ViT’s last four layers, then applies separate normalization and linear projections before routing them to the radial and angular modules.Projected channel dimensions are 512, 384, and 256 for Large, Base, and Small ViT variants, respectively.Source: A. Architecture · p. 13
  • The Angular Module processes 18 tokens and uses three to define the spherical-harmonic domain, then combines 15 harmonics with corresponding tokens to construct the pencil of rays.The harmonics are computed up to degree three, excluding the constant component.Source: A. Architecture · p. 13
  • The Radial Module conditions dense features on sine-encoded camera rays through cross-attention, then uses an FPN-style decoder to predict log-radius and log-confidence maps.Exponentiating the maps yields radius and confidence values.Source: A. Architecture · p. 13

A.1. Complexity

  • UniK3D is among the most efficient models and maintains robust performance across resolutions, with added cost from angular and scale components for metric depth and camera adjustment.The comparison uses similar input sizes where possible, but some methods are sensitive to resolutions that differ from their training setup.Source: §A.1 · pp. 13–14, Table 7 · p. 14

A.2. Architectural Alternatives

  • Addition and concatenation underperform UniK3D’s attention-based camera conditioning, which the authors say is important for generalization.Table 8 compares these conditioning designs; the supplied passages do not give numerical results.Source: §A.2 · p. 14, Table 8 · p. 14
  • Processing camera tokens with the Transformer Encoder produces no large changes, indicating that class tokens from different layers are already informative.Table 9 compares encoder processing with directly projecting the tokens.Source: §A.2 · p. 14, Table 9 · p. 14

B. Training Details · B.1. Hyperparameters.

The appendix identifies where UniK3D’s training hyperparameters and loss specifications are documented.

B.1. Hyperparameters.

  • Optimization, scheduling, and augmentation parameters are described in Table 10.Source: §B.1 · p. 14
  • Table 11 outlines the losses used, their inputs, and their corresponding weights.Source: §B.1 · p. 14

B.2. Data · B.3. Camera Augmentations

Training combines diverse camera and domain data, while camera augmentation synthesizes distorted views from pinhole images and creates manually checked validation data with unchanged 3D ground truth.

B.2. Data

  • Training uses more than 8M samples from mixed camera and domain datasets, with sequence frames at least 0.5 seconds apart and batch sampling adjusted to favor dataset diversity.No post-processing is applied; initial sampling probabilities relate to the number of scenes in each dataset.Source: §B.2 · pp. 14–15
  • Distorted-image datasets with pinhole-only calibration, such as Mapillary, are masked out during camera-loss computation.Other training data includes pinhole or rectified images, including MegaDepth and NianticMapFree [3].Source: §B.2 · p. 15
  • Validation datasets are grouped into small-, large-, and panoramic-FoV categories, and the KITTI Eigen split excludes 45 images with inaccurate ground-truth depth.The split uses corrected and accumulated ground-truth depth maps from the original 697 images.Source: §B.2 · p. 15, Table 13 · p. 15

B.3. Camera Augmentations

  • Validation images are synthesized by deforming RGB from ETH3D, IBims-1, and Diode (Indoor) [70], while retaining 3D ground truth unchanged.Nearly complete depth maps make deformation well-posed; small gaps are inpainted, and each deformed image is manually checked for realism.Source: §B.2 · p. 15, §B.3 · p. 16
  • Camera augmentation deforms pinhole images by unprojecting depth into 3D and projecting points through a randomly sampled distorted camera model.The sampled models include Fisheye624 and radial Kannala-Brandt.Source: §B.3 · pp. 15–16
  • Softmax-based splatting warps images using the deformation field, with inverse depth prioritizing nearby pixels to preserve details and parallax while avoiding holes.For non-synthetic images without ground-truth depth, the method uses inference-only depth predictions and applies augmentation only after 10,000 training steps.Source: §B.3 · p. 16

C. Additional Quantitative Results · C.1. Fine-tuning

UniK3D benefits from fine-tuning on in-domain data, adapting to both structured KITTI and less-structured NYU settings. The evaluation uses dataset-specific protocols without test-time augmentation or tuning.

C.1. Fine-tuning

  • Fine-tuning resumes from the large-scale pretraining weights and optimizer states, using standard SILog loss, batch size 16, and 40,000 additional training steps.KITTI or NYU is used as the sole training dataset.Source: §C.1 · pp. 16–17
  • In-domain fine-tuning significantly benefits UniK3D across datasets with distinct properties and domain-specific challenges.The paper describes strong performance on both KITTI and NYU after fine-tuning.Source: §C.1 · p. 17
  • Fine-tuned UniK3D performs exceptionally well on highly structured, calibrated KITTI data despite its focus on flexibility and cross-domain generalization.KITTI results use the Garg evaluation crop.Source: §C.1 · p. 17
  • On less-structured NYU indoor data, fine-tuned UniK3D remains competitive.The paper presents this as further evidence that in-domain data helps address dataset-specific properties and challenges.Source: §C.1 · p. 17
  • During fine-tuning, all augmentations except horizontal flipping are disabled, and the asymmetric angular-loss component is omitted.Source: §C.1 · p. 17
  • Evaluation follows standard dataset practices, caps depth at 80 meters for KITTI and 10 meters for NYU, and uses no test-time augmentation or tuning.The authors omit test-time changes such as varying input size to avoid confounding factors.Source: §C.1 · p. 17

C.2. Per-dataset Evaluation

Across per-dataset zero-shot evaluations, UniK3D achieves the highest average metric overall despite not ranking first on every dataset. The authors attribute its cross-domain robustness to flexible design and large-scale training.

C.2. Per-dataset Evaluation

  • Results are reported independently across validation datasets, while the Pano group is omitted because it contains only Stanford-2D3D.The listed datasets include NYUv2, KITTI, IBims-1, ETH3D, Diode Indoor, nuScenes, ScanNet++ DSLR, ADT, and KITTI360, with additional distance variants.Source: §C.2 · p. 17
  • UniK3D achieves the highest average metric overall, although it does not rank first on every individual dataset.The authors describe pinhole-camera performance as having reached a saturation point.Source: §C.2 · p. 17
  • The authors attribute UniK3D’s robust performance across diverse domains without overfitting to its flexible design and large-scale training.Source: §C.2 · p. 18
  • The reported additional metrics are absolute relative error (A.Rel, percentage) and root-mean-squared error (RSME, meters).Source: §C.2 · p. 18

D. Q&A

The Q&A explains how training-data diversity affects scale generalization, why UniK3D’s camera and 3D representations support broader camera models, and where its methods remain limited.

Q&A

  • Data diversity supports generalization across broad domains such as NYU, but can hurt specialized domains such as KITTI; a few hundred in-domain fine-tuning steps can mitigate scale gaps.Monocular metric scale prediction is ill-posed, depends on training-domain coverage, and is sensitive to domain shifts.Source: D. Q&A · pp. 18–19
  • UniK3D consistently outperforms UniDepth [60] with identical training data by predicting spherical-harmonic ray coefficients and radial distance rather than pinhole calibration and z-depth.UniDepth’s pinhole and z-depth choices restrict it to pinhole cameras and FoV < 180°; UniK3D directly predicts SH coefficients to generate camera rays.Source: Table 30 · p. 19, D. Q&A · pp. 19–20
  • Parametric camera models can be unstable to optimize and limit expressiveness, while UniK3D’s representation offers greater flexibility by avoiding reliance on those models.Standard models may require non-differentiable backprojection; differentiable alternatives still face joint parameter-to-ray instability and camera-family constraints.Source: D. Q&A · p. 19
  • DUSt3R and MASt3R can theoretically represent any camera, but fully non-parametric models struggle on diverse-data edge cases, and their test-time alignment requires a pinhole camera [74] [39].The alignment limitation further restricts their applicability to generic cameras.Source: D. Q&A · p. 19
  • Confidence prediction mainly supports downstream tasks and legacy use, but domain gaps can make confidence unreliable in strongly out-of-domain scenarios.The authors note its vulnerability is similar to that of most regression tasks.Source: D. Q&A · p. 19
  • Camera augmentations address the shortage of diverse real-camera data, but the simple pipeline gives minor gains and often generates unrealistic cameras.Softmax-based warping proved effective for realistic images; the authors hypothesize ray-based camera sampling could improve robustness across practical camera models.Source: D. Q&A · p. 19
  • Earlier efforts to remove the pinhole assumption focused on single-domain self-supervised depth estimation, unlike UniK3D’s supervised large-scale monocular 3D estimation [35].Those methods need camera models to define warping-based photometric losses.Source: D. Q&A · p. 20
  • δSSI_1 scores of 92.1 and 92.2 for rows 3 and 4 of Table 5 are similar, while stronger small-FoV distortion augmentation yields (FA, ρA) = (43.1, 62.3).The radial- and SH-based model in row 4 overestimates image FoV under lens distortion; the authors interpret the score similarity and FA and ρA drops as evidence of the angular module’s role.Source: D. Q&A · p. 20

E. Additional Qualitative Results

The section presents additional qualitative results on distorted-camera validation domains and in-the-wild imagery with deformed cameras or unusual viewpoints. The displayed examples are casually selected rather than cherry-picked.

E. Additional Qualitative Results

  • Figure 5 compares results on ScanNet++ (DSLR), IBims-1Dist, and DiodeDist (Indoor), validation domains with distorted cameras not shown in the main paper.Examples are randomly selected; the figure shows RGB inputs and absolute-relative-error maps, plus ground truth and predicted 3D point clouds. Errors use GT-based shifted and scaled outputs for all models.Source: E. Additional Qualitative Results · p. 20, Figure 5 · p. 21
  • The in-the-wild images in Figures 6 and 7 depict deformed cameras or unusual points of view, and the examples are casually selected rather than cherry-picked.The authors state that both the validation-set and in-the-wild visualizations are not cherry-picked.Source: E. Additional Qualitative Results · p. 20, Figure 6 · p. 22, Figure 7 · p. 23
  • Figures 6 and 7 show UniK3D producing point clouds from single images of movies, TV, YouTube, music videos, and anime.The frames span varied camera types and unusual viewpoints, including egocentric GoPro and doorbell-camera views.Source: Figure 6 · p. 22, Figure 7 · p. 23

Related papers

Loading 2503.16591v1…