Source-linked AI summary
UniK3D: Universal Camera Monocular 3D Estimation
Luigi Piccinelli, Christos Sakaridis, Mattia Segu, Yung-Hsu Yang, Siyuan Li, Wim Abbeloos, Luc Van Gool
TL;DR
Existing monocular 3D methods rely on restricted camera assumptions, limiting metric reconstruction across diverse camera models. UniK3D uses spherical representations, spherical-harmonic camera rays, and asymmetric angular loss, achieving state-of-the-art zero-shot generalization across 13 datasets, including fisheye and panoramic settings.
Problem
Restricted camera assumptions limit monocular metric 3D estimation across diverse camera models, despite its importance for accurate 3D reconstruction and geometric scene understanding.
Method
UniK3D combines a fully spherical 3D representation, spherical-harmonic modeling of camera rays, and asymmetric angular loss for camera-universal monocular 3D estimation.
Results
UniK3D achieves state-of-the-art monocular metric depth and 3D estimation while generalizing zero-shot across 13 datasets and diverse camera models without preprocessing or camera-specific training domains.
Takeaways & Limitations
UniK3D supports accurate metric 3D estimation across pinhole, fisheye, panoramic, and other unconstrained camera models, including unseen cameras and 3D scene domains.
Abstract
from arXiv · showhide
Monocular 3D estimation is crucial for visual perception. However, current methods fall short by relying on oversimplified assumptions, such as pinhole camera models or rectified images. These limitations severely restrict their general applicability, causing poor performance in real-world scenarios with fisheye or panoramic images and resulting in substantial context loss. To address this, we present UniK3D, the first generalizable method for monocular 3D estimation able to model any camera. Our method introduces a spherical 3D representation which allows for better disentanglement of camera and scene geometry and enables accurate metric 3D reconstruction for unconstrained camera models. Our camera component features a novel, model-independent representation of the pencil of rays, achieved through a learned superposition of spherical harmonics. We also introduce an angular loss, which, together with the camera module design, prevents the contraction of the 3D outputs for wide-view cameras. A comprehensive zero-shot evaluation on 13 diverse datasets demonstrates the state-of-the-art performance of UniK3D across 3D, depth, and camera metrics, with substantial gains in challenging large-field-of-view and panoramic settings, while maintaining top accuracy in conventional pinhole small-field-of-view domains. Code and models are available at github.com/lpiccinelli-eth/unik3d .
1. Introduction
Monocular metric 3D estimation remains limited by camera assumptions and scale ambiguity, especially for wide-angle imagery. UniK3D addresses these constraints with spherical scene and camera representations, robust conditioning, and broad zero-shot evaluation.
- Metric-scaled monocular depth is crucial for accurate 3D reconstruction and geometric scene understanding in embodied applications.
- Existing methods assume pinhole cameras or require rectification parameters, limiting performance on fisheye and panoramic imagery.
- UniK3D represents both scene geometry and camera rays spherically, supporting monocular metric 3D estimation across unrestricted camera projective geometries.
- Asymmetric angular loss, static encoding, and curriculum learning provide robust conditioning for the radial module and help prevent wide-field-of-view output contraction.
- Zero-shot experiments on 13 metric-depth datasets report state-of-the-art monocular metric depth and 3D estimation while generalizing across camera models without preprocessing or camera-domain training.
2. Related Work
Prior monocular depth methods often fail in zero-shot settings and rely on camera models or calibration assumptions. UniK3D instead predicts camera rays directly in a spherical-harmonics representation to recover coherent 3D point clouds from unconstrained images.
- Modern monocular depth networks achieve strong controlled-benchmark results but often fail in zero-shot scenarios.
- UniK3D recovers a coherent 3D point cloud from a single image without camera intrinsics, rectification, or test-time camera information.
- The architecture separates an Angular Module for camera projective geometry from a Radial Module that predicts scene range.
- Unlike parametric camera models requiring controlled environments, UniK3D models backprojection as a linear combination of spherical basis functions.
3. UniK3D
UniK3D introduces a camera-universal monocular 3D estimation framework for arbitrary projective geometries, using spherical representations to separate camera effects from scene geometry. Its camera representation and angular-training strategies target wide-field-of-view contraction while supporting zero-shot evaluation across diverse camera domains.
- Motivation and scope: UniK3D addresses the limitations of pinhole, equirectangular, and rectification-dependent methods by estimating monocular 3D geometry for arbitrary camera setups.These assumptions restrict real-world applicability to non-standard projective geometries.
- Spherical representations: The output space represents scene range with Euclidean radius in spherical coordinates, then converts it bijectively into Cartesian 3D point clouds.Using radius rather than perpendicular depth makes projected object dimensions a univocal function of range and improves stability near the xy-plane.
- Spherical representations: The camera’s pencil of rays is reconstructed from spherical-harmonic coefficients over a domain defined by the reference-frame pole and horizontal and vertical FoVs.The basis provides a compact, continuous, and differentiable representation of complex ray distributions.
- Preventing distribution contraction: An asymmetric angular loss based on quantile regression emphasizes underrepresented wide-FoV angles instead of relying on dataset rebalancing.The target quantile α controls the relative weighting of angular over- and underestimation; α = 0.5 recovers MAE.
- Network design: Static camera-ray encoding, curriculum learning, gradient detachment, and disabled learnable gains strengthen camera conditioning of the Radial Module.Training transitions from ground-truth camera parameters to predicted ones while preventing feedback shortcuts.
- Evaluation: UniK3D is evaluated zero-shot across diverse camera domains, including small-FoV, distorted, large-FoV, and panoramic settings, with missing values indicating some models cannot produce specific outputs.Table 1 groups validation sets into S.FoV, S.FoVDist, L.FoV, and Pano domains; Table 2 compares against equirectangular-specialized methods.
4. Experiments
UniK3D is evaluated through zero-shot experiments and ablations across diverse camera fields of view, showing especially strong performance for large-FoV and panoramic imagery while retaining accuracy in small-FoV settings.
- Evaluation setup: The evaluation uses 26 training sources and zero-shot testing with shared weights, no test-time augmentation, and metrics for depth, 3D estimation, and camera performance.The reported metrics are δSSI1, FA, and ρA.
- Comparison with the state of the art: UniK3D achieves δ1SSI of 91.2% and FA of 71.6% in L.FoV, exceeding the second-best method by more than 20% and 40%, respectively.These results are reported as state-of-the-art performance in the large-field-of-view domain.
- Comparison with the state of the art: In Pano, UniK3D reaches δ1SSI of 71.2% and FA of 66.1%, establishing new state-of-the-art results under extreme camera setups.The results are associated with the model’s SH-based camera representation and radial output design.
- Comparison with the state of the art: UniK3D retains top performance in conventional S.FoV settings, including δ1SSI of 94.3, while also providing accurate camera estimation according to ρA.The method estimates 3D geometry from varied and distorted cameras without sacrificing small-FoV effectiveness.
- Ablation studies: Training with strongly distorted camera images generally improves performance across domains, although gains on Pano remain limited because panoramic images are difficult to represent with log-depth.The SH camera basis provides the best overall ablation performance, particularly on L.FoV and Pano.
- Ablation studies: The asymmetric angular loss and enhanced camera conditioning jointly outperform their ablated alternatives, especially for distorted and L.FoV domains, mitigating backprojection contraction.The combined strategies also improve angular prediction accuracy and address extreme cases visible in qualitative examples.
5. Conclusion
UniK3D is presented as a universal monocular 3D framework for pinhole, fisheye, and panoramic cameras. Its flexible backprojection design supports metric 3D estimation and generalizes to unseen cameras and scene domains, while broader training-data coverage could further improve robustness.
- Conclusion: UniK3D generalizes across pinhole, fisheye, and panoramic cameras through flexible backprojection for accurate metric 3D estimation.The framework is described as universal across diverse camera models.
- Conclusion: Expanding the diversity and coverage of training data could further enhance UniK3D’s robustness and applicability.The paper states that the current model already generalizes to unseen cameras and 3D scene domains with a fair quantity of data.
A. Architecture
UniK3D combines ViT features with camera-aware spherical representations to estimate metric 3D geometry across varying image shapes and camera models. Its camera conditioning uses learned camera tokens and attention-based feature conditioning, while ablations compare alternative conditioning and token-processing designs.
- Encoder: The ViT encoder extracts feature maps and class tokens from its last four modified backbone layers, with residual connections in self-attention and MLP blocks.The final pooling, fully connected, and softmax layers are removed before feature extraction.
- Camera representation: The first three projected tokens encode horizontal field of view and spherical-harmonic projection poles from the image dimensions.HFov is computed as 2π · σ(T0), while the second and third tokens specify the projection center.
- Camera representation: Fifteen spherical-harmonic tensors are linearly combined with processed camera tokens to construct the pencil of rays C.The harmonics use degree up to 3 while excluding the constant component.
- Radial module: Sine-encoded camera rays condition dense feature maps through Transformer Decoder layers at each resolution level.This design injects camera information into the radial reconstruction pathway.
- Complexity and resolution: UniK3D is designed to maintain robust performance across different image resolutions, unlike methods whose performance drops when image shapes differ from training.The efficiency comparison uses similar input sizes across models, although matching sizes is difficult for multiresolution architectures.
- Ablations: Addition and concatenation conditioning alternatives underperform the proposed attention-based prompt conditioning, while camera-token processing changes produce smaller differences.The ablations separately evaluate camera conditioning and whether camera tokens pass through the encoder layer.
B. Training Details
The paper specifies optimization, scheduling, augmentation, and loss settings through dedicated training tables.
- Training configuration: Training parameters and losses, including optimization, scheduling, augmentations, inputs, and weights, are reported in Tables 10 and 11.The cited passage directs these details to the corresponding tables.
B.2. Data
Training combines more than 8M samples from varied cameras and domains, while validation spans small-, large-, and panoramic-field-of-view settings. Data augmentation synthesizes distorted-camera views from pinhole imagery.
- Training datasets: The training mixture contains more than 8M samples from different cameras and domains, with diverse datasets sampled more frequently.Sequence datasets are subsampled so consecutive frames are at least half a second apart, without post-processing.
- Dataset documentation: The training and validation dataset inventories report acquisition, scene, sampling, and evaluation-distance metadata in Tables 12 and 13.The table descriptions include abbreviations for acquisition types and camera categories.
- Training datasets: Most training data comes from pinhole or rectified cameras, while distorted Mapillary samples use only pinhole calibration and are masked in camera-loss computation.This describes the camera-data composition and handling of clearly distorted images with incomplete calibration.
- Validation datasets: Validation datasets are grouped into small-FoV, large-FoV, and panoramic categories, with distorted small-FoV data synthesized and manually checked for realism.The synthetic distorted validation images are generated from ETH3D, Diode Indoor, and IBims-1.
- Camera augmentation: Camera augmentation computes a deformation field by reprojecting pinhole-derived 3D points into randomly sampled distorted camera models, then warps images using softmax-based splatting.The procedure simulates models such as Fisheye624 and radial Kannala–Brandt while preserving image details.
- Camera augmentation: The augmentation is enabled only after 10,000 training steps, once the model has learned a sufficiently reliable scale-invariant depth representation.Camera parameters for the sampled views are listed separately.
C. Additional Quantitative Results
Additional experiments examine fine-tuning on single datasets and zero-shot evaluation under different camera-information settings. The supplied passages describe the protocols and table organization but do not provide numerical outcomes.
- Fine-tuning: Fine-tuning resumes from large-scale pretrained weights and optimizer states, using KITTI or NYU as the sole training dataset for 40,000 additional steps.The protocol uses SILog loss, batch size 16, limited augmentation, and omits the asymmetric angular-loss component.
- Zero-shot evaluation: Zero-shot NYUv2 results are organized by available outputs, with separate markers for ground-truth camera use in 3D reconstruction and 2D depth inference.Missing values indicate that a model cannot produce the corresponding output.
C.2. Per-dataset Evaluation
UniK3D is evaluated independently across diverse validation datasets, including conventional and distorted-camera domains. It achieves the highest average metric overall, although it does not rank first on every dataset.
- Dataset coverage: Results are reported separately for NYUv2, KITTI, IBims-1, ETH3D, Diode Indoor, nuScenes, IBims-1Dist, ETH3DDist, Diode IndoorDist, and ScanNet++ DSLR.The cited evaluation tables cover the listed validation datasets; the Pano group is omitted because it contains only Stanford-2D3D.
- Overall performance: UniK3D achieves the highest average metric overall across the evaluated datasets.The paper notes that UniK3D does not always rank first on every individual dataset.
- Overall performance: Performance on pinhole camera models has reached a saturation point, making average cross-dataset performance especially informative.The comparison spans conventional pinhole datasets and distorted-camera datasets such as IBims-1Dist, ETH3DDist, and DiodeDist.
- Metrics: The evaluation reports absolute relative error as A.Rel percentage and root-mean-squared error in meters among the included metrics.These metrics are described as additional and more typical reporting measures.
D. Q&A
The Q&A discusses how data diversity, camera representation, and comparisons with related methods affect UniK3D’s generalization. It also reports a small δSSI result and clarifies the scope of competing approaches.
- Data and generalization: Data diversity is crucial for generalizing monocular depth estimation because scale prediction is ill-posed and sensitive to training-domain coverage.The discussion states that excessive diversity can hurt narrow domains such as KITTI, while broader domains such as NYU may benefit.
- Camera representation: Parametric camera models can produce unstable optimization because parameter changes compound on the pencil of rays.They also constrain applicability to cameras representable by the chosen model.
- Camera representation: UniK3D models any camera using spherical harmonics and radial distance instead of relying on a pinhole ray map and perpendicular depth.The method directly predicts spherical-harmonic coefficients used to generate the ray map.
- Reported results: δSSI 1 scores for rows 3 and 4 of Table 5 are 92.1 and 92.2, respectively.These values are reported as supplementary results in the Q&A.
- Related work: Related methods that remove the pinhole assumption focus on single-domain self-supervised depth estimation rather than supervised large-scale monocular 3D estimation.The distinction concerns both domain scope and the role of the camera in photometric warping losses.
E. Additional Qualitative Results
Additional qualitative results test UniK3D on distorted cameras and unconstrained in-the-wild imagery. The visualizations include movie, television, YouTube, anime, and egocentric video frames with unusual viewpoints.
- Distorted-camera validation: Additional comparisons cover ScanNet++ DSLR, IBims-1Dist, and DiodeDist Indoor validation domains with distorted cameras.Figure 5 presents qualitative comparisons for these domains.
- Distorted-camera validation: Each Figure 5 sample pairs an RGB image and relative-error map with ground truth and predicted 3D point clouds.The error map uses blue for 0% absolute relative error and red for 25%, with errors computed on ground-truth-based shifted and scaled outputs.
- In-the-wild results: UniK3D produces point clouds from single images drawn from movies, television, YouTube, and anime.The examples include Poor Things, The Revenant, Eminem, Trainspotting, Breaking Bad, Naruto, and online video frames.
- In-the-wild results: The in-the-wild examples span varied camera types and unusual viewpoints, with the rendered point-cloud view slightly tilted for 3D inspection.Figures 6 and 7 include egocentric GoPro and doorbell-camera footage among the examples.