PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPoint: Universal Monocular Metric Pointcloud from Any Camera

Botao Ye, Marc Pollefeys, Ming-Hsuan Yang, Abhijit Kundu

arXiv:2609.09394v1cs.CV

TL;DR

Monocular metric 3D reconstruction remains fragmented by fixed camera assumptions and inflexible geometric inputs. OmniPoint addresses this with a decoupled ray-distance representation, bidirectional cross-camera augmentation, and robust prior injection, achieving state-of-the-art zero-shot performance across pinhole, fisheye, and panoramic domains.

  • Problem

    Existing monocular geometry methods are typically tied to fixed camera models and cannot dynamically incorporate optional camera intrinsics or sparse-depth measurements.

  • Method

    OmniPoint decouples camera projection from scene structure with ray-distance prediction, bridges perspective and omnidirectional data through bidirectional 3D augmentation, and injects optional priors robustly.

  • Results

    OmniPoint achieves state-of-the-art zero-shot performance across pinhole, fisheye, and panoramic domains.

  • Takeaways & Limitations

    The framework provides a unified monocular geometry system that handles heterogeneous camera models and geometric priors within one approach.

  • Takeaways & Limitations

    OmniPoint still has difficulty reconstructing extremely fine structures such as thin edges and hair.

Abstract

from arXiv · show

Recovering metric 3D geometry from monocular images is a fundamental computer vision task, yet current methods remain heavily fragmented by fixed camera model assumptions and inflexible input schemes. We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors, including pinhole, fisheye, and equirectangular projections, while accommodating varying geometric priors. To overcome projection rigidity, OmniPoint abandons conventional planar depth regression. It instead adopts a decoupled ray and distance representation alongside a decoupled training objective, explicitly separating the camera projection model from the scene structure. To address the severe scarcity of training data for alternative cameras, we introduce a bidirectional augmentation strategy that explicitly bridges labeled perspective data and unlabeled omnidirectional domains in 3D space. Furthermore, to seamlessly integrate optional inputs like camera intrinsics or sparse depth without destabilizing the network through feature distribution shifts, we propose a robust information injection mechanism. This mechanism utilizes learnable input state embeddings to resolve architectural ambiguity and applies vectorized Gaussian smoothing to densify irregular measurements. Extensive experiments demonstrate that OmniPoint achieves state-of-the-art zero-shot performance across multiple benchmarks, establishing a robust new standard for unified monocular 3D reconstruction.

1 Introduction

OmniPoint addresses the fragmentation of monocular 3D reconstruction across camera models and geometric inputs with a unified, camera-agnostic framework. It decouples projection geometry from scene structure, bridges perspective and omnidirectional training domains, and injects optional priors robustly.

  • Motivation: Existing monocular geometry methods are tied to fixed camera models and cannot flexibly incorporate optional intrinsics or sparse-depth measurements.This fragmentation limits deployment across heterogeneous sensor suites, such as pinhole cameras combined with fisheye lenses and varying depth sensors.
  • Camera-agnostic representation: OmniPoint represents each 3D point as P = d · r, separating camera-dependent ray direction from camera-agnostic radial distance.The same distance prediction can therefore apply across camera types once the ray direction is known.
  • Camera-agnostic representation: A decoupled training objective independently guides ray and distance learning, preventing their errors from interfering during optimization.The paper reports that ablations validate this design for high-fidelity point clouds across diverse camera geometries.
  • Data augmentation: OmniPoint uses bidirectional 3D augmentation to connect labeled perspective data with unlabeled omnidirectional domains.Perspective-to-Any synthesizes fisheye and panoramic supervision, while Any-to-Perspective creates virtual pinhole views and reprojects pseudo-ground truth for consistency.
  • Geometric priors: Learnable input-state embeddings and vectorized Gaussian smoothing integrate optional intrinsics and sparse depth without architectural changes or unstable feature shifts.The mechanism densifies irregular measurements and supports both zero-shot monocular estimation and depth densification.
  • Results: OmniPoint reports state-of-the-art zero-shot performance across heterogeneous camera models and benchmarks, establishing a unified framework for monocular geometry estimation.Its stated scope includes pinhole, fisheye, and 360° cameras, together with intrinsics and sparse-depth priors.

2 Related Work

Prior work is divided among perspective monocular depth estimation, specialized omnidirectional models, and conditional geometry estimation. OmniPoint unifies these previously disjoint directions in one model without architectural changes.

  • Perspective monocular depth methods generalize well in the wild but primarily assume pinhole cameras and produce planar-depth outputs.
  • Omnidirectional methods target panoramas or fisheye images with camera-specific unprojection models, limiting cross-camera use.
  • Conditional geometry methods incorporate sparse depth or auxiliary cues but remain tied to pinhole cameras and conditioned inputs.
  • OmniPoint bridges these directions as a single model for pinhole depth, wide-FoV estimation, and conditional prior-aware geometry without architectural changes.

3 Method

OmniPoint predicts metric point clouds across camera models by separating camera-dependent rays from camera-agnostic distances, while augmenting scarce non-pinhole data and flexibly injecting geometric priors. Its method combines bidirectional 3D reprojection, decoupled supervision, input-state embeddings, geometric ray features, and smoothed sparse-depth inputs.

  • 3 Method: OmniPoint maps a single image and optional geometric priors to dense metric 3D points represented by predicted rays, radial distances, and a global scale.The final point map is obtained by combining the predicted ray directions and distances with the predicted metric scale.
  • Camera-Agnostic 3D Representation: Planar depth and direct pixel-wise coordinates entangle scene structure with camera projection, causing problems for wide-FoV, non-pinhole, and cross-camera estimation.
  • Camera-Agnostic 3D Representation: The ray-distance representation factorizes each prediction into a unit ray and radial distance, leaving distance invariant to the camera model.A decoupled objective separately supervises ray geometry and structural distance, using ground-truth rays in the point loss to avoid cross-contamination.
  • Bidirectional Data Augmentation: Bidirectional augmentation bridges labeled perspective and unlabeled omnidirectional data through 3D reprojection and self-training.Perspective-to-Any synthesizes fisheye and panoramic views, while Any-to-Perspective samples virtual pinhole patches for pseudo-label generation.
  • Optional Geometric Inputs: Optional priors are handled through input-state embeddings, encoded ray maps, and normalized Gaussian-smoothed sparse depth rather than raw irregular measurements.The training objective additionally enforces metric scale, normal and local consistency, and sky masking.

4 Experiments

OmniPoint is evaluated across pinhole, fisheye, and panoramic settings, with experiments covering metric depth, geometric conditioning, output representations, loss design, and bidirectional augmentation. Results show competitive or improved performance across camera models and demonstrate benefits from the proposed representation, conditioning mechanisms, and data strategy.

  • Experimental Setup: OmniPoint is evaluated on 29 labeled datasets spanning pinhole, fisheye, and panoramic imagery, using standardized relative error and robust-inlier metrics.Baselines are reevaluated with identical input formats and a standardized metric computation pipeline.
  • Relative Geometry and Depth: OmniPoint preserves strong perspective performance with 96.8 δ1 and surpasses baselines in large-FoV fisheye settings affected by projection distortion.Qualitative comparisons also show geometrically consistent walls and roads, plus finer depth-map and point-cloud details.
  • Metric Depth: 83.5 Mean achieves state-of-the-art zero-shot metric depth, exceeding UniK3D at 82.6 Mean and MoGe V2 at 77.5 Mean.Conditioning on intrinsics improves performance, while sparse depth raises it to 98.0 Mean.
  • Output Representations: The decoupled Ray+D representation outperforms direct XYZ prediction across perspective, fisheye, and panoramic images.Reported relative errors are 4.04 vs 4.28 for perspective, 6.37 vs 6.74 for fisheye, and 5.79 vs 6.28 for panoramic images.
  • Decoupled Loss: The P + GT Ray loss yields more accurate and stable point prediction than disjoint D + Ray supervision or naive entangled P loss.D + Ray reaches 7.88 Rel, while the compared entangled setting reports 5.65 vs 5.55 Rel.
  • Bidirectional Augmentation and Conditioning: Perspective-to-Any synthesis and Any-to-Perspective self-training jointly improve panoramic geometry under scarce non-pinhole supervision.Perspective-to-Any alone improves relative error from 8.45 Rel to 6.21 Rel, while Gaussian smoothing and state embeddings support robust sparse conditioning.

5 Conclusion

OmniPoint unifies monocular geometry estimation across arbitrary camera models and optional geometric priors through a ray-distance representation, bidirectional augmentation, and prior injection. Experiments report state-of-the-art zero-shot performance across pinhole, fisheye, and panoramic domains.

  • Conclusion: OmniPoint jointly handles arbitrary camera models and diverse input priors with a universal ray-distance representation, bidirectional augmentation, and prior-injection mechanism.The framework integrates sparse depth and camera intrinsics when available.
  • Conclusion: OmniPoint achieves state-of-the-art zero-shot performance across pinhole, fisheye, and panoramic domains.The conclusion presents this as a robust new standard for 3D vision.

A Training Datasets

OmniPoint is trained with diverse labeled and unlabeled data covering multiple camera models, scene types, and real or synthetic sources. Bidirectional augmentation addresses limited fisheye and panoramic supervision through synthetic views and pseudo-labeling.

  • Dataset Diversity: The datasets cover indoor, outdoor, mixed, object-centric, and large-scale scenes using both real and synthetic sources.Synthetic data supplies clean ground-truth geometry, while real data reduces domain gaps and supports in-the-wild generalization.
  • Data Augmentation: Perspective-to-Any converts pinhole data into synthetic wide-FoV views with ground truth, while Any-to-Perspective uses unlabeled panoramic videos for pseudo-labeling.This strategy is applied to mitigate scarce fisheye and panoramic supervision.

B More Implementation Details

The implementation uses a DINOv2 ViT-Large encoder with DPT-style prediction heads and a three-stage training schedule. Geometric conditions are introduced during later stages, with sparse-depth sampling and fixed evaluation settings for benchmarking.

  • Backbone and Prediction Heads: DINOv2 ViT-Large serves as the encoder, while DPT-style heads predict rays, distances, masks, and metric scale.The metric head is a two-layer MLP operating on a learned metric token.
  • Training Strategy: Training proceeds through perspective pretraining, camera-universal finetuning with bidirectional augmentation, and mask-head refinement.The third stage freezes all parameters except the mask head.
  • Training Strategy: The model uses Perspective-to-Any sampling at 2% per batch during camera-universal finetuning.The finetuning stage uses separate learning rates for the backbone and heads.
  • Geometric Conditioning: Sparse depth is sampled at 0.05%–0.1% per pixel during training, with geometric conditions enabled in later stages at 90% probability.Evaluation fixes sparse depth to 1,000 points per image for consistent benchmarking.

C More Experimental Results

OmniPoint’s explicit ray-distance representation is evaluated against Spherical Harmonics across standard, wide, and full 360° fields of view. The comparison identifies sharper geometric reconstruction as a key advantage over SH-based ray fields.

  • C More Experimental Results: Spherical Harmonics provides a continuous ray-field representation but tends to introduce unwanted smoothing artifacts.This limits its ability to preserve abrupt geometric changes.
  • C More Experimental Results: OmniPoint’s representation is associated with improved geometric sharpness in depth and point-cloud predictions, particularly after synthetic-data finetuning.The paper contrasts this sharpness with the smoothing behavior observed for SH parameterization.
  • C More Experimental Results: Table 10 compares OmniPoint’s explicit ray-distance pairing with Spherical Harmonics across S.FoV, L.FoV, and full 360° images.The evaluation spans multiple camera-model regimes.

D Limitations

OmniPoint improves geometric sharpness but remains limited in reconstructing extremely fine structures. Its wide-FoV training data is also modest relative to the abundance of real-world fisheye and 360° imagery.

  • D Limitations: OmniPoint still has difficulty reconstructing extremely fine structures such as thin edges and hair.The limitation persists despite improved geometric sharpness, particularly after synthetic-data finetuning.
  • D Limitations: The scale of OmniPoint’s wide-FoV data remains modest compared with abundant real in-the-wild fisheye and 360° imagery.The authors identify dataset expansion and camera-geometry-specific self-supervised learning as future directions.
Loading 2609.09394v1…