PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields

Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa, Anand Bhattad

arXiv:2610.31645v1cs.CV

TL;DR

Pinhole assumptions can distort wide-angle monocular reconstructions, and existing camera-aware ray fields use a single basis for broad projection structure and finer variation. OmniRay composes radial and residual ray representations with predicted radial distance, improving over evaluated baselines in ray estimation and monocular 3D reconstruction across camera settings.

  • Problem

    Pinhole-focused monocular models can distort wide-angle reconstructions, while camera-aware ray fields use one basis for overall projection structure and finer variation.

    Source: §1 · p. 2

  • Method

    OmniRay predicts per-pixel rays and up-to-scale radial distances from one image, composing a monotone radial base with spherical-harmonic and free-form residuals.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    OmniRay improves over evaluated baselines in ray-field estimation and monocular 3D reconstruction; on seen camera types, its mean ray error is 0.14° versus 6.34° for UniK3D.

    Source: §1 · p. 2, §4.3 · p. 7

  • Takeaways & Limitations

    The composite representation handles diverse central camera geometries without camera-type input or model selection, with a shared distance module producing a point cloud in one forward pass.

    Source: §5 · p. 10

    Aerial panoramas with horizons near the top of the frame can yield rays covering only a bounded, forward-facing region rather than the full viewing sphere.

    Source: §5 · p. 10

Abstract

from arXiv · show

Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models remove this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry can be viewed as having a hierarchy of complexity for representing rays. We present OmniRay, a feed-forward model whose Ray Module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form residual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse camera geometries. Given a single image, OmniRay predicts a ray direction and an up-to-scale radial distance for every pixel, from which a 3D point cloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.

1 INTRODUCTION

Pinhole-focused monocular models can distort wide-angle reconstructions, while existing camera-aware ray fields use one basis for both broad projection structure and finer variation. OmniRay combines a radial base with two residuals and improves over evaluated baselines across ray-field estimation and monocular 3D reconstruction.

1 INTRODUCTION

  • Wide-angle images expose a limitation of pinhole-focused models, while rectification can crop the field of view or stretch peripheral regions.A pinhole cannot represent rays at or beyond 90° from its optical axis.Source: §1 · p. 2
  • Existing camera-aware ray fields can handle diverse cameras without metadata, but use one basis for both overall projection and finer variations.For approximately radially symmetric lenses, a smooth radial function can capture the structure, with spherical harmonics complementing it.Source: §1 · p. 2
  • OmniRay predicts per-pixel rays and radial distances from one image using a monotone radial base, a spherical-harmonic residual, and a free-form residual.Its output is an up-to-scale 3D reconstruction without intrinsics or camera-type input.Source: §1 · p. 2
  • OmniRay improves over evaluated baselines in ray-field estimation and monocular 3D reconstruction across seen and held-out scenes, datasets, and camera configurations.The paper also evaluates representation expressiveness with an oracle analysis and tests proposed components through ablations.Source: §1 · p. 2

2 RELATED WORK

Prior camera calibration uses fixed-form models or more flexible generic models, while learning-based approaches increasingly predict dense pixel-to-ray geometry. The representation question remains how to balance compactness and expressiveness across camera geometries.

2 RELATED WORK

  • Classical calibration fits low-dimensional, camera-specific models, while generic models capture more complex lens behavior with higher-dimensional calibration.Examples include pinhole distortion models, Kannala–Brandt wide-angle models, and unified omnidirectional models.Source: §2 · pp. 2–3
  • Learning-based calibration ranges from regressing camera parameters to predicting dense cues such as per-pixel up-vectors, latitude, or rays.Neural Ray Surfaces learn a free-form pixel-to-ray mapping from video observations.Source: §2 · p. 3
  • Across calibration and monocular reconstruction, prior work leaves open how to represent pixel-to-ray mappings compactly while retaining expressiveness across camera geometries.This question is raised explicitly for learned calibration and for camera-agnostic monocular geometry.Source: §2 · p. 3
  • Camera-aware monocular methods model per-pixel rays, use spherical harmonics, or transform images to a common equirectangular representation.Depth Any Camera and UniDAC require input intrinsics for their transformations, whereas UniK3D handles diverse camera geometries without metadata.Source: §2 · p. 3

3 METHOD

OmniRay predicts a ray direction and positive radial distance for each pixel, then combines them into a camera-frame point. Its ray field is built through a radial base and two successive residual corrections, while its training objective supervises both rays and point clouds.

3 METHOD

  • A shared DINOv2 encoder drives ray and distance branches that predict a unit ray and positive radial distance per pixel from a single image.Using radial distance rather than perpendicular depth keeps the quantity well-defined as rays approach or pass 90° from the optical axis.Source: §3 · p. 4

3.1 RAY MODULE

  • The monotone radial base maps image radius to incidence angle to model approximately radially symmetric central cameras, with 16 knots spanning radii 0 to 1.3.Its angle is not capped at 90°, so the base can represent rays passing the equator.Source: §3.1 · p. 4
  • A shared spherical-harmonic correction handles smooth angular deviations, and a dense-feature convolutional residual captures remaining local variation through successive rotations on the sphere.The spherical-harmonic basis is finite-order; residual corrections are applied to the preceding ray field, and their tangent-vector norms set rotation angles.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 DISTANCE MODULE

  • The Distance Module reuses the encoder, fuses four intermediate blocks with a DPT decoder, and predicts positive radial distance to scale each ray into a point cloud.The resulting camera-frame point cloud is produced in one forward pass.Source: §3.2 · p. 5

3.3 TRAINING OBJECTIVE

  • Training combines angular ray supervision with a point-cloud loss after normalizing each cloud by its mean point norm to remove undetermined global scale.Deep supervision assigns responsibility to intermediate ray stages, while region weighting emphasizes high-error areas, including the image periphery.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments test representation ceilings, train across panoramic datasets and camera types, and evaluate OmniRay on seen and held-out settings. Results favor the staged radial–spherical-harmonic–free-form representation, with random-lens augmentation helping held-out camera performance.

4.1 ORACLE TEST: REPRESENTATION CEILINGS

  • Among compact representations, only the composition stays below 0.25° on all three analytic geometries; the radial profile’s equirectangular error remains 22.8° as knots increase.The spherical-harmonic field removes that ceiling but remains two orders of magnitude behind the radial profile on fisheye; composition retains at least the radial base’s capacity because its residuals can vanish.Source: §4.1 · p. 6

4.3 BENCHMARK AND ABLATION

  • On seen camera types, OmniRay achieves 0.14° mean ray error and 0.0179 relative CD, versus UniK3D’s 6.34° and 0.0760 and Depth Anything 3’s 33.03° and 0.1999.On the held-out Nayar catadioptric camera, baselines exceed 75° while OmniRay reaches 1.12° and 0.0131 CD; on PanoCity, OmniRay reaches 1.35° ray error versus 21.22° and 50.53° for UniK3D and Depth Anything 3.Source: Table 2 · p. 7, §4.3 · p. 7

4.2 DATASETS AND CAMERA TYPES

  • Training uses 2D3DS and 360Loc, with held-out scenes, unseen PanoCity and CO3Dv2 domains, and GoPro fisheye and Nayar catadioptric configurations withheld from training.The evaluation reports ray-field, radial-distance, and point-cloud metrics; random-lens augmentation samples synthetic central cameras while rejecting profiles near held-out geometries.Source: §4.2 · pp. 6–7, §4.3 · p. 7

4.3 BENCHMARK AND ABLATION

  • Without random-lens augmentation, seen-camera error remains 1.39°, but held-out catadioptric and GoPro errors rise to 30.87° and 7.97°; augmentation restores them to 1.12° and 5.53°.The augmented model also improves seen-camera error to 0.14°; the authors interpret the results as indicating that random-lens sampling matters for extending the representation beyond training camera configurations.Source: §4.3 · pp. 7–8
  • On unseen panoramic examples, OmniRay preserves scene structure where UniK3D breaks the road across the image and Depth Anything 3 fails to recover the ground structure.The qualitative comparisons also report coherent reconstructions on in-the-wild panorama, fisheye, and pinhole photographs without camera intrinsics or camera-type input.Source: Figure 4 · p. 8, §4.3 · p. 9, Figure 6 · p. 9
  • Adding the spherical-harmonic residual improves panoramic and held-out accuracy, while the free-form residual further improves panorama, fisheye, and catadioptric results in the full staged model.The full staged model reaches 0.399° panorama, 1.322° fisheye, and 1.538° catadioptric error, and outperforms both jointly trained three-term variants across all five configurations.Source: §4.3 · p. 10
  • The full model remains competitive on pinhole cameras at 0.418°, while VGGT initialization attains the best pinhole error, 0.330°, but performs worse on the other four configurations.The authors suggest this trade-off may reflect VGGT’s perspective-image pretraining favoring narrow-field projections, rather than a model-capacity limitation.Source: §4.3 · p. 10

5 CONCLUSION

OmniRay composes a monotone radial base with spherical-harmonic and free-form residuals to represent varied camera geometries. Aerial panoramas with high horizons expose a limitation in the training distribution, motivating broader orientation and altitude coverage.

5 CONCLUSION

  • A radial base, spherical-harmonic residual, and free-form residual are composed as rotations on S2 to represent camera geometry at increasing levels of complexity.The composition adapts to different camera geometries without camera-type input or model selection.Source: §5 · p. 10
  • On an aerial panorama with the horizon near the image top, predicted rays cover a bounded forward-facing region rather than the full viewing sphere.Training panoramas are captured near ground level, with horizons approximately aligned; the example suggests generalization also depends on represented viewing altitudes.Source: §5 · p. 10, Figure 7 · p. 11
  • A proposed next step is to extend OmniRay to multi-view reconstruction and include more varied camera orientations and altitudes in training.Source: §5 · p. 11

A.1 IMPLEMENTATION AND TRAINING

OmniRay uses a shared image encoder with separate ray and distance modules. Training first fits the ray components in sequence, then attaches the distance module for end-to-end training.

A.1 IMPLEMENTATION AND TRAINING

  • The DINOv2 ViT-L/14 encoder feeds radial, spherical, free-form, and distance heads, with the radial head initialized to a canonical camera and residual heads zero-initialized.The radial and spherical heads use global-token features; the free-form head decodes dense features, and the distance module uses a DPT decoder.Source: §A.1 · p. 14
  • Training proceeds by fitting the Ray Module alone with its three components activated in order, then attaching the Distance Module for full-model end-to-end training.Source: §A.1 · p. 14
  • The training setup balances camera types, caps images at 8192 tokens, and supervises 4096 pixels sampled round-robin across occupied cells of a 32×32 grid.The second phase uses AdamW with weight decay 0.01 and cosine decay after warm-up, on 8 NVIDIA A40 GPUs.Source: §A.1 · p. 14

A.2 EXPERIMENTAL DETAILS

Experiments use analytic rays from sampled cameras and calibration-provided ground-truth rays for two real datasets. The sampling protocol excludes configurations close to the held-out GoPro and catadioptric geometries.

A.2 EXPERIMENTAL DETAILS

  • Oracle fitting measures each representation’s approximation capacity for a single camera; its fitted error lower-bounds the error of any predictor built on that representation.Each row is an independent fit under a fixed optimization budget, with multi-start sweeps over axis sign, roll, and, where applicable, focal length.Source: §A.2 · p. 14
  • Synthetic cameras sample viewing orientation, aspect ratio, principal-point offset, sensor boundary, and incidence angles; panoramic cameras instead sample vertical field of view, azimuth, and latitude stretch.Ground-truth rays and distances follow analytically from sampled cameras; PanoCity and CO3Dv2 use dataset calibration.Source: §A.2 · p. 14
  • Training and evaluation split 2D3DS by areas 1–4 versus 5–6, and 360Loc by Atrium and Piatrium versus Concourse and Hall.Draws near held-out GoPro and catadioptric geometries are rejected; panoramic sampling keeps the horizon gravity-aligned as in the source captures.Source: §A.2 · p. 14

A.3 ADDITIONAL RESULTS

Additional results report metrics across camera types, datasets, and aggregate seen and held-out splits, with qualitative reconstructions on seen test scenes and in-the-wild examples. The camera-sampling illustration depicts continuously varying projections, while the table captions specify evaluation scope and conventions.

A.3 ADDITIONAL RESULTS

  • Tables 4–6 report metrics per camera type, per dataset, and over aggregate seen and held-out splits.Table 6 covers a 4500-sample benchmark; its held-out geometries are the catadioptric mirror and GoPro fisheye.Source: §A.3 · p. 14, Table 6 · p. 20
  • Figures 9 and 10 show reconstructions on 2D3DS and 360Loc test scenes that share training camera families but contain no training scenes.Figure 4 provides the unseen-dataset comparison; Figures 9 and 10 complement it with seen-dataset scene visualizations.Source: §A.3 · p. 14, Figure 9 · p. 16, Figure 10 · p. 17
  • Random-lens augmentation varies viewing direction, radial profile, and sensor region, spanning perspective crops, wide-angle and fisheye views, off-center and anisotropic sensors, and annular domains.Samples close to either held-out geometry are rejected, so no fixed image-position-to-ray map fits more than one sampled projection.Source: Figure 8 · p. 15
  • Table 5 reports per-dataset results, with CO3Dv2's zero-shot axis being domain rather than camera because its images are perspective and fisheye only.Its object-masked COLMAP ground truth covers roughly 12% of pixels; Figure 11 adds in-the-wild visualizations.Source: Figure 11 · p. 17, Table 5 · p. 19

Related papers

Loading 2610.31645v1…