Any3R: Feedforward 3D Reconstruction from Any Camera Type

1. Introduction

• Existing feed-forward models read frames as pinhole images, while prior adaptations support fixed camera models or require conversion or retraining for unseen cameras.Mixing camera types within one reconstruction is a further challenge; a pinhole intrinsic matrix cannot describe fields of view of 180° or more.Source: §1 · pp. 1–2
• Pixel-index positional encoding and equal per-pixel loss both misrepresent viewing direction or scene coverage when the lens changes.A fisheye pixel near the edge covers a different amount of the scene than one near the center.Source: §1 · p. 2
• Any3R conditions positional encoding and training loss on every pixel’s ray direction, enabling reconstruction from mixed central cameras without rectification or lens-specific retraining.Its angular coordinates use ray angles rather than pixel indices, while solid-angle weighting accounts for how much of the scene each pixel sees.Source: §1 · p. 2
• Any3R gives the most accurate reconstruction and camera poses on wide-angle and panoramic lenses, and is on par with the strongest pinhole baseline at 90°.The comparison covers three public datasets and thirteen camera models, with each method using its own predicted camera poses; Any3R’s margin over pinhole models is larger on wider lenses.Source: §1 · p. 2

2.1. 3D Geometry Foundation Models

• Geometry foundation models progress from pointmap regression to models that jointly predict poses, depth, pointmaps, and tracks, but still assume pinhole input.Some extend pairwise models with global stages, streaming latent states, or set-level attention; MapAnything and OmniVGGT optionally condition on camera parameters or depth.Source: §2.1 · pp. 2–3

2.2. 3D Reconstruction for Different Camera Types

• Any3R uses per-view ray fields to give patches angular coordinates for rotary encoding and weights point loss by pixel solid angle.The overview depicts equal-count panorama cells covering unequal areas of the sphere.Source: Figure 3 · p. 3
• Camera-specific reconstruction methods use spherical convolutions, resampled charts, or panoramic and fisheye depth models, with each approach fixing one projection.Rectification to a rectilinear grid is lossy at wide fields of view and impossible beyond 180°.Source: §2.2 · p. 3
• Recent feed-forward adaptations handle camera variation through retraining, tokens, adapters, pairwise alignment, cubemaps, or multi-lens renderings, but retain pixel-based rotary coordinates.RayTun3R adapts per sequence at test time, while G-ray parameterizes rotary phase by camera-local ray angles.Source: §2.2 · p. 3

3.1. Preliminary

• Each view supplies a sampled unit-ray map for its central camera; Any3R predicts a camera-to-world pose and a per-pixel distance along each ray.The ray field is the camera representation, with no lens type or distortion coefficient supplied to the network.Source: §3.1 · pp. 3–4

3.2. Angular Coordinates

• Rotary patch positions use elevation and azimuth from each patch’s mean ray, evaluated per view so shared frequencies encode the same angle across mixed cameras.The released encoding also adjusts coordinate extents for each view’s field of view.Source: §3.2 · p. 4

3.3. Solid-Angle Weighting

• Point-loss weights incorporate each pixel’s solid-angle area, making supervision integrate over the sphere rather than over each lens’s pixel grid.Normalizing by the valid-pixel mean solid angle keeps the loss magnitude and learning-rate schedule unchanged.Source: §3.3 · pp. 4–5
• Table 1 compares reconstruction accuracy and pose AUC across eight camera models and indoor and outdoor datasets, with every method fusing views using its own predicted poses.Acc measures nearest-ground-truth-point distance; AUC measures pairwise pose error up to 15° and 30°.Source: Table 1 · p. 5

4.1. Setup

• Evaluation uses 24 held-out scenes per dataset with three views each, rendered through 13 central camera models, including five held out from training.Every method fuses views using its own predicted poses; evaluation measures point-cloud accuracy and pairwise pose accuracy.Source: §4.1 · p. 5

4.2. Comparison with the State of the Art

• Any3R is more accurate and better posed than camera-aware methods on supported cameras and geometry foundation models except for indoor-pinhole AUC at 15°, where π3 leads.Figure 4 compares methods fused with their own predicted poses; n/a marks unsupported camera inputs.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3. Zero-Shot Real Lenses

• On real fisheye lenses, Any3R is most accurate and best posed on FIORD, and second only to Wid3R on ScanNet++, while Any3R∗ falls far behind on both.Neither dataset is in the training data; Table 2 reports zero-shot reconstruction and pose results.Source: Table 2 · p. 6, §4.3 · p. 7

4.4. Ablation

• Per-view ray fields give the largest indoor pose gain, while solid-angle weighting alone gives a smaller gain by counting scene directions rather than pixels.Per-view evaluation computes angular coordinates from each view’s own ray field; the joint ablation removes the ray map and its associated modules.Source: §4.4 · pp. 7–8
• The pose gains trade off against depth accuracy, with per-view evaluation raising Matterport3D mean log10 depth error from 0.033 to 0.045.On PanoCity, per-view evaluation lowers median rotation error but also lowers AUC.Source: §4.4 · p. 8

• Geometry foundation models typically assume pinhole cameras, including in their positional encoding and training objectives.Source: §5 · p. 8
• Any3R uses angular coordinates and solid-angle weighting on dense ray fields to adapt each camera to the backbone.Source: §5 · p. 8
• Any3R reconstructs from arbitrary combinations of central cameras without calibration search, image undistortion, or lens-specific retraining.Source: §5 · p. 8
• Across thirteen central camera models and three datasets, Any3R achieves state-of-the-art reconstruction accuracy using its own predicted poses.Source: §5 · p. 8
• The paper identifies heterogeneous crowdsourced and archival imagery from smartphones, fisheye surveillance cameras, and panoramic systems as an application scope.Source: §5 · p. 8

A. Evaluation Details

• The main evaluation uses 24 scenes per dataset, three views per scene, and thirteen camera models, with five models held out from training.Structured3D is reported only in the supplement because its scene views share no content and relative pose cannot be recovered.Source: A. Evaluation Details · p. 12
• Point-cloud accuracy is mean distance to the nearest ground-truth point after similarity alignment and ICP; pose AUC uses the larger relative rotation or translation-direction error.ICP correspondence limits are 0.1 m indoors and 1.0 m on PanoCity; pose errors are evaluated up to specified angular thresholds.Source: A. Evaluation Details · p. 12
• For baselines with two input options, the reported table uses the option with lower median accuracy in each cell, while a supplement lists both.MapA-ray receives calibrated per-pixel rays after MapAnything preprocessing fits pinhole intrinsics.Source: A. Evaluation Details · p. 12
• Real-lens testing uses calibrated Kannala–Brandt rays on ScanNet++ validation views and FIORD views registered to laser-scan ground truth.FIORD triplets follow a fixed selection rule requiring sufficient view separation and overlap.Source: A. Evaluation Details · p. 12

B. Further Results on the Eight Camera Models

• On Matterport3D, Stanford2D3DS and PanoCity, Any3R is more accurate on wide than narrow cameras, while perspective-native baselines do not improve toward wider lenses.On Structured3D, where views share no content, every method improves on wide cameras; Any3R gains the most and leads every cell from 180° onward.Source: B. Further Results on the Eight Camera… · p. 12, Table 7 · p. 14
• With three different cameras in one forward pass, Any3R is best on all three datasets for Brown–Conrady, double-sphere and cropped equirectangular inputs.For pinhole, KB4 and full equirectangular inputs, PanoVGGT has the lowest median on all three datasets; Any3R ties on two and trails on PanoCity.Source: B. Further Results on the Eight Camera… · p. 12, Table 8 · p. 14
• In qualitative comparisons, Any3R keeps the room intact across lenses, while perspective-native methods flatten or fold it outside narrow views.The panoramic baseline keeps the room intact only on full equirectangular input.Source: B. Further Results on the Eight Camera… · pp. 12–13

C. Camera-Aware Methods per Camera

• On Structured3D, MapA-ray is more accurate than Any3R on pinhole and MEI cameras, while Wid3R is scored only on camera classes its parameterization can represent.The comparison omits nine of thirteen camera models for Wid3R, including four 200° lenses its classes cannot represent.Source: C. Camera-Aware Methods per Camera · pp. 13–14

D. Per-Cell Accuracy on All Thirteen Cameras

• The five added cameras separate radial mapping from angular extent and distinguish panoramic extent from the equirectangular grid.Equisolid and stereographic pair with cameras at the same field of view; cylindrical and equirectangular both cover 360° horizontally but map elevation differently.Source: D. Per-Cell Accuracy on All Thirteen… · p. 14
• Across all thirteen cameras, Any3R has lower error at every wider field of view than at 90°, and the best pinhole model fares worse against it as field of view widens.Tab. 15 averages accuracy by field of view.Source: D. Per-Cell Accuracy on All Thirteen… · p. 14
• On the three paired-pool datasets, Any3R is best in 32 of 39 cells and tied in six; on Structured3D, it is best in nine of 13 and tied in three.It trails π3 on the Stanford2D3DS pinhole cell and trails VGGT-Ω and MapAnything on the Structured3D pinhole cell.Source: D. Per-Cell Accuracy on All Thirteen… · p. 14
• Cylindrical input is the main format-specific weakness: Any3R’s accuracy is worse there than on equirectangular input in all four datasets.Equirectangular images make up 83.6% of the training mixture, and cylindrical is the only wide PanoCity cell where Any3R ties rather than leads.Source: D. Per-Cell Accuracy on All Thirteen… · p. 14
• At fixed field of view, Any3R is best in 15 of 16 radial-mapping cells and ties π3 in the remaining Structured3D Equisolid cell.The paired table also reports each arm’s pose error, so its accuracy comparison does not isolate the mapping’s effect.Source: D. Per-Cell Accuracy on All Thirteen… · p. 14

E. Per-Point Error

• On wide lenses, perspective-native baselines make smooth but confidently wrong surfaces, bent into shells or folded inward, while Any3R’s error stays low across the room.The per-point visualization shows that pose errors can move whole walls of points in a fused cloud, not merely scattered points.Source: E. Per-Point Error · pp. 14–15, Figure 9 · p. 21

F. The Ray Map Withdrawn

• Withdrawing the ray map raises error on every dataset; an approximate predicted map recovers only part of the gap on Matterport3D and performs worse than the fallback elsewhere.Direction errors move decoded points in proportion to their distance, with the largest effect on PanoCity; on equirectangular input, the fallback is the true ray map and results are indistinguishable.Source: F. The Ray Map Withdrawn · p. 15, Table 19 · p. 21

G. The Camera Models and How Their Images Are Made

• Each camera model supplies a ray map, after which the downstream representation has the same tensor shape; pixels outside an invertible domain are marked invalid rather than assigned fabricated rays.The benchmark’s models include varied projections, and the validity mask records which pixels lack valid rays.Source: G. The Camera Models and How Their… · p. 16, Figure 8 · p. 17

H. Accuracy Against Field of View

• Any3R’s error stays below its own 90° baseline at every wider field of view, while every perspective-native method rises above its own baseline.The comparison normalizes each method by its own 90° accuracy and averages across the three paired-pool datasets.Source: H. Accuracy Against Field of View · p. 18, Table 15 · p. 19

I. Real Pinhole Imagery

• On real DL3DV pinhole photographs, Any3R has the lowest mean per-scene median rotation error of seven methods and the second-highest AUC at 30°.DA3 leads AUC at 5°; DL3DV comprises real video without panoramic origin or resampling, and Any3R is at VGGT’s level on every measure.Source: §I · p. 19
• DL3DV supplies no dense depth ground truth, so the reconstruction comparison is by eye and uses sparse tracks only to place each cloud.Benchmark-rendered pinhole frames are inscribed discs, unlike the full-rectangle DL3DV frames; accuracy is computed over observed points only.Source: §I · pp. 20–21

J. Matched Coverage

• With coverage matched by cropping, Any3R is best on both cropped equirectangular datasets and ties π3 on four equidistant and KB4 crop cells.It trails on four narrow pinhole and Brown–Conrady crop cells, where VGGT-Ω or π3 has the lowest median.Source: J. Matched Coverage · p. 21, Table 21 · p. 22
• The wide-end advantage survives removal of coverage differences, but does not extend to narrow pinhole crops; withdrawing the ray field raises error several times in every cell.Each crop is scored against only the ground-truth points it can see.Source: J. Matched Coverage · pp. 21–22
• Across fifty camera-and-crop pairs, field of view and error have rank correlation −0.57, but the trend is non-monotonic.The median error is lowest at the 0.75 crop on both datasets and rises slightly at the full frame; the narrowest crops also weaken pose when views lose shared content.Source: J. Matched Coverage · pp. 22–23

K. Training Convergence

• Final depth is below the panoramic baseline on three of four datasets and level on the fourth; final pose is slightly under it on three.The comparison uses an earlier full run evaluated on equirectangular input only.Source: K. Training Convergence · p. 23
• Under the equirectangular-input-only protocol, Fig. 17 shows final depth below the released panoramic baseline on three of four datasets and level on the fourth, while final pose is slightly under it on three.Figure 17 shows per-epoch pose accuracy and depth error for an earlier full run from the same initialization, trained on eight A100s, against the released panoramic baseline on the same protocol.Source: K. Training Convergence · p. 23
• Any3R is presented as better across camera models, not as a better equirectangular model than one trained specifically for equirectangular input.Source: K. Training Convergence · p. 23

L. The Frequency Split

• At 180° and 200° the constant-angle arm had the lowest Matterport3D median, while the released split was nearer the field-of-view arm; at 360° the three medians were within 0.001 on two indoor datasets.At 90°, the constant-angle arm also had the lowest median on both indoor datasets.Source: L. The Frequency Split · p. 23
• One coordinate unit varies by a factor of 1.000 in elevation and azimuth on the constant-angle half, versus 2.82 and 5.74 on the other half.Counting individual cameras rather than model families gives 5.92 and 12.01 for the other half.Source: L. The Frequency Split · p. 24
• 14.0% and 15.9% worse medians at 180° and 200° for the split versus constant-angle arm, with no cell improved; the constant-angle arm’s pose accuracy collapses while the split’s does not.Source: L. The Frequency Split · p. 26, Table 22 · p. 27
• The split did not meet the strict-improvement criterion, and the constant-angle arm again had the lowest median at 180° and 200° on every dataset with predicted poses.The split was released for its pose behavior, with its geometry shortfall recorded.Source: L. The Frequency Split · p. 27