Source-linked AI summary
Unified Panoramic Geometry Estimation via Multi-View Foundation Models
Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer, Nando Metzger, Denis Rozumny, Konrad Schindler, Nikolai Kalischek
TL;DR
Single-image panoramic reconstruction must extend perspective geometry models to full 360° scenes despite limited panoramic data and equirectangular distortions. PaGeR uses cubemap-based adaptation, mixed perspective/panoramic training, and unified multi-task decoding, achieving state-of-the-art zero-shot performance across indoor and outdoor scenes. Its remaining limitations include unreliable predictions on reflective or transparent surfaces and occasional cubemap boundary misalignments.
Problem
Panoramic geometry estimation needs full 360° reconstruction, but panoramic datasets are scarce and equirectangular projections introduce severe distortions.
Method
PaGeR adapts a perspective 3D foundation transformer with cubemap representation, mixed panoramic/perspective training, and coupled heads for depth, normals, and sky masks.
Results
PaGeR achieves state-of-the-art zero-shot panoramic reconstruction across diverse indoor and outdoor environments.
Takeaways & Limitations
The framework transfers perspective 3D priors to unified panoramic geometry estimation and remains compatible with alternative geometric transformers.
Takeaways & Limitations
PaGeR can be unreliable on specular, reflective, or transparent surfaces and may produce subtle boundary misalignments in rare complex scenes.
Abstract
from arXiv · showhide
Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available $\href{https://github.com/prs-eth/PaGeR}{\text{here}}$.
1 Introduction
Perspective foundation models have advanced monocular and multi-view geometry estimation, but panoramas require models that handle full 360° context despite scarce data and projection distortions. PaGeR adapts these models through cubemaps, hybrid training, and unified multi-task decoding.
- Panoramic images provide full 360° context, whereas perspective models require multiple limited-field-of-view images to capture complete environments.
- Scarce metrically accurate panoramic datasets cause existing models to overfit small datasets and struggle on unseen scenes.
- Equirectangular projection introduces severe geometric distortions that complicate panoramic geometry estimation.
- PaGeR repurposes perspective foundation models using six-face cubemaps and hybrid perspective/panoramic training.
- A shared transformer backbone with specialized heads jointly predicts SI depth, metric scale, surface normals, and sky masks in one forward pass.
- PaGeR reports zero-shot generalization to unseen indoor and outdoor scenes and introduces ZüriPano for zero-shot evaluation.
2 Method
The method represents panoramas as synchronized cubemap views processed by a perspective multi-view transformer, then adds spherical adaptations and coupled geometric task heads. Joint decoding and mixed training preserve perspective priors while producing globally consistent panoramic geometry.
- Panoramic Representations: ERP maps spherical coordinates to a planar grid but causes polar stretching and a domain shift from perspective imagery.
- Panoramic Representations: Cubemaps divide the sphere into six standard 90° perspective faces, providing uniform sampling aligned with perspective-model priors.
- Geometry Transformer Backbone: DA3 supplies a vision-transformer encoder and dense-prediction decoder with intra-image and cross-image attention for multi-view spatial reasoning.
- Panoramic Adaptation: Fixed camera parameters and cross-view attention synchronize cubemap features, while cross-face padding reduces decoder boundary artifacts.
- Training: Mixed co-training alternates six-face panoramic batches with single-view perspective batches to preserve perspective priors and mitigate the sim-to-real gap.
- Multi-Task Geometric Decoding: Specialized heads jointly decode scale-invariant depth, metric depth, surface normals, and sky masks in a single forward pass.
- Multi-Task Geometric Decoding: The depth branch combines confidence-aware pixel loss, edge-aware gradient loss, and normal consistency to preserve accuracy, discontinuities, and geometric alignment.
- Multi-Task Geometric Decoding: The normal branch uses depth initialization and combines cosine similarity with perceptual loss to promote sharp edges.
3 Experiments
PaGeR is evaluated across indoor and outdoor panoramic geometry tasks, with experiments covering depth, metric scale, surface normals, qualitative reconstruction, and ablations. It consistently outperforms existing methods, including under zero-shot outdoor evaluation, while ablations identify key architectural and training choices.
- Quantitative Comparison: PaGeR consistently outperforms existing methods across indoor Matterport3D360 and Stanford2D3DS and outdoor ZüriPano depth benchmarks.The evaluation includes both scale-invariant and metric depth settings.
- Quantitative Comparison: 9.36 AbsRel on ZüriPano improves over RPG360's 18.27, nearly halving the previous best error.This result demonstrates strong cross-domain performance on the challenging outdoor benchmark.
- Quantitative Comparison: 530.85 RMSE on ZüriPano improves over DepthAnyCamera's 716.38 while PaGeR also outperforms UniK3D and DAP indoors.Separate indoor and outdoor metric-scale heads specialize to distinct spatial layouts while sharing transformer features.
- Quantitative Comparison: 5.49° Mean Angular Error and 174.9 MSE establish state-of-the-art surface-normal results on Structured3D, improving over the previous 246.6 MSE.The reported results support the use of coupled task heads for fine-grained surface structure.
- Qualitative Comparison: Qualitative results show sharper boundaries, continuous global layouts, fewer warped geometries, and better preservation of complex indoor and outdoor structures than competing methods.Point-cloud reconstructions and surface-normal visualizations provide additional qualitative evidence.
- Ablation Studies: Ablations show that camera conditioning, log-space targets, sky segmentation, cross-face valid padding, joint perspective training, perceptual loss, encoder configuration, and metric-scale supervision affect accuracy.Removing camera conditioning or sky segmentation causes severe degradation, while poor downsampling choices and simplified objectives also reduce precision.
4 Related Work
Related work spans perspective geometry foundation models, unified visual geometry transformers, and panorama-specific estimation methods. The literature addresses perspective priors and panoramic distortions through increasingly general architectures and multi-projection representations, but seamless panorama reconstruction remains difficult.
- Perspective Geometry Estimation: Perspective monocular geometry has progressed from limited convolutional models to general-purpose architectures using large-scale data, transformers, diffusion priors, and unified depth-normal prediction.Absolute metric scale remains challenging, and naive metric-depth or localized-scale regression can disrupt perspective priors.
- Visual Geometry Foundation Models: Visual Geometry Foundation Models recast 3D reconstruction as dense correspondence regression, avoiding iterative Structure-from-Motion while supporting monocular, multi-view, and varying-camera inputs.Their input flexibility allows changing camera intrinsics or extrinsics within a unified feed-forward framework.
- Panoramic Geometry Estimation: Panoramic geometry methods address equirectangular distortion and data scarcity through specialized architectures, multi-projection formats, and training-free cubemap adaptations.Cubemap methods mitigate distortion but can struggle to merge predicted faces into a seamless equirectangular projection.
5 Conclusion
PaGeR lifts perspective depth-model representations into panoramic geometry through synchronized cubemaps, separate decoding heads, and joint panoramic/perspective training. The framework achieves zero-shot panoramic reconstruction across diverse indoor and outdoor environments and is presented as compatible with alternative geometric transformers.
- Conclusion: PaGeR combines synchronized cubemap representations, separate decoding heads, and joint panoramic/perspective training to transfer perspective priors to multiple 360° geometric tasks.The framework performs panoramic adaptation without extensive retraining.
- Conclusion: PaGeR establishes state-of-the-art zero-shot panoramic reconstruction across diverse indoor and outdoor environments.Its architectural principles are described as compatible with alternative geometric transformers beyond the DA3 instantiation.
6 Limitations
PaGeR has limitations on challenging materials and rare complex cubemap scenes. Predictions may be unreliable for reflective or transparent surfaces, and subtle boundary misalignments can remain.
- Specular, reflective, or transparent surfaces can produce unreliable predictions, while material variation creates depth ambiguities across datasets.
- Rare, structurally complex scenes can still exhibit subtle geometric or photometric misalignments at cubemap face boundaries.These artifacts are minimal and do not disrupt the global geometric layout.
A PanoInfinigen
PanoInfinigen addresses limited panoramic training data with complete, high-quality synthetic geometry across indoor, natural, and urban scenes. Its samples provide native 4K panoramas with pixel-perfect metric depth and surface-normal ground truth.
- Existing panoramic benchmarks are limited by static indoor coverage and acquisition artifacts, including missing depth values in reflective regions.
- PanoInfinigen provides complete, high-quality ground truth for depth and surface normals across diverse indoor and outdoor scenes.The dataset is extensible through an accompanying open-source generation tool.
- 20 urban environments yield roughly 7,000 outdoor panoramas with dense depth and surface-normal maps.The scenes vary in configuration and visual style, with approximately 400 panoramas rendered per city.
- Samples span indoor environments, natural landscapes, and urban surroundings, each rendered at native 4K with pixel-perfect metric depth and surface normals.
B ZüriPano
ZüriPano is designed as a rigorous outdoor panoramic benchmark for long-range accuracy and structural consistency. It uses high-resolution LiDAR panoramas from diverse urban locations, with unreliable regions masked during post-processing.
- Outdoor panorama collection is complicated by dynamic occlusions and limited high-resolution, long-range sensing, hindering reliable LiDAR-based evaluation.
- A Leica RTC360 scanner captures 8K panoramas across 100 scans and 11 urban locations in Zürich.The scanner has a 130-meter operating range and uses HDR imaging with automated double scans.
- Post-processing masks infinite-depth sky regions and highly specular areas, producing dense depth maps with validity masks.
- Figure 5 presents RGB, depth, and surface-normal panoramas from PanoInfinigen.
- Figure 6 presents samples from the ZüriPano dataset.
C Seam Consistency Metrics
The seam-consistency evaluation measures cubemap depth discontinuities at local, edge-level, and systemic granularities. Together, SDD, SP, and SS distinguish widespread mild jitter from isolated catastrophic failures.
- Seam Defect Density, Seam Prevalence, and Seam Severity monitor depth discontinuities across the N = 12 shared cubemap face boundaries.
- Adjacent pixel pairs across cubemap boundaries are partitioned into N disjoint edge sets for seam analysis.The log-depth jump uses Euclidean ERP linear depths at the paired pixels.
- Seam Defect Density measures the fraction of boundary pixels whose depth jump exceeds tolerance τ.
- An edge is prevalent when more than 10% of its pixels exceed τ, and Seam Prevalence is the fraction of such corrupted edges.
- Seam Severity measures the fraction of edges whose mean boundary jump exceeds magnitude threshold γ.
- The metric triplet separates local defect density, geometric coverage, and systemic misalignment across cubemap seams.This distinguishes widespread mild jitter from isolated catastrophic failures.
D Performance Trade-offs of Joint vs. Independent Model Training
PaGeR’s unified multi-task model efficiently predicts several geometric modalities, but joint training creates task-specific performance trade-offs, especially for metric depth.
- The ablation compares unified PaGeR against independently trained SI depth, surface normal, and metric depth models.The independent metric model uses dense supervision without depth alignment and directly predicts metric depth.
- Independent SI depth and surface normal models modestly outperform their counterparts in the unified model.
- The largest gap occurs between the independent metric depth model and the unified model’s metric scale head.
- Unfreezing the shared ViT for metric-scale gradients conflicts with other tasks, severely degrading SI depth and surface normal predictions.The independent metric model benefits from an unlocked backbone that extracts metric-specific cues early in the encoder.
- Freezing the ViT preserves SI depth and surface normal integrity but bottlenecks metric depth performance.The shared representation is therefore capacity-constrained when balancing scale-invariant and purely metric cues.
F Computational Efficiency and Resource Benchmark
PaGeR’s computational benchmark evaluates practical efficiency alongside qualitative reconstruction quality across panoramic depth, normals, and point-cloud outputs.
- The resource benchmark compares inference runtime, peak evaluation memory, native processing resolution, and total training data requirements.
- Qualitative depth comparisons cover Matterport3D360, Stanford2D3DS, and ZüriPano panoramas against selected baselines.
- Point-cloud comparisons include indoor and outdoor scenes, with zoomed-in novel-view renderings for indoor examples.
- Distance measurements in predicted point clouds are reported in meters.
- Panoramic surface-normal comparisons show PaGeR alongside the best available baseline, RGB input, and ground-truth depth on Structured3D.