Source-linked AI summary
GeoPano: Towards Geometrically Accurate 3D Reconstruction from a Single Panorama
Jing OU, Zidong Cao, Liaoyuan Fan, Zhuoxiao Li, Shuai Zhang, Yinrui Ren, Dongli Wu, Haoang Li, Hui Xiong, Wufan Zhao
TL;DR
ERP distortions and inconsistencies across decomposed perspective views complicate panoramic 3D reconstruction. GeoPano uses geometry-aware attention and hybrid local-global supervision; experiments report better accuracy, consistency, and visual fidelity than prior single-view panoramic methods.
Problem
ERP imagery is affected by spherical distortion, while processing perspective patches independently can introduce cross-view geometric inconsistencies.
Source: §1 · p. 2Method
GeoPano combines overlap-guided Pano-Attention with local perspective-space and global spherical-space point-cloud supervision.
Source: §1 · p. 3, §3.4 · p. 6Results
GeoPano outperforms prior single-view panoramic methods in reconstruction accuracy, geometric consistency, and visual fidelity.
Source: §5 · p. 10Takeaways & Limitations
GeoPano maintains practical efficiency while targeting fine-grained local geometry and holistic structural coherence.
Source: §3.4 · p. 6, §5 · p. 10
Abstract
from arXiv · showhide
Panoramic 3D reconstruction enables the holistic recovery of complete 360 $^\circ$ scene geometry, offering significant benefits for robotics and AR/VR applications. However, directly estimating geometry from ERP images is inherently challenged by spherical distortions. To reduce spherical projection distortions, a common strategy is to decompose panoramas into distortion-free perspective views, but independently processing these views often introduces cross-view geometric inconsistencies. To address these challenges, we introduce GeoPano, a feed-forward framework for single-view panoramic 3D reconstruction that recovers a geometrically accurate and consistent point cloud from a single panorama. Our approach is driven by two key innovations: First, we propose a panoramic-aware attention aggregator, featuring a streamlined attention architecture equipped with a novel Pano-Attention mechanism, which explicitly exploits the intrinsic spherical geometry of the panorama to guide cross-view interactions, thereby suppressing geometric inconsistencies. Furthermore, we formulate a hybrid supervision strategy to jointly enforce fine-grained local details in perspective space and global structural coherence in spherical space. Experiments demonstrate that our method significantly outperforms previous SOTA approaches in reconstruction accuracy, geometric consistency, and visual fidelity.
1 Introduction
Panoramic reconstruction is challenged by ERP distortions and by inconsistencies between decomposed perspective views. GeoPano adapts a feed-forward 3D model with geometry-aware attention and hybrid supervision to target both local detail and global consistency.
1 Introduction
- ERP’s polar stretching distorts visual-feature geometry, while independently processed perspective patches disrupt global context and create cross-view inconsistencies.Existing supervised fusion is limited by dense 360° annotation costs and computational overhead; training-free methods rely on costly alignment or panoramic references.Source: §1 · p. 2
- Feed-forward 3D foundation models motivate reformulating panorama reconstruction as a sparse multi-view problem using decomposed views.Their pretrained geometric priors support holistic recovery and require only minimal fine-tuning data.Source: §1 · pp. 2–3
- π3’s early global attention misses spatial-overlap correlations, while later layers still assign significant interactions to non-overlapping views.The latter interactions are argued to harm cross-view consistency.Source: §1 · p. 3
- GeoPano combines intra-frame attention followed by overlap-guided Pano-Attention with hybrid local perspective-space and global spherical-space supervision.The framework is reported to improve cross-view consistency and outperform prior single-view panoramic methods in accuracy and visual fidelity.Source: §1 · p. 3
2 Related Work
Prior panoramic depth methods either estimate directly in ERP space or fuse decomposed perspective views, each retaining limitations. Feed-forward reconstruction models provide geometric priors, but existing approaches lack explicit constraints tailored to spherical topology.
2.1 Panoramic Monocular Depth Estimation
- ERP depth methods use distortion-aware architectures, synthesized or transferred data, and larger datasets, yet distortions and data scarcity still hinder accuracy in highly distorted regions.Multi-view fusion reduces spherical distortion but can disrupt global structure and depend on time-consuming alignment or cross-domain fusion.Source: §2.1 · p. 3
2.2 Feed-forward 3D Reconstruction Models
- DUST3R regresses point maps and camera poses from uncalibrated images; VGGT and π3 extend feed-forward reconstruction, but these models lack explicit spherical-topology constraints.π3 uses permutation-equivariant design to address reference-frame dependency.Source: §2.2 · pp. 3–4
- Recent spherical-domain feed-forward methods target panoramic reconstruction through perspective pre-training, joint prediction from unordered panoramas, or perspective foundation models.The cited examples are PanoSplatt3R, PanoVGGT, and VGGT-360.Source: §2.2 · p. 4
3 Methodology
The methodology analyzes cross-view attention in π3, then builds a panoramic reconstruction pipeline with geometry-guided attention and a hybrid loss. Its design addresses diffuse early interactions, residual non-overlap interactions, and the need to combine local detail with global consistency.
3.1 Analysis of Cross-View Interactions
- Early global attention in π3 spreads uniformly across horizontal views without favoring spatial overlap, while later layers still retain significant interactions between non-overlapping views.These findings motivate improving cross-view feature interactions to support reconstruction consistency.Source: §3.1 · pp. 4–5
3.2 Framework Overview
- The framework decomposes a panorama into 10 overlapping perspective views, encodes them with DINOv2, predicts per-view point clouds, and fuses them into a globally aligned point cloud.A hybrid loss supervises both local perspective clouds and the global panoramic cloud.Source: §3.2 · p. 5, Figure 4 · p. 5
3.3 Panoramic-aware Attention Aggregator
- The aggregator uses nine intra-frame attention layers followed by nine alternating-attention layers, with Pano-Attention added residually in layers 10, 12, 14, and 16 to guide interactions toward overlapping regions.Its candidate mask uses cosine similarity between spherical token rays and a threshold τ to identify geometrically correlated tokens.Source: §3.3 · p. 6
3.4 Hybrid Loss
- The hybrid loss weights local and panoramic point-cloud losses: local predictions are scale-aligned with ground truth using inverse-depth weights, while per-view clouds are reprojected into spherical space for panoramic comparison.The local term supervises fine-grained view geometry; the panoramic term couples that precision with holistic consistency.Source: §3.4 · pp. 6–7
4 Experiments
GeoPano is evaluated on indoor and outdoor panoramic datasets against recent baselines, with reported gains across 3D and 2D metrics. Ablations examine its attention design and hybrid loss.
4.1 Experiment Setup
- Training uses 23,000 frames from 75 indoor and outdoor scenes; evaluation covers two indoor datasets and one outdoor dataset with point-cloud and depth metrics.Training panoramas are 2880 × 1440; evaluation metrics include Chamfer Distance, F-Score, IoU, AbsRel, RMSE, and threshold accuracy.Source: §4.1 · p. 7
4.2 Qualitative and Quantitative Evaluation
- GeoPano establishes a new state-of-the-art across all evaluated 3D and 2D metrics.The evaluation also reports reduced spherical distortions and geometric consistency across indoor and outdoor scenes.Source: Table 1 · p. 7, §4.2 · p. 7
- Qualitative comparisons report precise outdoor long-range geometry and indoor global coherence with fine local geometry against DA2, DAP, and RPG360.Outdoor comparisons also report mitigated structural distortions in long-range regions.Source: §4.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
4.3 Ablation Studies
- Dropping early global attention from layers 0–8 gives the best trade-off between computational efficiency and geometric accuracy.The ablation evaluates dropping initial layers 0–4, 0–6, 0–8, or 0–10; diffuse early interactions are reported to adversely affect global consistency.Source: §4.3 · pp. 8–9, Table 3 · p. 9
- On Matterport3D, the introduced modules improve reconstruction accuracy; the full model reduces inference time by 10.3% while increasing memory consumption by 5.6% versus baseline.The ablation compares a panoramic-finetuned π3 baseline with added components.Source: §4.3 · p. 8, Table 2 · p. 9
- Applying Pano-Attention at every other layer performs best, while standard global attention at those layers does not yield comparable gains.The reported comparison indicates that moderate geometric constraints balance reliable cross-view interaction and global information aggregation.Source: §4.3 · p. 9, Table 4 · p. 9
- Combining local and global losses outperforms single-loss baselines, with peak performance at λ = 1 and β = 0.75.The weighting balances local and global representation learning while preserving local structures and global coherence.Source: §4.3 · p. 9, Table 5 · p. 9
5 Conclusion
GeoPano combines spherical-geometry-aware attention with local and global point-cloud supervision for single-view panoramic reconstruction. The paper reports improved accuracy, consistency, and visual fidelity over prior single-view panoramic methods while maintaining practical efficiency.
5 Conclusion
- GeoPano is a feed-forward framework for single-view panoramic 3D reconstruction.Its panoramic-aware attention aggregator leverages spherical geometry to improve cross-view consistency.Source: §5 · p. 10
- A hybrid supervision strategy combines local and global point-cloud losses to support fine-grained detail and holistic structural coherence.Source: §5 · p. 10
- Experiments report better accuracy, consistency, and visual fidelity than prior single-view panoramic methods, while maintaining practical efficiency.Source: §5 · p. 10