Source-linked AI summary
PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery
Yijing Guo, Mengjun Chao, Luo Wang, Tianyang Zhao, Haizhao Dai, Yingliang Zhang, Jingyi Yu, Yujiao Shi
TL;DR
Perspective-oriented feed-forward models do not directly address panoramic distortions, and panoramic datasets often lack the scale, annotations, or viewpoint continuity needed for joint reconstruction. PanoVGGT jointly predicts pose, depth, and 3D structure from panoramas, with PanoCity providing outdoor data and geometric supervision. Experiments report strong performance across benchmarks, outperforming prior methods on most metrics while maintaining robust cross-domain generalization.
Problem
Perspective-focused feed-forward models face panoramic distortions, while existing panoramic datasets can lack scale, complete geometric annotations, and viewpoint continuity.
Source: §3 · pp. 2–3Method
PanoVGGT uses a permutation-equivariant Transformer with spherical-aware positional embeddings, three-axis SO(3) augmentation, and stochastic anchoring to jointly predict pose, depth, and 3D points.
Source: §1 · p. 2Results
Experiments show strong overall performance across public benchmarks and PanoCity, outperforming prior methods on most metrics while maintaining robust cross-domain generalization.
Source: §1 · p. 2Takeaways & Limitations
PanoCity adds a large-scale outdoor panoramic dataset with dense depth and pose supervision to support panoramic 3D reconstruction research.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Panoramic imagery offers a full 360{\deg} field of view and is increasingly common in consumer devices. However, it introduces non-pinhole distortions that challenge joint pose estimation and 3D reconstruction. Existing feed-forward models, built for perspective cameras, generalize poorly to this setting. We propose PanoVGGT, a permutation-equivariant Transformer framework that jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass. The model incorporates spherical-aware positional embeddings and a panorama-specific three-axis SO(3) rotation augmentation, enabling effective geometric reasoning in the spherical domain. To resolve inherent global-frame ambiguity, we further introduce a stochastic anchoring strategy during training. In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations. Extensive experiments on PanoCity and standard benchmarks demonstrate that PanoVGGT achieves competitive accuracy, strong robustness, and improved cross-domain generalization. Code and dataset will be released.
1. Introduction
Perspective-focused methods do not directly address panoramic distortion, while existing panoramic work has largely treated isolated tasks and small indoor datasets. PanoVGGT jointly reconstructs pose, depth, and 3D structure from panoramas, alongside PanoCity, a large-scale outdoor dataset.
1. Introduction
- Panoramic coverage captures the full surround from one vantage, simplifying global mapping and reducing the need for viewpoint changes.Source: §1 · p. 2
- End-to-end panoramic 3D reconstruction that unifies multi-view geometry in a single pass remains largely unexplored, despite prior panoramic methods and datasets.Prior work often addresses isolated tasks and is limited to small indoor datasets.Source: §1 · p. 2
- PanoVGGT jointly predicts camera poses, dense depth maps, and globally consistent 3D point clouds from unordered panoramas in one forward pass.It uses spherical-aware positional embeddings, three-axis SO(3) rotation augmentation, and stochastic anchoring to address spherical geometry and global-frame ambiguity.Source: §1 · p. 2
- PanoCity contains over 120,000 high-fidelity, high-overlap panoramas with complete pose and depth supervision, particularly expanding outdoor urban coverage.Source: §1 · p. 2
- Experiments show strong overall performance across public benchmarks and PanoCity, outperforming prior methods on most metrics while maintaining robust cross-domain generalization.Source: §1 · p. 2
2. Related Work
Feed-forward reconstruction models have replaced iterative geometric pipelines with joint prediction, but prominent approaches remain designed for perspective imagery. Panoramic research has developed distortion-aware representations, yet unified feed-forward reconstruction remains largely unexplored.
Feed-Forward 3D Reconstruction
- Classical structure-from-motion and multi-view stereo achieve accurate reconstruction but rely on iterative optimization and are sensitive to texture and scene dynamics.Source: §2 · p. 2
- Transformer-based feed-forward models jointly predict depth and pose from multi-view inputs, but their pinhole assumptions limit direct application to panoramas.Source: §2 · p. 2
Panoramic 3D understanding
- Panoramic methods use distortion-aware representations, while most address single tasks such as depth or pose rather than jointly inferring pose, depth, and coherent 3D structure.Source: §2 · p. 2
3. Dataset - PanoCity
Panoramic datasets often lack scale, complete annotations, or continuous viewpoint overlap, limiting their suitability for joint feed-forward reconstruction. PanoCity provides large-scale urban panoramas with synchronized depth and pose labels along continuous, varied trajectories.
3. Dataset - PanoCity
- Existing panoramic datasets have limitations in scale, geometric annotations, RGB–depth consistency, and viewpoint sampling.Source: §3 · p. 2
- Stable overlap across frames is needed for feed-forward joint learning of dense depth and 6DoF camera motion.Source: §3 · p. 3
- Prior datasets may lack overlap, contain RGB–depth mismatches, or omit pose or depth annotations, motivating a dataset with complete labels and continuous trajectories.Source: §3 · p. 3
- PanoCity uses smooth, kinematically constrained trajectories through three virtual cities and five weather and illumination conditions.Source: §3 · p. 3
- PanoCity contains over 120,000 frames with 4096 × 2048 RGB images, metrically consistent 16-bit depth, and accurate 6DoF camera-to-world poses rendered synchronously.The aligned annotations provide supervision for joint depth, pose, and 3D reconstruction.Source: §3 · p. 3
- PanoCity combines accurate geometric annotations, continuous multi-view trajectories, and diverse urban conditions to support panoramic pose, depth, and 3D reconstruction.Source: §3 · p. 3
4. Method – PanoVGGT
PanoVGGT adapts a permutation-equivariant Transformer to unordered panoramic views, jointly predicting poses, depth, and point clouds. Its design combines spherical-aware embeddings, geometric aggregation, stochastic anchoring, rotation augmentation, and multi-task supervision.
4.1. Framework Overview
- For unordered panoramas, PanoVGGT jointly predicts camera poses, dense depth, and globally consistent point clouds; explicit point regression adds redundant supervision that the authors report improves accuracy and robustness.Source: §4 · p. 3, §4.1 · pp. 3–4
4.2. Spherical-aware Position Embedding
- Sin/cos encodings of patch-center angles preserve wrap-around continuity at panorama seams, while fixed-grid embeddings paired with rotated content help disentangle projection distortion from scene semantics.Source: §4.2 · pp. 4–5
4.3. Geometry Aggregator
- Alternating frame-wise and global attention captures within-view structure and cross-view relationships without encoding view order; stochastic anchoring randomizes the global frame rather than fixing the first input.Source: §4.3 · p. 5
4.4. Panorama-Specific Data Augmentation
- Three-axis SO(3) rotations jointly transform RGB, depth, and camera poses on the sphere, yielding physically correct panorama views and unbounded augmentation.Source: §4.4 · p. 5
4.5. Loss Functions
- A single per-sample scale aligns local and global point-cloud losses, with s* fitted against local ground-truth points and shared across geometric heads.Source: §4.5 · p. 5
- Normal consistency penalizes angular differences between surface normals estimated from neighboring 3D points with SmoothL1, excluding invalid pixels near depth edges.Source: §4.5 · pp. 5–6
- Relative rotations use angular-distance supervision and translations use scale-consistent L1 supervision; the final objective combines these with local/global geometry and normal losses.Source: §4.5 · p. 6
5. Experiments
Experiments evaluate pose, depth, and point-cloud reconstruction across indoor and outdoor datasets under shared protocols. PanoVGGT performs strongly across tasks, while ablations examine its components and note an early translation-error trade-off from spherical augmentation.
5. Experiments
- Evaluations cover three core tasks and five datasets, with revised multi-view splits and a three-view pose protocol shaped by dataset overlap constraints.All models are retrained or assessed under identical settings; Structured3D has minimal usable multi-view data, and weak or absent overlap also challenges indoor pose estimation.Source: §5 · p. 6, §5.1 · p. 6
5.1. Camera Pose Estimation
- PanoVGGT achieves the highest AUC and lowest rotation and translation errors in most pose-estimation settings.Pose metrics are reported after filtering zero-translation pairs, and use angular errors to avoid scale ambiguity.Source: §5.1 · p. 6, Table 2 · p. 7
5.2. Monocular Depth Estimation
- PanoVGGT achieves state-of-the-art depth performance on Matterport3D and Stanford2D3D in a unified multi-task setting.Multi-view inputs improve on its competitive monocular accuracy; some depth-only models have slight advantages on Structured3D and PanoCity.Source: §5.2 · p. 7, Table 3 · p. 7
- Zero-shot Pano3D depth evaluation finds PanoVGGT consistently outperforms prior methods across most metrics under scale-only and scale+shift alignment.Pinhole-based methods are evaluated through dodecahedral projection, while the benchmark tests cross-dataset generalization.Source: §5.2 · p. 7, Table 4 · p. 7
5.3. Point Cloud Reconstruction
- PanoVGGT achieves consistently lower point-cloud errors across accuracy, completion, and overall-quality metrics on PanoCity.The reported comparison uses 10 input panoramas; additional dataset evaluations are in supplementary material.Source: §5.3 · p. 8, Table 5 · p. 8
5.4. Ablation Studies
- At the 100-epoch ablation checkpoint, tri-axis spherical augmentation slightly increases translation error as rotational alignment converges first.The full model is trained for 200 epochs; the authors attribute the temporary delay to greater orientation diversity from full SO(3) rotations.Source: §5.4 · p. 8, Table 6 · p. 8
6. Conclusion
PanoVGGT jointly reconstructs pose, depth, and point clouds from unordered panoramas, and PanoCity supplies outdoor depth and pose supervision. The conclusion reports strong reconstruction results while noting sensitivity to extreme domain shifts and an equirectangular-projection-specific design.
6. Conclusion
- PanoVGGT jointly predicts camera poses, dense depth, and globally consistent point clouds from unordered panoramas in one forward pass.Its spherical-aware positional encoding and three-axis SO(3) augmentation support rotation-aware representations for panoramic projection distortions.Source: §6 · p. 8
- PanoCity adds large-scale outdoor panoramic data with dense depth and pose supervision.The conclusion describes experiments across diverse indoor and outdoor benchmarks as demonstrating strong reconstruction accuracy and geometric reasoning.Source: §6 · p. 8
- Pose estimation remains sensitive to extreme domain shifts, and the current design is tailored to equirectangular projections.Source: §6 · p. 8
A. Training and Evaluation Details
Training combines four panoramic datasets with dynamically sampled multi-view sequences and spherical augmentations. Evaluation uses consistent protocols, while two indoor datasets retain limited-overlap scenes to broaden coverage and test robustness.
A. Training and Evaluation Details
- Training uses DINOv2 ViT-B/14 and a 24-block alternating-attention geometry aggregator on Matterport3D, Stanford2D3D, Structured3D, and PanoCity.Training sequences contain 2–24 panoramas; spherical rotations are resampled consistently across RGB, depth, and pose.Source: A. Training and Evaluation Details · p. 11
- The retrained panoramic π3* adapts its geometric head to equirectangular imagery but disables spherical-rotation augmentation.Its datasets, augmentations otherwise, and optimization settings match PanoVGGT; other baselines use official implementations or released weights.Source: A. Training and Evaluation Details · p. 11
- Evaluation uses 518×1036 inputs, single-view zero-shot depth on Pano3D, and 10-frame PanoCity or 3-frame indoor point-cloud samples.Predicted camera poses transform point maps into world coordinates for point-cloud evaluation.Source: A. Training and Evaluation Details · p. 11
- Stanford2D3D and Matterport3D include limited or negligible panorama overlap even after re-splitting and postprocessing.They are retained to broaden coverage and test robustness across different levels of viewpoint overlap.Source: A. Training and Evaluation Details · p. 11
B. Additional Depth Results
Additional depth results compare PanoVGGT with a retrained panoramic π3 variant on standard datasets and zero-shot Pano3D. PanoVGGT retains the reported depth advantages across these comparisons, including under distribution shift.
B. Additional Depth Results
- Across Matterport3D, Stanford2D3D, Structured3D, and PanoCity, PanoVGGT has the best Abs Rel and δ<1.25 accuracy, while retrained π3* remains inferior.All methods use identical preprocessing, data splits, and evaluation protocols.Source: B. Additional Depth Results · p. 11
- On zero-shot Pano3D, PanoVGGT has lower errors than π3* across metrics under scale-only alignment, and retains an advantage with scale+shift alignment.π3* improves slightly with scale+shift alignment but still lags in accuracy and structural consistency.Source: B. Additional Depth Results · p. 11
C. Additional Point-Cloud Results
Across Matterport3D and Stanford2D3D, PanoVGGT achieves the lowest average point-cloud errors among the compared methods. Qualitative comparisons show more consistent geometry and alignment than perspective-based alternatives.
C. Additional Point-Cloud Results
- The dodecahedral π3† protocol expands a three-panorama input into 36 pinhole images, improving over π3 but adding significant computational overhead.Source: C. Additional Point-Cloud Results · pp. 11–12
- Across Matterport3D and Stanford2D3D, PanoVGGT achieves the lowest average errors on point-cloud Accuracy, Completion, and Overall metrics.Lower values indicate better performance; Tables 9 and 10 report the respective dataset results.Source: C. Additional Point-Cloud Results · p. 12, Table 9 · p. 12, Table 10 · p. 12
- π3 produces warped geometry, π3* remains incomplete or inconsistent, and π3† introduces patch discontinuities and inaccurate relative poses; PanoVGGT yields seamless, globally aligned scenes.The comparisons include two unordered panoramas on Stanford2D3D and ten on PanoCity.Source: C. Additional Point-Cloud Results · p. 12
- On Stanford2D3D, PanoVGGT shows higher geometric accuracy and cross-view consistency than the three π3 baselines.Source: Figure 4 · p. 13
D. Additional Visualization of PanoCity
PanoCity visualizations present diverse urban scenes alongside panoramic RGB, point-cloud, depth, and trajectory data. Its long outdoor routes also provide a setting for comparing multi-view reconstruction coherence and alignment.
D. Additional Visualization of PanoCity
- PanoCity’s three illustrated regions span coverage areas of 15 km², 113 km², and 6 km² and include downtown, residential, and arterial-road settings.Source: D. Additional Visualization of PanoCity · p. 13
- For ten unordered panoramas along a PanoCity long trajectory, PanoVGGT reconstructs large outdoor scenes with coherent structure and accurate alignment, while the baselines struggle with geometry.Source: Figure 5 · p. 14
- The five visualization rows show urban scenes, route-sampled panoramic RGB images, corresponding ground-truth point clouds, color-coded depth maps, and bird’s-eye camera trajectories.The routes illustrate long paths with large baselines and substantial viewpoint changes.Source: D. Additional Visualization of PanoCity · p. 13, Figure 6 · p. 15