Source-linked AI summary
PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation
Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, Yao Zhao
TL;DR
Panoramic depth estimation is hindered by ERP distortions and CNNs’ fixed receptive fields, which limit geometric-structure perception. PanoFormer uses spherical tangent patches, learnable token flow, and panorama-specific metrics, and experiments show strong results for panoramic depth estimation and semantic segmentation.
Problem
ERP distortions deform panoramic objects, while CNNs’ fixed receptive fields limit geometric-structure perception for panoramic depth estimation.
Method
PanoFormer combines spherical tangent patches, learnable token flow, a redesigned panoramic self-attention block, and two panorama-specific metrics.
Results
PanoFormer significantly outperforms current state-of-the-art methods on panoramic depth estimation and achieves the best performance when applied to semantic segmentation.
Takeaways & Limitations
The proposed PST block extends beyond depth estimation to other pixel-level panoramic tasks, including semantic segmentation.
Abstract
from arXiv · showhide
Existing panoramic depth estimation methods based on convolutional neural networks (CNNs) focus on removing panoramic distortions, failing to perceive panoramic structures efficiently due to the fixed receptive field in CNNs. This paper proposes the panorama transformer (named PanoFormer) to estimate the depth in panorama images, with tangent patches from spherical domain, learnable token flows, and panorama specific metrics. In particular, we divide patches on the spherical tangent domain into tokens to reduce the negative effect of panoramic distortions. Since the geometric structures are essential for depth estimation, a self-attention module is redesigned with an additional learnable token flow. In addition, considering the characteristic of the spherical domain, we present two panorama-specific metrics to comprehensively evaluate the panoramic depth estimation models' performance. Extensive experiments demonstrate that our approach significantly outperforms the state-of-the-art (SOTA) methods. Furthermore, the proposed method can be effectively extended to solve semantic panorama segmentation, a similar pixel2pixel task. Code will be available.
1 Introduction
PanoFormer addresses the difficulty of panoramic depth estimation by combining distortion reduction with geometric-structure perception. It introduces panorama-specific evaluation metrics and reports strong performance on depth estimation and semantic segmentation.
- Motivation: Panoramic depth estimation is challenging because ERP distortions deform object shapes and CNNs’ fixed receptive fields hinder geometric-structure perception.These distortions increase from the image center toward both sides along latitude.
- Approach: PanoFormer is introduced as a panorama Transformer that simultaneously reduces distortions and perceives panoramic geometric structures.The method is designed for panoramic dense prediction tasks such as depth estimation and semantic segmentation.
- Approach: The PST block uses spherical tangent-domain patches, relative position embedding, and learnable token flow to support panoramic perception.Its spherical token locating model guides non-distortion token sampling directly on the ERP domain.
- Evaluation: P-RMSE and LRCE evaluate polar-region accuracy and left-right boundary depth consistency, respectively.These metrics address panoramic properties not considered by metrics designed for perspective imagery.
- Results: PanoFormer significantly outperforms current state-of-the-art methods in panoramic depth estimation and achieves strong results on semantic segmentation.The paper presents semantic segmentation as an extension to another pixel-level panoramic task.
2 Related Work
Prior panoramic depth methods reduce ERP distortions through projection fusion or distortion-aware architectures, while Transformer-based approaches offer global dependency modeling. These approaches motivate methods designed directly for panoramic structure and distortion handling.
- Panoramic depth estimation: Projection-fusion methods combine equirectangular and cube projections to attenuate panoramic distortions, but repeated projection changes increase model complexity.Dual-cube projection also addresses discontinuities associated with single-cube projection.
- Panoramic depth estimation: Models transferred from conventional 2D imagery can produce low-quality panoramic depth, motivating learning directly on the 360° domain.Related work also explores distortion-aware filters, geometric priors, strip pooling, and deformable convolutions.
- Vision Transformers: Transformers model long-range dependencies through global self-attention, while hybrid designs add convolutional or locally enhanced components for local feature extraction.Examples include convolutional attention and locally enhanced feed-forward networks.
3 PanoFomer
PanoFormer redesigns Transformer components for pixel-level panoramic prediction by combining dense tangent patches, distortion-aware positioning, and panorama self-attention with learnable token flow. Its hierarchical encoder-decoder uses these components to preserve detail, model panoramic geometry, and recover depth.
- 3.1 Architecture Overview: The hierarchical architecture contains input and output stems, four encoder stages, a bottleneck, and four decoder stages built around PST blocks.Encoder convolutions downsample and increase feature dimension, decoder transposed convolutions upsample and decrease it, and circular padding is used horizontally.
- 3.2 Transformer-customized Spherical Token: PanoFormer uses pixel-level patches so each feature sampling position forms a patch, preserving detailed features for dense prediction.Each patch contains one central feature and eight surrounding features, balancing detail and computational cost.
- 3.2 Transformer-customized Spherical Token: Handcrafted tokens separate each patch into a central token and eight related tokens, with the central token locating the most relevant neighbors.The method uses distortion-based relative position embedding to encode positions for these tokens.
- 3.3 Relative Position Embedding: STLM projects central tokens from ERP to the spherical domain, locates eight tangent-plane neighbors, and projects their positions back to ERP.The spherical tangent projection is used to reduce distortion, while the tangent-to-spherical and spherical-to-ERP mappings locate related tokens.
- 3.4 Panorama Self-Attention with Token Flow: PanoFormer replaces standard attention with panorama self-attention and uses three branches to compute attention scores, learnable token flows, and resampled features.The learned flow is added to STLM-initialized positions before related tokens are sampled.
- 3.4 Panorama Self-Attention with Token Flow: The initialized positions remove panoramic distortion, while learned token flows adjust token distributions to represent panoramic geometry.Token-flow visualizations from the first PST block are used to suggest the learned panoramic structures.
4 Panorama-specific Metrics
The paper introduces LRCE and P-RMSE to evaluate panorama-specific properties: seamless boundary consistency and performance in highly distorted polar regions.
- Left-Right Consistency Error: LRCE measures depth consistency across the left-right boundaries of panoramic depth maps.It addresses the spherical domain’s continuous and seamless property.
- Pole Root Mean Square Error: P-RMSE is designed because conventional RMSE does not visually reflect adaptation to longitude-varying ERP distortions.It focuses on massively distorted regions to assess panoramic perception.
- Pole Root Mean Square Error: P-RMSE measures RMSE in the polar regions, where ERP distortions are particularly severe.The method converts ERP images to cube projections and selects the top and bottom faces corresponding to polar regions.
- Left-Right Consistency Error: The LRCE design compares horizontal gradients from the ground-truth and predicted depth maps.Ground-truth gradients help distinguish real object-edge discontinuities from model-induced boundary discontinuities.
5 Experiments
Experiments evaluate PanoFormer on four panoramic datasets, compare it with SOTA methods, analyze its components, and test extension to semantic segmentation.
- Datasets and Implementations: Experiments use Stanford2D3D, Matterport3D, PanoSUNCG, and 3D60, including real-world and virtual datasets.The real-world datasets were rendered from common viewpoints, while prior data omitted areas near the poles.
- Comparison Results: 42% improvement on RMSE is reported on PanoSUNCG, compared with 16% on 3D60.The paper also reports approximately 40% average gain on P-RMSE across the other two virtual datasets.
- Comparison Results: Qualitative results show that SliceNet degrades in distorted regions, while UniFuse can lose detailed information despite handling deformation.PanoFormer’s results are described as competitive near boundaries and poles.
- Ablation Study: Ablation studies on Stanford2D3D compare CNN, traditional-attention, no-token-flow, and full-network variants.The variants are labeled a, b, c, and d in Table 2.
- Extensibility: Without changing the network structure, PanoFormer outperforms current SOTA approaches on panoramic semantic segmentation.The segmentation results are averaged over the official three folds of Stanford2D3D.
6 Conclusion
The paper proposes PanoFormer for indoor panoramic depth estimation, combining distortion reduction, geometric-structure modeling, and panorama-specific evaluation. Experiments report strong performance on depth estimation and semantic segmentation.
- Conclusion: PanoFormer is presented as the first panorama Transformer for indoor panoramic depth estimation.It uses learnable token flow to model geometric structures while reducing distortion effects.
- Conclusion: The PST block is designed for panoramic perception and can be extended to other learning frameworks.The paper also proposes two metrics based on equirectangular-image priors.
- Conclusion: Experiments report significant outperformance over current SOTA methods on depth estimation and semantic segmentation.Semantic segmentation is identified as another pixel2pixel panoramic task.