PaperlayerView exact revision on arXiv

Source-linked AI summary

AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation

Soumyaratna Debnath, Weiming Zhang, Shriram Damodaran, Dingwen Xiao, Addison Lin Wang

arXiv:2608.29081v1cs.CV

TL;DR

Existing spherical Transformers struggle with contextual and geometric ambiguity under unseen spherical transformations. AdapToPASS addresses this with adaptive attention, bifocal spherical representation, and boundary supervision, while its compact Swift variant preserves the approach under efficiency constraints. The method achieves the best reported performance across the evaluated datasets and transformation settings, although severe perturbations still cause noticeable degradation.

  • Problem

    Existing spherical architectures do not adapt to contextual and geometric ambiguity caused by noncanonical viewpoints and spherical transformations, limiting robustness to unseen transformations.

  • Method

    AdapToPASS combines Adaptive Spherical Attention, Bifocal Spherical Representation, and SDF-based boundary supervision to model contextual, geometric, and boundary ambiguity.

  • Results

    AdapToPASS achieves the best overall performance across both datasets, with 70.80% mIoU on Stanford2D3D and 69.58% mIoU on WildPASS, while leading across transformation types and severity levels.

  • Takeaways & Limitations

    AdapToPASS retains a clear margin over prior approaches under unseen spherical transformations, and AdapToPASS-Swift extends this robustness to a variant with fewer than 2M parameters.

  • Takeaways & Limitations

    Performance still degrades noticeably under extremely severe perturbations, especially orientation and viewpoint shifts that disrupt spherical layout and semantic consistency.

Abstract

from arXiv · show

Spherical Transformers have emerged as a promising framework for panoramic semantic segmentation (PASS) by operating directly on spherical geometry and alleviating projection-induced distortions. However, existing architectures often assume canonical spherical structure and stable viewpoints, which are frequently violated in real-world imagery due to unconstrained camera motion, introducing contextual and geometric ambiguity. Consequently, they lack adaptive mechanisms to handle such ambiguity, limiting robustness to unseen spherical transformations. In contrast, biological perception is inherently ambiguity-aware, adapting to fluctuations in cue reliability caused by geometric and contextual variations to maintain stable interpretation under complex transformations. Motivated by this, we first systematically analyze existing PASS architectures under various unseen spherical transformations. We then introduce AdapToPASS, a novel bio-inspired Spherical Transformer that adaptively models contextual and geometric ambiguities for robust PASS. At its core, Adaptive Spherical Attention (AdaSpA) blocks dynamically modulate attention according to local contextual ambiguity, mimicking adaptive, context-driven biological perception. To address geometric ambiguity, AdapToPASS employs Bifocal Spherical Representation to balance field of view and spatial resolution, together with boundary supervision inspired by the boundary-sensitive nature of biological vision. Across indoor and outdoor semantic segmentation, AdapToPASS consistently outperforms prior state-of-the-art methods. Under unseen spherical transformations, it surpasses the next-best method by +13.38% relative mIoU on Stanford2D3D and +18.77% on WildPASS. We further introduce AdapToPASS-Swift, a lightweight variant with fewer than 2M parameters, which surpasses compact baselines while retaining robustness to spherical transformations.

1 Introduction

Existing spherical segmentation architectures degrade under unseen spherical transformations because they do not adapt to contextual and geometric ambiguity. AdapToPASS addresses these limitations with ambiguity-aware attention, bifocal spherical representation, and boundary supervision, achieving stronger robustness across indoor and outdoor benchmarks.

  • Motivation: Existing spherical Transformers degrade under unseen spherical transformations because they aggregate context uniformly and treat spatial locations equally despite contextual and geometric ambiguity.These limitations arise when local class boundaries are ambiguous or spherical transformations introduce varying distortion and structural irregularity.
  • Approach: AdapToPASS adaptively modulates attention according to local contextual ambiguity through Adaptive Spherical Attention blocks.The design is motivated by biological vision’s adaptive adjustment of contextual integration.
  • Approach: AdapToPASS balances field of view and spatial resolution through Bifocal Spherical Representation, using dual pathways for high-resolution detail and wide-field context.The framework also uses SDF-based boundary supervision to capture boundary ambiguity and improve localization.
  • Results: +13.38% relative mIoU on Stanford2D3D and +18.77% on WildPASS over the previous best-performing method under unseen spherical transformations.The method is evaluated across indoor and outdoor panoramic semantic segmentation benchmarks and varying transformation difficulty levels.
  • Results: AdapToPASS-Swift preserves ambiguity-aware spherical perception in a compact variant with fewer than 2M parameters.The compact model is reported as retaining robustness under strict efficiency constraints.

2 Related Works

Related work addresses panoramic distortion, spherical representations, transformation robustness, and ambiguity-aware context modeling, but the paper positions AdapToPASS as a unified architecture covering contextual, geometric, and boundary ambiguity.

  • Panoramic Semantic Segmentation: ERP remains widely used for 360° imagery but introduces latitude-dependent distortions that compromise geometric consistency and semantic reasoning.Prior work therefore explores distortion-aware and alternative spherical representations.
  • Panoramic Semantic Segmentation: Existing spherical approaches include hierarchical attention, multimodal cues, icosahedron-based bi-projection fusion, and localized spherical self-attention.These methods represent different strategies for panoramic dense perception and spherical processing.
  • Transformation Robustness: Most panoramic segmentation methods implicitly assume a canonical camera orientation, limiting robustness when camera motion and viewpoint changes induce spherical distortions.Transformation-aware modeling has begun addressing this fragility, but the problem remains central to real-world panoramic perception.
  • Geometric and Context Modeling: Panoramic imagery introduces contextual, geometric, and boundary ambiguity through unreliable long-range cues, spherical distortions, viewpoint changes, and unstable semantic transitions.Prior analysis specifically motivates adaptive context modeling instead of uniform feature aggregation.
  • Positioning: AdapToPASS is presented as the first unified architecture to jointly model contextual, geometric, and boundary ambiguity for panoramic semantic segmentation.Its stated scope includes adaptive context aggregation, spherical distortion handling, and sharper boundary prediction directly on the sphere.

3 Methodology

AdapToPASS models semantic, contextual, and geometric ambiguity on an icosphere through adaptive attention, bifocal streams, and boundary-aware supervision.

  • Overview: AdapToPASS projects omnidirectional RGB images onto an icosphere and predicts per-node semantic logits plus per-class signed distance fields.The spherical signal contains N vertices, and predictions cover K semantic classes.
  • Overview: Its energy formulation combines semantic prediction with contextual-ambiguity modeling through AdaSpA and geometric-ambiguity modeling through Bifocal Spherical Representation.The formulation also uses an ambiguity signal and graph-based boundary guidance.
  • Adaptive Spherical Attention: AdaSpA performs local spherical self-attention with normalized queries and keys, learnable head scales, and ambiguity-aware geodesic biases.The geodesic bias incorporates distances between neighboring icosphere nodes and preserves spherical proximity in attention.
  • Adaptive Spherical Attention: AdaSpA rescales geodesic distance using a learned contextual ambiguity score, broadening aggregation for ambiguous tokens while retaining sharper locality for confident tokens.Each block otherwise follows a pre-normalization attention-FFN structure.
  • Bifocal Spherical Representation: The Bifocal Spherical Representation uses an Acuity Stream for full-resolution local structure and a Lateral Stream for broader context, combining their features through gated fusion.The Lateral Stream progressively downsamples and later lifts features back to the full icosphere before fusion.
  • Boundary Supervision: SDF-based boundary supervision supplies signed-distance targets and alignment constraints so semantic predictions remain consistent with inside-outside boundary structure.The joint objective combines semantic, Lovasz, SDF, alignment, and ambiguity-regularization terms.

4 Experiments and Evaluations

AdapToPASS is evaluated on indoor and outdoor panoramic benchmarks, including robustness to unseen spherical transformations and component contributions. It achieves strong overall and transformation-robust performance, while its Swift variant improves compact-model robustness but severe perturbations remain challenging.

  • Settings: Experiments use Stanford2D3D and WildPASS, with WildPASS restricted to three foreground classes covering approximately 92% of annotated labels.All methods are trained on untransformed data; transformed variants are used only for evaluation.
  • Settings: Evaluation covers Rotation, Scale, Translation, Orientation Shift, and Viewpoint Shift across multiple difficulty levels without transformation-specific training augmentation.This setup tests generalization beyond canonical panoramic views.
  • Overall Results: 70.80% mIoU on Stanford2D3D and 69.58% mIoU on WildPASS give AdapToPASS the best overall performance across both datasets.It also achieves the highest average prediction confidence.
  • Robustness: +13.38% and +18.77% relative mIoU improvements on Stanford2D3D and WildPASS, respectively, establish the strongest robustness under unseen spherical transformations.AdapToPASS achieves the highest mIoU across transformation types and severity levels.
  • Ablation: Jointly integrating contextual ambiguity modeling, Bifocal Spherical Representation, and boundary supervision produces the strongest performance.Removing contextual modeling reduces confidence, while removing SDF supervision causes a notable mIoU drop.
  • Lightweight Variant: AdapToPASS-Swift has fewer than 2M parameters and generalizes better than similarly sized panoramic segmentation baselines, including under unseen transformations.It replaces AdaSpA with the lightweight AdaSpX module.
  • Limitations: Performance still degrades under extremely severe perturbations, particularly orientation and viewpoint shifts that introduce severe geometric ambiguity.The limitation is most pronounced in the stress setting.

5 Conclusion and Future Work

AdapToPASS is a bio-inspired Spherical Transformer for ambiguity-aware panoramic semantic segmentation. Its icosphere representation supports spherical processing through hierarchical sampling, resolution changes, and geometry-aware feature propagation.

  • AdapToPASS models contextual and geometric ambiguity through uncertainty-driven contextual and geometric processing inspired by biological vision.
  • Icosphere Representation: An icosphere rank r contains V_r = 10 · 4^r + 2 vertices and F_r = 20 · 4^r faces, with each subdivision increasing resolution fourfold.
  • Icosphere Representation: ERP panoramas are lifted onto the icosphere by converting vertex normals to spherical coordinates and bilinearly sampling RGB and depth features.
  • Icosphere Representation: At the default rank 7, the icosphere provides 163,842 vertices for resolving fine semantic boundaries while remaining computationally tractable.
  • Icosphere Representation: Parameter-free downsampling gathers matching vertices because each coarser icosphere vertex reappears exactly after subdivision.
  • Icosphere Representation: Hierarchical decoding preserves coarse-vertex features directly and averages adjacent coarse vertices to initialize new mid-edge vertices.

A.2 Training Objective and Optimization

AdapToPASS is trained with a unified objective combining semantic segmentation, signed-distance boundary supervision, and ambiguity regularization. Optimization uses AdamW with a warmup-cosine schedule over 400 epochs.

  • Training Objective: The unified training objective combines semantic segmentation supervision, SDF-based boundary terms, and an ambiguity regularizer.
  • Training Objective: Ground-truth labels, unsigned geodesic boundary distances, derived signed distance fields, and predicted SDFs define the boundary-supervision terms.
  • Training Objective: Semantic supervision uses class-balanced cross-entropy and Lovász-Softmax loss, with the latter weighted by λ_Lov = 1.0.
  • Training Objective: The ambiguity regularizer constrains CxAGB modules whose per-node ambiguity scores rescale geodesic attention windows from 0.8 to 1.6.
  • Optimization: Optimization uses AdamW with learning rates from 10^-4 to 10^-6, weight decay 0.01, and a warmup-cosine schedule over 400 epochs.

B.1 Unseen Spherical Transformations

The robustness evaluation generates transformed panoramic samples across several spherical perturbation families and three difficulty regimes. Transformations include image-space spherical changes and depth-based geometric viewpoint shifts.

  • Transformation Families: The evaluation tests Rotation, Scale, Translation, Orientation Shift, and Viewpoint Shift transformation families.
  • Transformation Families: Scale uses a spherical Möbius transformation that simulates zoom while preserving local angular structure.
  • Evaluation Scope: Stanford2D3D uses all five transformation families, whereas WildPASS uses Rotation, Scale, and Orientation Shift because translation requires paired depth panoramas.
  • Difficulty Levels: Mild, robust, and stress regimes represent progressively stronger perturbations from near-nominal conditions to substantial viewpoint and scale changes.
  • Generation Protocol: Samples are generated independently per panorama, transformation family, and difficulty regime using predefined severity-specific parameter sets.
  • Generation Protocol: Translation-based transformations lift ERP pixels into 3D with depth, move and optionally rotate points, then reproject them with nearest-depth collision resolution.

B.2 WildPASS Dataset and Label Space

WildPASS annotations are highly imbalanced, with road, sidewalk, and car dominating the labeled pixels. The experiments therefore use a four-class label space focused on these foreground categories and background.

  • Label Distribution: Road, sidewalk, and car together account for approximately 92% of labeled pixels in WildPASS.
  • Label Space: All categories except road, sidewalk, and car are collapsed into a single background class excluded from loss computation and evaluation.
  • Label Space: WildPASS experiments use the four-class label space {background, road, sidewalk, car}.
  • Evaluation Scope: Restricting evaluation to the three dominant classes prevents metrics from being dominated by background pixels or classes with too few samples.
  • Evaluation Scope: The selected classes have qualitatively different geometric footprints on the sphere, supporting robustness and transformation-invariance evaluation.

B.3 Baseline Implementation Details

The baseline comparisons standardize training, inputs, representations, and evaluation while detailing the spherical and perspective architectures used as references.

  • All models are trained from scratch with RGB inputs and evaluated on a common uniformly sampled sphere to avoid ERP-induced bias.
  • Baseline variants are selected to match AdapToPASS in depth, downsampling stages, parameter count, and spatial resolution as closely as possible.
  • The comparison includes spherical Transformers, grid-based models, ERP-icosahedral designs, and lightweight encoder-decoder architectures.
  • AdapToPASS uses a rank-7 icosphere, four encoder-decoder scales, depth two per scale, and an Acuity Stream with depth one.
  • AdapToPASS is trained for 300 epochs with AdamW, a 10^-4 learning rate, 0.01 weight decay, and warmup-cosine scheduling without explicit augmentation.

C Additional Results and Discussions

Additional experiments examine robustness, qualitative stability, ablations, and efficient spherical aggregation. Across unseen transformations, AdapToPASS maintains stronger segmentation consistency while its lightweight variant replaces attention with geometry-conditioned context aggregation.

  • Robustness Results: AdapToPASS achieves the highest WildPASS mIoU across difficulty levels and transformation types, especially under robust and stress settings.The evaluated transformations include rotation, scale, and orientation shifts.
  • Qualitative Analysis: AdapToPASS maintains more consistent semantic structure and confidence than existing methods on transformed Stanford2D3D and WildPASS inputs.Qualitative comparisons cover rotation, scale, orientation shift, translation, and viewpoint shift where applicable.
  • Ablation Study: Removing contextual ambiguity modeling, the Acuity Stream, or boundary supervision degrades semantic consistency and boundary quality.The full model produces predictions that better align with the ground truth.
  • Lightweight Variant: AdapToPASS-Swift uses fewer than 2M parameters while preserving ambiguity-aware spherical modeling for resource-constrained perception.Its design targets reduced architectural complexity and computational overhead.
  • AdaSpX Module: AdaSpX aggregates near and far geodesic neighborhoods to combine local geometric evidence with wider spherical context.The module is an attention-free spherical aggregation block operating on icosphere features.
  • AdaSpX Module: Uncertainty-guided fusion shifts toward broader context when uncertainty is high and preserves stronger local evidence when uncertainty is low.The block also uses lateral inhibition and a residual feed-forward structure.

D.2 Acuity Lateral Lightweight Architecture

The lightweight architecture preserves AdapToPASS’s bifocal perception through Acuity and Lateral streams that exchange information using uncertainty-gated fusion across spherical resolutions.

  • Acuity Stream: The Acuity Stream preserves fine local geometry and stable predictions at the finest available icosphere resolution.It uses compact near and far neighborhoods.
  • Lateral Stream: The Lateral Stream captures broad semantic context at coarser spherical resolutions and supports the Acuity Stream in ambiguous regions.Its larger neighborhoods provide wide-field contextual support.
  • Cross-Stream Fusion: Acuity and Lateral features exchange information through uncertainty-gated bidirectional fusion at each hierarchical level.Cross-stream messages and a predicted fusion uncertainty signal determine the gated updates.
  • Cross-Stream Fusion: Under higher uncertainty, fusion can rely more strongly on Lateral context, while confident regions retain stronger Acuity detail.This preserves the trade-off between contextual stabilization and fine-resolution structure.
  • Prediction Head: The final segmentation logits are predicted from the fused Acuity features using a lightweight segmentation head.

D.3 Training Objective

AdapToPASS-Swift uses a simple semantic segmentation objective while learning ambiguity-aware spherical representations through AdaSpX; its implementation specifies a compact rank-5 icosphere configuration.

  • Training Objective: AdapToPASS-Swift is trained with a semantic segmentation objective whose Dice term is optional and zero by default.The objective is designed to keep the training pipeline simple.
  • Training Objective: The training objective supports learning ambiguity-aware spherical representations through AdaSpX forward computation.
  • Takeaway: AdapToPASS-Swift combines adaptive context aggregation, geodesic local reasoning, surround stabilization, lateral inhibition, and uncertainty-gated stream fusion.These components preserve ambiguity-aware spherical perception while reducing full-model complexity.
  • Implementation: The Swift implementation uses rank-5 icosphere vertices, embedding dimension 24, dimension multipliers [1] [4], and encoder depths.It uses RGB inputs and an ERP grid width of 1024.
Loading 2608.29081v1…