Source-linked AI summary
EmbPASS: Towards Cross-Embodiment Open Panoramic Segmentation
Pujun Guo, Yuanfan Zheng, Fei Teng, Mengfei Duan, Guoqiang Zhao, Yuheng Zhang, Kai Luo, Kailun Yang
TL;DR
Heterogeneous embodied platforms produce different panoramic viewpoints and spatial layouts, yet systematic study of cross-embodiment panoramic segmentation remains limited. The paper introduces the task and EmbPASS benchmark, and proposes EPONet, which achieves the best platform-balanced EmbPASS performance.
Problem
Systematic study of panoramic semantic segmentation across heterogeneous embodied platforms is limited, despite platform-dependent observation shifts that challenge consistent perception and make dense annotations costly.
Source: §I · p. 1Method
The paper introduces Cross-Embodiment Open Panoramic Segmentation and EmbPASS, and proposes EPONet with RAMA and CAST for heterogeneous-platform panoramic perception.
Source: §I · pp. 1–2Results
35.82% mIoU gives EPONet the best platform-balanced EmbPASS performance, 1.10% above the strongest competing method, while remaining competitive on Stanford2D3D and DensePASS.
Source: §I · p. 2Takeaways & Limitations
EmbPASS provides a unified-taxonomy testbed for systematically studying cross-embodiment open panoramic segmentation across four representative platforms.
Source: §I · p. 2
Abstract
from arXiv · showhide
Panoramic images provide a complete 360-degree field of view, enabling comprehensive scene understanding for embodied perception. However, heterogeneous embodied platforms exhibit substantial differences in observation viewpoints and spatial layouts, giving rise to cross-embodiment observation shifts that pose additional challenges to consistent and reliable panoramic perception, while systematic studies of this problem remain limited. To bridge this gap, we introduce a new task, termed Cross-Embodiment Open Panoramic Segmentation. Meanwhile, we establish EmbPASS, a multi-platform panoramic semantic segmentation benchmark spanning Vehicle, Drone, Wearable, and Quadruped platforms under a unified semantic taxonomy, providing a testbed for systematically studying cross-embodiment panoramic perception. We further propose EPONet, an open-vocabulary panoramic semantic segmentation network that integrates Relation-Aware Metric Adapter (RAMA) and Content-Adaptive Semantic Transfer (CAST) to enhance spatial modeling and semantic transfer under heterogeneous embodied observations. Extensive experiments show that EPONet achieves the best platform-balanced performance on EmbPASS with 35.82% mIoU, outperforming the strongest baseline by 1.10%, while remaining competitive on existing panoramic segmentation benchmarks. The source code and EmbPASS benchmark will be made publicly available at https://github.com/guopj1/EmbPASS.
I. INTRODUCTION
The paper frames cross-embodiment observation shifts as a challenge to reliable panoramic perception and introduces a task and benchmark for studying it. It proposes EPONet and reports the best platform-balanced EmbPASS performance.
I. INTRODUCTION
- Different platform viewpoints and sensor mounting conditions create observation shifts, while dense panoramic annotation for each platform is costly as platform diversity grows.Source: §I · p. 1
- The task applies open-vocabulary segmentation directly to panoramic images from unseen embodied platforms, using pinhole images from a label-rich source domain without target-platform annotations.Source: §I · pp. 1–2
- EmbPASS spans Vehicle, Drone, Wearable, and Quadruped under a unified taxonomy, providing a testbed for platform-dependent observation variation and cross-embodiment segmentation.Source: §I · p. 2
- EPONet combines RAMA, which structures spatial aggregation under local geometric distortion, with CAST, which adaptively transfers intermediate CLIP representations across observations.Source: §I · p. 2
- 35.82% mIoU gives EPONet the best platform-balanced EmbPASS performance, 1.10% above the strongest competing method, while it remains competitive on Stanford2D3D and DensePASS.Source: §I · p. 2
(STATIONARY SCANNER),
Prior work addresses open-vocabulary segmentation and panoramic perception, but open-vocabulary panoramic segmentation across heterogeneous platforms remains largely unexplored. The section also summarizes EmbPASS statistics and cross-platform distributions.
A. Open-Vocabulary Semantic Segmentation
- Open-vocabulary segmentation methods adapt vision-language models through pixel-level similarities or mask proposals, but most are designed and evaluated primarily on perspective images.Source: A. Open-Vocabulary Semantic Segmentation · p. 2
B. Panoramic Segmentation and Cross-Platform Perception
- Panoramic segmentation research addresses projection distortion and annotation needs, while OPS trains on FoV-restricted pinhole images and evaluates on unseen panoramic domains.Source: B. Panoramic Segmentation and… · p. 2
- Open-vocabulary panoramic segmentation across heterogeneous embodied platforms remains largely unexplored despite advances in panoramic and cross-platform perception.Source: B. Panoramic Segmentation and… · p. 2
- Fig. 2 encodes platform image statistics and latitude coverage, semantic-group fractions, and class-wise latitude distributions across the four subsets.Source: Figure 2 · p. 3
C. Adaptive Spatial Sampling
The section situates adaptive spatial sampling among methods for modeling geometric variation, then describes EmbPASS as a benchmark for heterogeneous-platform evaluation. Its construction yields balanced, pixel-annotated platform subsets.
C. Adaptive Spatial Sampling
- Adaptive spatial sampling can use low-dimensional, signal-dependent local metrics to construct structured, content-adaptive sampling supports, unlike pointwise learned offsets.Source: C. Adaptive Spatial Sampling · p. 3
A. Benchmark Overview
- EmbPASS addresses limited heterogeneous-platform coverage in existing panoramic benchmarks with four platforms under a unified taxonomy for evaluating generalization.Source: A. Benchmark Overview · p. 3
B. Dataset Construction
- EmbPASS retains 250 panoramic images per platform and provides pixel-level annotations for 1,000 images using the 19-class Cityscapes taxonomy.Source: B. Dataset Construction · p. 3
C. Dataset Statistics and Analysis
EmbPASS exhibits platform-dependent semantic composition and spherical layouts, with latitude-related distortion varying across observations. EPONet uses a frozen CLIP encoder and trainable side branch to generate masks and open-vocabulary predictions.
C. Dataset Statistics and Analysis
- Drone and Wearable cover the full spherical latitude range, while Vehicle covers most of it and Quadruped a narrower band; semantic-group proportions also vary by platform.Source: C. Dataset Statistics and Analysis · p. 3
- Ground-related and several movable-object classes shift toward more negative latitudes in Drone, while Quadruped distributions cluster near the equator, where ERP distortion conditions differ.Source: C. Dataset Statistics and Analysis · pp. 3–4
A. Overall Framework
- A frozen CLIP encoder and trainable side branch use RAMA for spatial modeling and CAST for adaptive feature transfer; the branch proposes masks that guide open-vocabulary region recognition.Source: A. Overall Framework · p. 4
B. Relation-Aware Metric Adapter
RAMA structures local spatial aggregation with a Randers metric that constrains sampling geometry, adapting that geometry from content and directional neighborhood relations.
B. Relation-Aware Metric Adapter
- RAMA parameterizes local sampling geometry with a Randers metric that captures anisotropy and directional asymmetry across varying panoramic distortions.The shared local geometry constrains the entire sampling support rather than predicting pointwise offsets.Source: B. Relation-Aware Metric Adapter · p. 4
- RAMA combines content features with directional center–neighbor similarities to predict relation-aware local metrics.It uses a center plus eight surrounding directions from a 3×3 neighborhood to form relation features.Source: B. Relation-Aware Metric Adapter · p. 4
- RAMA projects the Side feature P into a compact representation and applies GELU to produce Z.Z is the content representation used in relation-aware metric prediction.Source: B. Relation-Aware Metric Adapter · p. 4
- The local metric and drift predictors feed Π, which maps their raw outputs to valid Randers parameters.The drift is constrained to ensure a valid Randers metric.Source: B. Relation-Aware Metric Adapter · p. 4
- Metric-guided sampling locations become offsets from a canonical 3×3 grid for deformable aggregation over Z, followed by projection and residual fusion.The residual path uses a learnable channelwise scale.Source: B. Relation-Aware Metric Adapter · p. 5
C. Content-Adaptive Semantic Transfer
CAST adaptively reorganizes intermediate CLIP features using both CLIP and Side features, then controls their spatially varying contribution to the Side branch.
C. Content-Adaptive Semantic Transfer
- CAST adaptively transfers pretrained CLIP representations to the Side branch according to the current CLIP and Side features.It is designed to exploit transferable semantics under varying observations.Source: C. Content-Adaptive Semantic Transfer · p. 5
- At each stage, lightweight projections and a shared 3×3 content encoder produce logits from Side and CLIP features, which are combined and normalized into kernel K.K drives dynamic reassembly of the CLIP feature.Source: C. Content-Adaptive Semantic Transfer · p. 5
- A spatial modulation weight controls the residual transfer, while K determines how CLIP features are locally reorganized from the CLIP and Side representations.A modulates their spatially varying contribution to the Side branch.Source: C. Content-Adaptive Semantic Transfer · p. 5
A. Datasets and Evaluation Metrics
The evaluation trains on pinhole COCOStuff-164k images and tests open-vocabulary panoramic segmentation on EmbPASS and two public benchmarks, using mIoU and open mIoU. The implementation uses a frozen CLIP encoder and shared training and inference settings for baselines.
A. Datasets and Evaluation Metrics
- COCOStuff-164k supplies 118k training images from its 164k-image dataset, annotated with 171 semantic categories.Evaluation uses EmbPASS, Stanford2D3D, and DensePASS; Stanford2D3D averages three cross-validation folds.Source: A. Datasets and Evaluation Metrics · p. 5
- mIoU requires strict predicted-to-ground-truth category matching, while open mIoU grants partial credit using WordNet-based semantic similarity.Stanford2D3D has 1,413 evaluation images with 13 categories, and DensePASS has 100 images with 19 categories.Source: A. Datasets and Evaluation Metrics · p. 5
B. Implementation Details
- The implementation uses a frozen CLIP ViT-B/16 visual encoder and trains for 60k iterations with AdamW on one RTX 3090 GPU.All baselines use the same training and inference settings; inference uses sliding 640×640 windows with a 320×320 stride.Source: B. Implementation Details · pp. 5–6
C. Comparison with State-of-the-Art Methods
EPONet achieves the best platform-balanced performance on EmbPASS and strong results across the four platforms. It is also competitive on public panoramic benchmarks, with performance varying by benchmark.
C. Comparison with State-of-the-Art Methods
- 35.82% mIoU and 42.30% open mIoU are EPONet’s best platform-balanced EmbPASS results, exceeding OOOPS with RERP by 1.10% and 1.16%, respectively.EPONet uses 0.37M fewer learnable parameters than OOOPS with RERP.Source: C. Comparison with State-of-the-Art… · p. 6
- EPONet achieves the best mIoU on Vehicle, Wearable, and Quadruped at 40.78%, 38.87%, and 37.74%, respectively.On Drone, its 25.90% mIoU is 0.11% below the best, while its 33.66% open mIoU ties for best.Source: C. Comparison with State-of-the-Art… · p. 6
- On Stanford2D3D, EPONet ranks second in mIoU at 43.87%, while on DensePASS it achieves the best mIoU and open mIoU at 49.15% and 55.22%.On Stanford2D3D it also reaches 48.96% open mIoU.Source: C. Comparison with State-of-the-Art… · p. 6
D. Ablation Studies
Ablations show gains from RAMA and CAST, with their combination improving over the base model. Further comparisons support RAMA’s spatial aggregation design and show that CAST placement matters.
D. Ablation Studies
- 35.82% mIoU with both RAMA and CAST improves 2.00% over the 33.82% base; RAMA alone reaches 34.99% and CAST alone 34.78%.The reported gains indicate complementary effects from the two modules.Source: D. Ablation Studies · p. 6
- 34.99% mIoU makes RAMA the strongest tested spatial aggregation operator, 0.51% above MetricConv.RAMA constrains the full sampling support through shared local geometry and incorporates directional center–neighbor relations; the comparison also reports convolution, deformable operators, and DAO.Source: D. Ablation Studies · p. 7, Table V · p. 7
- Across nonzero projection-center latitudes, RAMA’s average gains are 0.53% over MetricConv, 0.31% over DAO, and 0.35% over DCNv3.The controlled SPT analysis averages paired positive and negative latitudes from −80° to 80° at 10° intervals.Source: D. Ablation Studies · p. 7
- CAST at stages {1, 2, 3} reaches 34.78% mIoU, while extending it to stage {0} reduces performance to 34.48%.Performance rises from 34.64% at stage {3} to 34.68% at stages {2, 3}; the authors adopt stages {1, 2, 3}, suggesting shallow features may be less suited to early reassembly.Source: D. Ablation Studies · pp. 7–8
E. Qualitative Analysis
The conclusion presents Cross-Embodiment Open Panoramic Segmentation, EmbPASS, and EPONet as contributions to panoramic perception across embodied platforms. Qualitative comparisons report more complete, spatially coherent segmentations from EPONet than from baselines.
E. Qualitative Analysis
- Compared with baselines, EPONet produces more complete, spatially coherent segmentation results across four embodied platforms.It better preserves semantic-region continuity and boundary integrity, with fewer local misclassifications and fragmented regions.Source: E. Qualitative Analysis · p. 8
VI. CONCLUSION
- EPONet achieves the best platform-balanced performance on EmbPASS while remaining competitive on Stanford2D3D and DensePASS.Source: §VI · p. 8
- EPONet integrates Relation-Aware Metric Adapter (RAMA) and Content-Adaptive Semantic Transfer (CAST) for cross-embodiment panoramic perception.Source: §VI · p. 8
- EmbPASS spans Vehicle, Drone, Wearable, and Quadruped platforms for cross-embodiment panoramic segmentation research.Source: §VI · p. 8