PaperlayerView exact revision on arXiv

Source-linked AI summary

PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World

Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen

arXiv:2605.13169v2cs.CVcs.AI

TL;DR

Perspective-based MLLMs struggle with spatial reasoning because they do not natively represent the continuous geometry of 360° panoramas. PanoWorld combines pano-native supervision, spherical spatial cross-attention, and PanoSpace-Bench, substantially outperforming baselines across panoramic reasoning and related benchmarks.

  • Problem

    Perspective-based MLLMs struggle with full-surround spatial reasoning because ERP panoramas introduce geometric distortions and representation gaps beyond ordinary perspective images.

  • Method

    PanoWorld formulates pano-native capabilities, constructs geometry-aware supervision and evaluation data, and injects spherical geometry through spherical spatial cross-attention.

  • Results

    56.5 overall, up from 30.8 for the Qwen3.5 baseline, with gains across direction, boundary, spherical relation, 3D spatial, and seam reasoning.

  • Takeaways & Limitations

    The results support pano-native spatial learning over treating ERP panoramas as wide 2D images or relying only on prompt-level coordinate descriptions.

  • Takeaways & Limitations

    Automatic detection, annotation, re-detection, and depth-estimation errors may propagate into metadata, while PanoSpace-Bench excludes dynamic and interactive panoramic tasks.

Abstract

from arXiv · show

Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

1 Introduction

PanoWorld addresses the limitations of perspective-image MLLMs by formulating pano-native spatial understanding as capability-structured reasoning over observer-centered 360° environments. It combines panoramic data construction, dedicated modeling, and benchmarking, with experiments showing competitive performance and transfer to existing 360° reasoning and VLN benchmarks.

  • Introduction: Perspective-image MLLMs inherit a limited instantaneous field of view, hindering spatial reasoning for visual search, navigation, and immersive scene understanding.
  • Introduction: Panoramic understanding cannot be obtained by directly transferring perspective models because the representations differ in geometric distortion, non-uniform spatial sampling, and boundary discontinuities.
  • Introduction: The proposed pano-native framework learns an observer-centered representation of 360° environments and decomposes understanding into spherical localization, direction reasoning, viewpoint transformation, 3D relations, and scene topology.
  • Introduction: A scalable metadata-driven pipeline constructs large-scale panoramic data, while a comprehensive benchmark evaluates the defined spatial reasoning capabilities.
  • Introduction: Experiments show competitive performance on the proposed benchmark and effective transfer to existing 360° reasoning and VLN benchmarks, with substantial gains over prior methods.

2 Related Work

Prior work addresses panoramic understanding through panorama-specific datasets, benchmarks, and pano-aware model designs, while broader multimodal research identifies spatial reasoning as a persistent bottleneck. Recent 360° visual search treats panoramas as discrete views, motivating ERP-native reasoning that explicitly models spherical geometry.

  • Pano-Native Panoramic Designing: Panorama-specific research develops task-specific datasets and benchmarks for perception and MLLM understanding, alongside pano-aware model designs.The surveyed efforts primarily target data and model adaptations for bridging panoramic and perspective understanding.
  • Spatial Reasoning in Multimodal Large Language Models: Spatial reasoning is a systematic bottleneck for multimodal large language models across spatial relations, 3D scenes, embodied interaction, and geometry-aware representation learning.Evaluation spans 2D/3D relations, depth order, relative distance, egocentric memory, and embodied question answering.
  • Pano-Native Panoramic Designing: Thinking in 360 studies human-centric visual search using discrete views, leaving panorama geometry implicit rather than treating the ERP panorama as the native spatial representation.This contrast motivates injecting spherical geometry into ERP visual tokens.

3 Method

PanoWorld defines pano-native understanding as continuous, observer-centered reasoning over ERP panoramas and builds capability-aligned supervision from verified geometric, semantic, and depth-aware metadata. Its spherical spatial adaptation injects ERP geometry into visual tokens while preserving the pretrained backbone.

  • 3.3 Metadata construction: The metadata pipeline collects mixed-source ERP panoramas, verifies detections through overlapping perspective views and ERP reprojection, and adds language-grounded annotations plus depth.It uses aligned depth when available and panoramic pseudo-depth otherwise, then aggregates depth over each entity’s ERP support region.
  • 3.1 Task formulation: Pano-native understanding requires reasoning over ERP panoramas as continuous spherical spaces, including seam continuity, viewpoint reorientation, and full-panorama entity relations.The model maps panoramas and text queries to answers, directions, spatial relations, or grounded targets.
  • 3.2 Capability taxonomy: The capability taxonomy covers semantic anchoring, spherical grounding, reference-frame transformation, and depth-aware 3D spatial reasoning.These families span entity semantics, yaw-pitch localization, rotation and seam-aware relations, and viewer-centered depth relations.
  • 3.3 Metadata construction: A structured metadata graph combines entity semantics, angular footprints, observer distances, visual context, and pairwise spherical and 3D relations to generate capability-aligned training tasks.The graph provides the interface between raw ERP data and downstream supervision, and also supports PanoSpace-Bench construction.
  • 3.4 Pano-aware model adaptation: PanoWorld computes spherical directions for ERP patch centers, encodes them with fixed sinusoidal embeddings, and adaptively fuses the resulting spatial tokens into visual tokens through a learnable gate.The adapted tokens pass through the remaining pretrained visual blocks, leaving the pretrained backbone unchanged while enabling pano-aware reasoning.

4 Experiments

Experiments show that pano-native supervision and spherical geometry adaptation substantially improve ERP-based spatial reasoning and transfer to panoramic search and navigation. Ablations further identify complementary ability-oriented data, metadata verification, patch-level cross-attention, and multimodal component updates as important contributors.

  • Evaluation: PanoSpace-Bench evaluates panoramic localization, spherical relational reasoning, omnidirectional 3D spatial reasoning, ERP properties, category-wise accuracy, and BFOV mIoU.The evaluation also includes H∗Bench and R2R-CE Val-Unseen.
  • PanoSpace-Bench: General-purpose MLLMs remain weakest on BFOV grounding, reference-frame transformation, and viewer-centered 3D reasoning despite relatively strong basic object recognition.The results identify continuous observer-centered ERP reasoning, rather than object semantics alone, as the central challenge.
  • PanoSpace-Bench: Prompt enhancement improves direct ERP inference, especially coarse localization, but yields limited gains on spherical relational and 3D spatial reasoning.Explicitly describing ERP layout does not resolve the deeper relational and 3D reasoning difficulties.
  • H∗Bench: 56.10 overall: the zero-shot ERP model outperforms the strongest perspective-view baseline at 38.40 on H∗Bench transfer.This transfer result indicates that pano-native supervision supports downstream panoramic search beyond PanoSpace-Bench.
  • VLN: 54.3 SR and 52.1 SPL: PanoWorld achieves these R2R-CE Val-Unseen results using only ERP panorama input.Unlike methods using waypoint predictors or panoramas mainly for candidate-view selection, PanoWorld directly consumes the ERP panorama as a unified full-surround input.
  • Ablations: 55.1 overall: combining detection and semantic verification outperforms the unverified baseline at 38.8, detection verification at 46.4, and semantic verification at 48.0.Patch-level cross-attention also improves accuracy from 0.484 to 0.551, while combining all ability families gives the best overall training-data results and multimodal updates remain important for ERP adaptation.

5 Conclusion

The paper presents a unified pano-native spatial learning framework that treats ERP panoramas as continuous observer-centered spaces. It organizes pano-native understanding into four capability families and supports them with large-scale metadata construction and capability-aligned supervision.

  • Pano-native understanding is formulated as reasoning over ERP panoramas viewed as continuous observer-centered spaces.
  • The framework decomposes pano-native understanding into semantic anchoring, spherical grounding, reference-frame transformation, and depth-aware 3D spatial reasoning.
  • A large-scale metadata construction pipeline is built to derive capability-aligned supervision for the framework.

A Dataset Details … A.4 Prompt Template

The paper supplements pano-native modeling with a mixed-source ERP corpus, geometry-aware metadata and instruction construction, and unified prompts that explicitly encode observer-centered spherical spatial reasoning. Its evaluation prompts further provide textual or visual ERP coordinate guidance for interpreting full-surround panoramas.

  • A.1 ERP Corpus Composition: The ERP corpus contains 570,321 full-surround panoramas, approximately balanced between 297,476 indoor and 272,845 outdoor scenes.The corpus combines mixed sources, including Realsee3D and 360+X, with redistribution limited by source licenses, data-use agreements, and terms of service.
  • A.1 ERP Corpus Composition: Panoramic data handling addresses licensing, privacy, and misuse because images may contain homes, bystanders, vehicles, or other sensitive content.The paper states that only assets compatible with corresponding source restrictions will be released.
  • A.2 Metadata Pipeline Details: Metadata construction renders overlapping 120° FoV perspective views at 60° yaw strides, then merges reprojected detections using 0.3 confidence and 0.5 view-level NMS IoU thresholds.WeDetect-Large is used as the open-world detector, and detected boxes are reprojected to ERP coordinates before cross-view merging.
  • A.3 Instruction Data Distribution: The pipeline instantiates 7.65M candidate instruction samples and samples a canonical training set of 2.998M examples across semantic, angular, reference-frame, and depth-aware tasks.Sampling limits repeated examples from the same scene within each task family.
  • A.4 Prompt Template: The unified system prompt defines each input as a continuous full-360-degree ERP observation centered on a fixed observer and uses an observer-centered reference frame.It standardizes BFOV localization, relative direction, camera rotation, object-conditioned reorientation, physical distance, and relative 3D position.
  • A.4 Prompt Template: Prompt-only ERP baselines receive either a textual reference appendix explaining ERP coordinates or a visual appendix combining coordinate grids with text instructions.These appendices are intended to teach models how to read ERP panoramas rather than replace spatial reasoning.
  • A.4 Prompt Template: The visual guidance grid uses yaw lines every 30° and pitch lines every 15°, with a yellow crosshair marking the front direction at (0°, 0°).Green vertical lines encode yaw, blue horizontal lines encode pitch, and labels support interpolation of target center directions [yaw, pitch].

B Benchmark Setting · B.1 PanoSpace-Bench · B.2 Human-centric visual search.

The benchmark setting evaluates pano-native spatial understanding through PanoSpace-Bench’s ERP-specific diagnostic tasks and H∗Bench’s human-centric visual-search protocols. It separates semantic recognition from spherical spatial correctness and uses distinct evaluation procedures for multiple-choice, BFOV localization, and interactive search.

  • B.1 PanoSpace-Bench: PanoSpace-Bench evaluates whether MLLMs understand equirectangular panoramas as continuous, observer-centered representations of omnidirectional 3D spaces.
  • B.1 PanoSpace-Bench: Benchmark panoramas and questions are separated from the ERP-native instruction-tuning corpus to reduce data leakage.Panoramas come from different Internet sources, are deduplicated against training data, and are manually verified for quality and valid ERP layout.
  • B.1 PanoSpace-Bench: The benchmark organizes evaluation into four pano-centered ability families covering eight diagnostic task categories.The taxonomy is presented in Table 12.
  • B.1 PanoSpace-Bench: PanoSpace-Bench separates semantic recognition from spatial correctness, requiring models to resolve object positions and relations in an observer-centered spherical frame.Its design emphasizes ERP-specific spatial modes rather than generic panoramic question answering.
  • B.1 PanoSpace-Bench: The benchmark contains 2,000 questions, with 250 questions per category; all tasks except BFOV localization use multiple-choice exact-choice accuracy.BFOV localization predicts [yaw, pitch, xfov, yfov] and is evaluated using angular IoU.
  • B.2 Human-centric visual search.: H∗Bench evaluates human-centric visual search in 360◦ panoramas through Humanoid Object Search and Humanoid Path Search.Its original protocol exposes a local field of view, allows iterative rotation actions, and ends with a submitted target direction.
  • B.2 Human-centric visual search.: For perspective-view baselines, the H∗Bench evaluation additionally reports average interaction steps and model calls before submission.Tolerance parameters follow the original paper, and hit(·) follows the benchmark evaluator.

B.3 R2R-CE vision-and-language navigation.

PanoWorld transfers to embodied vision-and-language navigation on R2R-CE Val-Unseen by directly using full-surround ERP panoramas to predict navigation directions. The evaluation follows the standard protocol and limits fine-tuning data for fair comparison with RGB/video-based methods.

  • R2R-CE transfer: PanoWorld directly takes the ERP panorama as visual observation and predicts navigation direction from the full-surround input on R2R-CE Val-Unseen.This contrasts with conventional VLN pipelines that construct candidate perspective views or use odometry, depth, or single-view RGB observations.
  • Evaluation protocol: The transfer evaluation follows the standard R2R-CE protocol and reports navigation error (NE) and oracle success rate (OSR).OSR uses the closest distance to the goal along the trajectory rather than the final distance.
  • Evaluation protocol: For fair comparison with recent RGB/video-based methods, VLN fine-tuning uses only the R2R and RxR training sets, with PanoWorld using 80% of the training data.SPL additionally penalizes unnecessarily long paths, alongside the reported navigation metrics.

C Case Study · D Efficiency Study

PanoWorld demonstrates pano-native reasoning through qualitative downstream-search and diagnostic examples, while direct ERP inference replaces iterative local-view rotation with a single, efficient full-surround prediction.

  • C Case Study: PanoWorld directly reasons over full ERP panoramas to infer target movement directions in practical 360° human-centric visual search without decomposing scenes into local views.These qualitative cases illustrate downstream transfer enabled by pano-native spatial learning.
  • D Efficiency Study: Direct ERP inference is more efficient than perspective-view rotation in interaction steps, sequential decision cost, and global spatial coverage on H* tasks.Perspective-view rotation observes one local FoV per step, whereas direct ERP reasoning consumes the full panorama and predicts the answer in one forward pass.
  • C Case Study: Representative PanoSpace-Bench cases cover 3D spatial relations, object-conditioned reorientation, and camera-rotation reasoning.The examples demonstrate controlled evaluation of observer-centered spherical reasoning.
  • D Efficiency Study: 3.58–6.34 interaction steps are required on average by perspective-view rotation, producing 18.7K–29.9K effective input tokens.These figures quantify the sequential burden of iterative local-view search.
  • D Efficiency Study: 1.13–1.81× is the effective input-token cost of perspective-view rotation relative to direct ERP inference.The comparison is reported for H* tasks in Table 14.
  • D Efficiency Study: One step, one model call, and 16.5K effective input tokens are sufficient for direct ERP inference while maintaining full 360° spatial coverage.This unified inference paradigm replaces iterative local-view search with full-surround prediction.

E Ablation study

The ablation study examines two pano-aware adapter design choices: the fusion mechanism and the adapter’s insertion position in the visual stream.

  • Architecture ablation: The fusion ablation compares simple residual fusion, which directly adds projected spherical features to visual tokens, with cross-attention fusion, which adaptively attends to spherical spatial tokens.These alternatives test whether visual tokens should receive spherical features directly or selectively integrate spherical spatial information.
  • Architecture ablation: The insertion-position ablation injects the spherical adapter at three stages of the visual stream, beginning immediately after patch embedding.The supplied passage introduces three insertion stages but does not specify the remaining two in the available text.

F Limitations

PanoWorld’s limitations arise from potential error propagation in automatically constructed metadata and the restricted scope of its diagnostic evaluation. Future work should improve panoramic perception and verification while extending benchmarks to interactive, temporal, and dynamic environments.

  • Limitations: Automatic detection, semantic annotation, referring re-detection, and panoramic depth estimation can propagate errors into the final metadata graph despite two-level verification.The pipeline relies on multiple automated components whose errors may accumulate in metadata construction.
  • Limitations: PanoSpace-Bench is designed as a diagnostic benchmark for observer-centered ERP reasoning, limiting its evaluation scope.The supplied passage identifies PanoSpace-Bench as diagnostic but is truncated before fully specifying its scope.
  • Future Work: Future work should develop more reliable panoramic perception modules and stronger cross-modal verification to improve metadata quality.These directions target the data-side weaknesses of the metadata construction pipeline.
  • Future Work: Benchmarks should expand from static ERP reasoning to interactive navigation, temporal panoramic videos, and dynamic 3D environments.This broader testbed is intended to assess full-surround spatial intelligence across more demanding settings.
Loading 2605.13169v2…