Source-linked AI summary
PanoFuse: Panorama-Enhanced Vision-Language-Action Learning with Decoupled Semantic-Geometric Routing
Peng Xu, Haoran Lin, Wanjun Jia, Kai Luo, Wenrui Chen, Zhiyong Li, Kailun Yang
TL;DR
Perspective-camera VLA policies have limited fields of view, while panoramic images introduce distortion and task-irrelevant content that complicate integration with pretrained semantics. PanoFuse adds panoramic semantic and geometric streams with structured routing to a pretrained VLA policy. Across seven real-robot settings, it reports 52.9% average success versus 30.0% for the best evaluated baseline.
Problem
Perspective-camera VLA policies provide limited workspace coverage, while panoramic observations pose distortion and task-irrelevant-content challenges for integration with pretrained semantic features.
Source: §I · p. 2Method
PanoFuse uses a panoramic branch to extract semantic and geometric representations and routes them separately to downstream state and action representations through structured block-wise attention.
Source: §I · p. 2, B. PanoFuse · p. 4Results
52.9% average success rate versus 30.0% for the best evaluated baseline across seven closed-loop real-robot settings.
Source: §I · p. 2Takeaways & Limitations
Real-world experiments report the highest overall average success rate and stronger generalization under unseen visual conditions, with ablations supporting complementary semantic and geometric roles.
Source: §VI · p. 7
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) policies have shown promising performance in language-conditioned robotic manipulation. However, most existing VLA systems rely on conventional perspective cameras with limited fields of view, often missing global scene context and leading to unreliable manipulation under visual occlusions, distractors, and unseen environments. In this work, we propose PanoFuse, a panorama-enhanced VLA framework that complements local manipulation observations with global panoramic perception. PanoFuse introduces a dedicated panoramic branch that leverages a pretrained panoramic foundation model to extract complementary semantic and geometric representations from omnidirectional observations. Rather than directly mixing these heterogeneous features, we introduce Decoupled Semantic-Geometric Routing (DSGR), which maintains semantic and geometric representations as separate context streams and selectively routes both to downstream state and action representations through structured block-wise attention. This design provides the action expert with global spatial context while preserving task-relevant semantic information from the pretrained VLA backbone. We further develop a synchronized data collection pipeline and construct a new real-world manipulation dataset containing panoramic RGB observations, wrist-view images, language instructions, robot states, and actions. Across seven evaluation settings, PanoFuse achieves an average success rate of 52.9%, outperforming the evaluated baselines and achieving consistent gains under novel-object, unseen-background, and distractor-rich settings. Code and data will be released publicly at https://xux-hnu.github.io/PanoFuse.
I. INTRODUCTION
Perspective-camera VLA policies observe only local workspace regions, while panoramic perception offers broader context but poses semantic and geometric integration challenges. PanoFuse adds a panoramic branch with decoupled routing, and its real-robot evaluation reports higher average success than the best evaluated baseline.
I. INTRODUCTION
- Most VLA policies use narrow-field perspective cameras, leaving task-relevant objects outside view, occluded, or lost as the robot moves.These limitations are pronounced in cluttered and unstructured environments.Source: §I · p. 2
- Panoramic encoders must handle equirectangular distortion and task-irrelevant content while extracting global structure for integration with pretrained VLA semantics.Prior approaches can add calibration, exploratory motion, or reconstructed scene content, while perspective-camera coverage remains restricted.Source: §I · p. 2
- PanoFuse keeps panoramic semantic and geometric representations in separate streams and routes them to state and action representations through structured block-wise attention.The design aims to provide global spatial context while preserving the pretrained VLA backbone’s semantic priors and action-generation capability.Source: §I · p. 2
- 52.9% average success rate versus 30.0% for the best evaluated baseline across seven closed-loop real-robot settings.The settings cover cross-view and sequential manipulation, plus generalization to novel objects, unseen backgrounds, and visual distractors.Source: §I · p. 2
- The work builds a panorama-based manipulation dataset and evaluates PanoFuse across seven settings, reporting consistent gains in standard and generalization settings.The collected data include 200 successful teleoperated trajectories and over 100K synchronized frames.Source: §I · p. 2
II. RELATED WORK
VLA systems generate actions from visual observations and language, with approaches spanning discrete-token prediction and continuous generative modeling. Related work also explores richer perception and broader scene coverage; the supplied pipeline figure depicts PanoFuse’s panoramic routing and action-generation stages.
A. Vision-Language-Action Systems
- Early VLA approaches predict discrete action tokens, while more recent methods use generative paradigms such as flow matching for continuous, multimodal action distributions.The cited approaches include, for autoregressive discrete tokens and, [14] for generative paradigms.Source: A. Vision-Language-Action Systems · p. 2
- VLA perception research addresses visual blind spots, occlusion, and spatial reasoning through complementary sensing, predictive or recovery methods, stereo, and multi-view observations.The passage cites visual blind-spot work [15]–[17], occlusion methods [18]–, stereo [22], [23], and multi-view methods [10], –.Source: A. Vision-Language-Action Systems · p. 2
- The PanoFuse pipeline extracts panoramic semantic and geometric tokens, routes their contexts to state and action representations, then conditions an action expert to generate an action chunk.The panoramic tokens come from frozen DAP [12].Source: Figure 2 · p. 3
B. Panorama-Based Perception in Robotics
Panoramic vision broadens workspace coverage from a single viewpoint, and PanoFuse organizes panoramic semantic and geometric representations as separate streams. Its formulation combines wrist-view, panoramic, language, and proprioceptive inputs to predict action chunks.
B. Panorama-Based Perception in Robotics
- Panoramic cameras provide nearly complete 360° × 180° coverage from one viewpoint, broadening scene context and reducing perceptual blind spots.The passage notes applications in object detection, segmentation, 3D understanding, navigation, and tracking.Source: B. Panorama-Based Perception in Robotics · p. 3
III. METHODOLOGY
- PanoFuse keeps panoramic semantic and geometric representations separate and uses structured attention to expose both streams to state and action representations.The method section describes integrating these streams with wrist-view, language, state, and action contexts to condition the action expert.Source: B. Panorama-Based Perception in Robotics · p. 3, §III · p. 3
A. Problem Formulation
- The observation combines local wrist-view and panoramic RGB views with proprioceptive state; given that observation and a language instruction, the policy predicts a horizon-H continuous action chunk.The panoramic view captures the surrounding workspace and provides global scene context.Source: A. Problem Formulation · pp. 3–4
B. PanoFuse
PanoFuse adds a pretrained panoramic branch to π0 while preserving its vision-language pathway. DSGR keeps panoramic semantic and geometric contexts separate and routes them to downstream state and action representations.
B. PanoFuse
- PanoFuse augments pretrained π0 with a dedicated panoramic branch while preserving its pretrained vision-language pathway.It uses DAP to extract complementary semantic and geometric representations from panoramic RGB observations.Source: B. PanoFuse · p. 4
- Panoramic semantic and geometric features are separately pooled into 8×8 token grids and mapped into the VLM embedding space with separate learnable projections.The semantic features come from the DINO backbone, while geometric features come from an intermediate depth-decoder layer.Source: B. PanoFuse · p. 4
- DSGR organizes semantic, geometric, state, and action tokens into four blocks and expands block-level visibility into a token-level attention mask.The mask specifies which query tokens may attend to key/value tokens; invalid or padded tokens are also masked.Source: B. PanoFuse · p. 4
- Semantic and geometric blocks cannot attend directly to each other, but each can independently inform the state and action blocks.The visibility pattern is fixed at block level, while attention weights within permitted connections remain input-dependent.Source: B. PanoFuse · p. 4
- PanoFuse retains π0’s conditional flow-matching objective and changes the multimodal conditioning structure through DSGR.The action expert processes the multimodal representations under the DSGR token-level attention mask.Source: B. PanoFuse · p. 4
IV. TELEOPERATION SYSTEM AND DATASET
The teleoperation system synchronously collects panoramic and wrist-view sensing during human operation for real-world manipulation data collection.
IV. TELEOPERATION SYSTEM AND DATASET
- A leader–follower teleoperation system is used for real-world manipulation data collection.Source: §IV · p. 4
- The system synchronizes panoramic sensing with wrist-view sensing during teleoperation.Source: §IV · p. 4
- The pipeline is established for real-world manipulation data collection.Source: §IV · p. 4
A. Teleoperation System
The teleoperation setup maps leader-arm motion to a Piper follower and combines panoramic workspace coverage with local wrist-camera observations.
A. Teleoperation System
- A seven-servo leader arm maps motion relative to its initial pose into absolute joint-position commands for the Piper follower.Six leader channels correspond to manipulator joints and one controls the gripper; the follower-side controller uses the Piper API.Source: A. Teleoperation System · p. 4
- The follower is a 6-DoF AgileX Piper manipulator with a 1-DoF gripper, controlled at 30 Hz.Source: A. Teleoperation System · p. 4
- An Insta360 X5 captures 1024 × 512 equirectangular panoramic RGB beside the platform, while a wrist-mounted RealSense D435i captures 640 × 480 RGB.The panoramic camera provides continuous 360° scene coverage; the wrist camera provides local views around the end effector and manipulated objects.Source: A. Teleoperation System · p. 4
B. Data Collection Protocol and Dataset
The collection protocol synchronously records panoramic and wrist-view observations, instructions, robot states, and actions. The dataset contains successful demonstrations spanning four manipulation tasks.
B. Data Collection Protocol and Dataset
- During teleoperation, panoramic and wrist-view RGB, language instructions, robot states, and executed actions are synchronously recorded.Each trajectory is associated with a single natural-language task instruction.Source: Figure 3 · p. 5, B. Data Collection Protocol and Dataset · p. 5
- Robot states and actions each comprise six arm-joint dimensions plus one gripper dimension, and data are recorded at 30 Hz.Source: B. Data Collection Protocol and Dataset · p. 5
- The dataset contains 200 successful expert demonstrations across four real-world tasks, with 50 trajectories per task and more than 100K synchronized timesteps.Source: B. Data Collection Protocol and Dataset · p. 5
- The four tasks cover cross-workspace transfers, local pick-and-place, and sequential multi-cup placement.They are Front-Take-Back-Place, Back-Take-Front-Place, Pick-Place, and Seq-Cup-Insert.Source: B. Data Collection Protocol and Dataset · p. 5
A. Experimental Setup
The experiments evaluate PanoFuse across seven real-world settings, using repeated closed-loop trials and comparisons with π0-based baselines under shared panoramic training conditions.
A. Experimental Setup
- The seven settings cover four standard tasks and three F→B generalization conditions: Novel-Object, Unseen-Background, and Distractor.Novel-Object changes the target bottle, Unseen-Background adds a white tablecloth, and Distractor adds nearby objects.Source: A. Experimental Setup · p. 5
- Each method receives 10 trials per setting, and success requires completing the task without human intervention.The primary metric is task success rate, reported per setting and averaged across all seven; comparisons include π0-base and π0-fast variants.Source: A. Experimental Setup · p. 5
- PanoFuse and panorama-augmented baselines use the same 200 panoramic demonstrations, while standard π0 baselines use separate demonstrations and perspective cameras.PanoFuse uses pretrained DAP features and trains for 20K steps; the panorama-augmented baselines process raw panoramas through their original visual pathways.Source: A. Experimental Setup · pp. 5–6
B. Real-World Experiments
Across real-world evaluations and ablations, PanoFuse outperformed the evaluated baselines, while results favored complementary panoramic representations, structured routing, and broader scene coverage.
B. Real-World Experiments
- 52.9% average success rate was the highest across seven settings, exceeding π0-base by 24.3 percentage points and π0-fast by 22.9 percentage points.PanoFuse also maintained strong performance across all four standard tasks.Source: B. Real-World Experiments · p. 6
- 12.9% and 10.0% average success rates were achieved by π0-base w/ Pano and π0-fast w/ Pano, respectively, below their standard counterparts.These panorama-augmented baselines directly processed raw equirectangular images through their original visual pathways.Source: B. Real-World Experiments · p. 6
- 50.0%, 40.0%, and 40.0% success rates were achieved by PanoFuse in Novel-Object, Unseen-Background, and Distractor, where no baseline succeeded.Each setting comprised 10 reported trials.Source: B. Real-World Experiments · p. 6
C. Ablation Studies
- 60.0% average success with both panoramic representations exceeded the semantic-only and geometric-only variants at 22.5% and 20.0%, respectively.PanoFuse also outperformed both single-representation variants across all four manipulation tasks.Source: C. Ablation Studies · p. 7
- 60.0% average success with DSGR exceeded Partial Block-wise Attention at 10.0%; Unrestricted Attention completed none of the four tasks.DSGR separately contextualizes semantic and geometric representations while allowing state and action blocks to access both.Source: C. Ablation Studies · p. 7
- 43.3% average success was achieved at 360° FoV, versus 6.7% at 70°, 10.0% at 120°, and 26.7% at 180°.For Novel-Object, success rose from 0% at 70° and 120° to 10.0% at 180° and 50.0% at 360°.Source: C. Ablation Studies · p. 7
VI. CONCLUSION
PanoFuse combines panoramic perception with structured integration of semantic and geometric context for real-world manipulation. Its experiments report strong overall success and generalization, while identifying broader robot embodiments and richer sensing as future directions.
VI. CONCLUSION
- PanoFuse extracts complementary semantic and geometric representations from panoramic observations and integrates them with local visual, language, proprioceptive, and action representations.The integration uses structured block-wise attention.Source: §VI · p. 7
- Across seven evaluation settings, PanoFuse achieved the highest overall average success rate and stronger generalization under unseen visual conditions.Ablations support complementary semantic and geometric representations and structured information integration.Source: §VI · p. 7
- Future work could extend PanoFuse to heterogeneous robot embodiments and explore richer multimodal sensing.The stated aims are improved cross-platform generalization and robustness in diverse manipulation environments.Source: §VI · p. 7