PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling World Model for Hierarchical Manipulation Policies

Qian Long, Yueze Wang, Jiaxi Song, Junbo Zhang, Peiyan Li, Wenxuan Wang, Yuqi Wang, Haoyang Li, Shaoxuan Xie, Guocai Yao, Hanbo Zhang, Xinlong Wang, Zhongyuan Wang, Xuguang Lan, Huaping Liu, Xinghang Li

arXiv:2602.10983v2cs.RO

TL;DR

VLA policies remain brittle under distribution shift and limited real-robot data, motivating a more grounded intermediate representation for long-horizon manipulation. VISTA uses a world model to generate textual subtasks and visual goals for a low-level GoalVLA controller, achieving 69% success versus 14% for language-guided π0 in novel scenarios. The method demonstrates improved out-of-distribution manipulation, while evaluation remains limited in task diversity and visual-goal accuracy can affect execution.

  • Problem

    VLA models remain brittle in out-of-distribution scenarios, particularly when real-world robot data is scarce, and existing subgoal representations trade semantic generalization against spatial and physical grounding.

  • Method

    VISTA uses a generative world model as a high-level planner for interleaved textual subtasks and visual goals, with a goal-conditioned VLA executing the resulting subgoals.

  • Results

    69% success was achieved on novel scenarios spanning 21 unseen objects using 2 hours of teleoperation data on 5 objects, versus 14% for language-guided π0.

  • Takeaways & Limitations

    Visually grounded subgoals provide a structured intermediate representation that supports robust and generalizable manipulation across unseen objects and scenes.

  • Takeaways & Limitations

    Real-world evaluation is limited to pick-and-place tasks, larger visual-subgoal deviations can cause execution failures, and transfer depends on corresponding interaction patterns in training data.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are promising for generalist robot manipulation but remain brittle in out-of-distribution (OOD) settings, especially with limited real-robot data. To resolve the generalization bottleneck, we introduce a hierarchical Vision-Language-Action framework \our{} that leverages the generalization of large-scale pre-trained world model for robust and generalizable VIsual Subgoal TAsk decomposition VISTA. Our hierarchical framework \our{} consists of a world model as the high-level planner and a VLA as the low-level executor. The high-level world model first divides manipulation tasks into subtask sequences with goal images, and the low-level policy follows the textual and visual guidance to generate action sequences. Compared to raw textual goal specification, these synthesized goal images provide visually and physically grounded details for low-level policies, making it feasible to generalize across unseen objects and novel scenarios. We validate both visual goal synthesis and our hierarchical VLA policies in massive out-of-distribution scenarios, and the performance of the same-structured VLA in novel scenarios could boost from 14% to 69% with the guidance generated by the world model. Results demonstrate that our method outperforms previous baselines with a clear margin, particularly in out-of-distribution scenarios. Project page: \href{https://vista-wm.github.io/}{https://vista-wm.github.io}

I. INTRODUCTION

VISTA addresses brittle out-of-distribution manipulation by decomposing tasks into visually grounded subgoals that connect high-level planning with low-level VLA control. With limited real-world data, it improves performance on novel objects and scenarios.

  • Motivation: VLA models remain brittle in out-of-distribution scenarios, especially when real-world robot data is scarce.Their training structure differs from pretrained vision-language models, and long-horizon tasks make direct instruction-to-action mapping data-intensive.
  • Motivation: Linguistic subgoals generalize semantically but lack spatial and physical constraints, while dense video prediction suffers temporal drift and physical inconsistency.This representation trade-off motivates an intermediate form that combines semantic guidance with grounded visual information.
  • Approach: VISTA uses a world model to generate compact visual subgoals interleaved with textual subtasks, while a VLA policy executes them.The visual subgoals provide explicit spatial guidance beyond language and avoid reliance on dense future-frame prediction.
  • Results: 69% success on novel scenarios spanning 21 unseen objects was achieved with 2 hours of teleoperation data on 5 objects, versus 14% for language-guided π0.The evaluation emphasizes multi-stage manipulation and out-of-distribution generalization.
  • Approach: The method includes a scalable pipeline that relabels millions of robot trajectories into interleaved textual subtasks and visual goal images.These representations support training the world model and goal-conditioned policy at scale.
  • Contribution: VISTA is presented as a hierarchical framework combining a generative world model with a goal-image-conditioned policy that outperforms standard baselines in out-of-distribution settings.The paper positions this combination as its central framework contribution.

B. World Models for Robot Manipulation

VISTA places world-model planning and goal-conditioned control within a hierarchical manipulation system. It uses discrete, visually grounded milestones to preserve task structure and provide actionable termination references.

  • World Models: World models learn physical dynamics from large-scale data and can support prediction, planning, and visual representations for robot manipulation.Related approaches include video pretraining, latent action models, trajectory scoring, and latent-space planning.
  • Hierarchical Architecture: VISTA separates high-level subtask planning from low-level motor control through a world model and a GoalVLA policy.The planner generates interleaved textual subtasks and visual goals, while the controller predicts executable action chunks.
  • Hierarchical Architecture: The world model predicts intermediate subgoals from the global instruction and execution history, and the policy maps the current observation plus subgoal to actions.The subtask specifies what to do, while the visual goal specifies how the target state should appear.
  • Execution: Execution continues until the current observation aligns visually with the goal, after which the system updates history and queries the world model for the next stage.This creates an explicit stage-transition mechanism for long-horizon manipulation.
  • Design Rationale: Discrete key milestones provide structured references for both action execution and termination verification, avoiding ambiguity associated with dense video prediction.The formulation is intended to improve compositional reasoning while retaining explicit spatial guidance.

A. World Model Planner

VISTA uses an embodied world model to autoregressively plan interleaved textual subtasks and visual goal images, then conditions a low-level VLA on each milestone to generate actions. Training and execution include mechanisms for coherent sequence generation and robustness to subtask-boundary misalignment.

  • World Model Planner: The world model represents global instructions, observations, textual subtasks, and visual goals as one token sequence for autoregressive planning.Images and text are jointly tokenized into a shared vocabulary and trained with causal next-token prediction.
  • World Model Planner: During inference, beam search expands candidate milestone sequences and selects the highest-probability sequence before reconstructing goal images.The selected discrete sequence is inverse-tokenized into pixel-level visual goals.
  • Goal-Conditioned VLA: The GoalVLA combines the current observation, textual subtask, and visual goal image to predict an action chunk toward the milestone.Goal-image tokens are concatenated with current-observation tokens before entering the policy backbone.
  • Goal-Conditioned VLA: Flow matching trains the policy to generate continuous action trajectories conditioned on the current subtask, observation, and goal image.The objective predicts a velocity field along interpolation between noise and a ground-truth action chunk.
  • Goal-Conditioned VLA: Zero-padding after milestone completion teaches the policy to stop before prematurely executing the next stage.Randomly switching between adjacent goals within a temporal overlap window improves robustness to predicted goal-image timing errors.

C. Dataset Curation

The dataset curation pipeline converts large-scale embodied trajectories into interleaved textual subtasks and visual goals, while experiments use limited demonstrations and structured in-domain and OOD scenarios. The resulting corpus combines multiple embodiments, views, and auxiliary image data for world-model training.

  • Dataset Source: The curation pipeline starts from Open-X-Embodiment, AgiBot World Beta, and Mobile Aloha data totaling more than 1M trajectories.These datasets provide global instructions and image–action pairs but lack intermediate milestones.
  • Automatic Milestone Labeling: An automated pipeline identifies atomic skills, detects milestone boundaries from motion and gripper changes, and generates subtask descriptions.It uses a 50-skill library, RDP motion segmentation, gripper transitions, and Qwen2.5-VL merging.
  • Automatic Milestone Labeling: The process converts 1.2M trajectories into interleaved subtask and goal-image sequences spanning 14 embodiments with multi-view support and 15.2B tokens.AgiBot task types are additionally refined into fine-grained instructions.
  • Any-to-Image Pretraining: World-model training is supplemented with a 15.0B-token Any-to-Image dataset assembled from several image-generation and editing sources.The cited passage directs further dataset details to Appendix B.
  • Real-World Tasks and Setups: The real-robot training set contains 2 hours of pick-place demonstrations involving five objects, with π0 and π0-subtask used as baselines.π0-subtask replaces the original dataset instructions with subtasks.
  • Real-World Tasks and Setups: Evaluation includes in-domain scenes with unseen distractors or targets and OOD scenes built from 21 novel objects across multiple scene setups.The experiments visualize these in-domain and novel scenarios in Fig. 4.

B. Visual Subgoal–based Task Decomposition

VISTA’s world model generates interleaved textual subtasks and visual goal images that remain physically, spatially, and semantically coherent across novel manipulation settings. These visual subgoals support generalization across objects, views, embodiments, and task compositions.

  • VISTA generates physically consistent manipulation processes and maintains multi-view consistency across diverse objects, backgrounds, and scenarios.
  • VISTA converts language-grounded semantic and spatial understanding into visual goal images for unseen long-horizon task compositions.
  • Under OOD visual shifts, generated subtask and goal-image sequences show coherent planning, plausible interactions, and realistic object motion without attachment artifacts.
  • VISTA transfers object affordances and skill knowledge across embodiments, including unseen spray bottles in RT-1 and laptops in Bridge.
  • VISTA preserves fine-grained instruction understanding by correctly removing only specified letters despite training on full-board cleaning tasks.

2) Qualitative Results on Real-Robot Scenarios:

Real-robot evaluations show that VISTA generates spatially and temporally coherent multi-view goals across novel scenarios, task compositions, and robot embodiments. Quantitative comparisons further show stronger robustness than language-conditioned baselines, especially for unseen targets.

  • VISTA generates spatially consistent goal images across head, left-wrist, and right-wrist cameras under unseen layouts, objects, backgrounds, and viewpoints.
  • Generated goals preserve actionable temporal structure, including approach trajectories, grasp poses, arm configurations, and object relocation.
  • Multi-view goals add precise spatial constraints beyond textual subtasks, improving GoalVLA’s interpretation of task requirements and downstream control robustness.
  • VISTA composes skills across different arm-usage constraints and transfers generated sequences to Aloha, AgiBot G1, and WidowX in unseen environments.
  • The evaluation compares basic, unseen-distractor, and unseen-target settings using approach success (App) and execution success (Suc).
  • 100% approach success is achieved in unseen-target settings, while π0 performs poorly and π0-subtask still fails during execution.

2) Qualitative Analysis:

In novel scenarios, VISTA succeeds where language-conditioned baselines fail because visual goals encode interaction regions, end-effector poses, and other spatial cues. The paper concludes that this hierarchy improves robustness, while identifying limits in task scope and goal precision.

  • With unseen objects, layouts, and backgrounds, π0 and π0-subtask fail qualitatively while VISTA succeeds in the illustrated scenarios.
  • Baseline grasp failures arise from interpolating action chunks from visually similar training objects, such as using a can-grasping posture for an unseen bottle.
  • VISTA’s temporally aligned multi-view goals encode interaction regions and end-effector poses, enabling accurate position and posture adjustment for unseen-object grasping.
  • Goal-image spatial cues improve arm selection, end-effector alignment, and execution when language-only guidance misidentifies or cannot precisely grasp unseen targets.
  • The framework bridges semantic reasoning and low-level control through an explicit, verifiable intermediate representation.
  • Real-world evaluation remains limited to pick-and-place, and larger visual-subgoal deviations can cause GoalVLA execution failures.

APPENDIX A IMPLEMENTATION DETAILS

The appendix details VISTA’s world-model and GoalVLA architectures, training procedures, dataset construction, and sequence-formatting choices. It also describes the multimodal data sources and annotation rules used to create interleaved subtask and goal-image sequences.

  • World Model Architecture: The world model contains 34.1 billion parameters and uses a unified vocabulary of 282,926 tokens with discrete visual patch tokenization.Its transformer layers contain 31.2 billion parameters, while embedding layers contain 2.9 billion.
  • Sequence Formatting: World-model sequences interleave text and visual tokens, with supervision requiring autoregressive prediction of the complete subtask and image sequence.Sequences contain at most 16 images and may begin at an arbitrary subtask stage.
  • World Model Training: VISTA is trained for 2,000 continued-training steps on EMU3.5 using 128 Nvidia H100 GPUs over 2 days.The setup uses Megatron-LM, a global batch size of 512, and a learning rate of 1 × 10^-5.
  • GoalVLA Architecture: GoalVLA adopts a PaliGemma-3B backbone with a simultaneous 0.3B-scale action expert and block-wise causal masking.The architecture separates VLM, proprioception, and action blocks while allowing the action block to attend to all blocks.
  • GoalVLA Training: GoalVLA uses two-stage training with 200,000 AgiBot-Beta trajectories followed by fine-tuning on 737 self-collected robot trajectories.The first stage adds annotated subtask text and goal images as conditioning.
  • Dataset Construction: The dataset construction pipeline splits 1.2 million trajectories into interleaved subtask and goal-image formats across 10 embodiments and multiple views.Subtasks are required to be concise, continuous, skill-specific descriptions linked to manipulated objects or targets.

APPENDIX C VISUALIZATION OF GENERATED RESULTS FOR UNCONSTRAINED SCENARIOS

VISTA is evaluated on unconstrained scenarios that differ substantially from training data in visual and contextual conditions. It generates alternating subtasks and physically plausible, instruction-following goal images in these settings.

  • Evaluation Scenarios: VISTA is tested on both text-to-image-generated scenes and real-world photographs captured with mobile devices.The evaluation targets diverse, unconstrained scenarios rather than a fixed training-style environment.
  • Generation Results: On novel scenarios, VISTA alternately generates reasonable subtasks and physically plausible goal images conditioned on robot arm type and task.The generated sequences are described as instruction-following across the unconstrained settings.
  • Distribution Shift: The evaluated scenarios differ from training samples in layout, object appearance, background, and camera view.These differences are visualized in the generated samples for unconstrained scenarios.

APPENDIX D ILLUSTRATION OF REAL-WORLD EXPERIMENT SETUP

The real-world experiments contrast a homogeneous five-object training setup with 63 novel scenarios containing unseen objects, surfaces, and plate colors. VISTA generates timing-accurate, appearance-consistent goal images and object-specific grasping strategies in these novel settings.

  • In-Domain Setup: The in-domain training set contains five fixed objects, five put-on-plate tasks, and 737 trajectories, with highly homogeneous scene composition and object placement.Each task includes approximately 150 trajectories.
  • Novel Scenarios: The novel evaluation includes 21 unseen objects, three tablecloth patterns, and unseen plate colors, forming 63 scenarios.Object positions are randomized while remaining reachable and graspable by the robot arm.
  • Goal Image Quality: VISTA’s generated goal images are generally timing-accurate and capture critical picking and placing moments for spatial guidance.The images maintain consistency in background and object appearance under novel scenarios.
  • Grasping Strategies: VISTA generates object-specific grasping modes, including a horizontal forward grasp for a soda water bottle instead of the common vertical downward grasp.The authors identify this diversity as important for handling unseen objects.
  • Visualizations: The appendix visualizes training data, novel-scene setups, and generated goal images for comparison across the real-world experiment conditions.These visualizations correspond to the in-domain and novel-scenario configurations.

APPENDIX F EMERGING CAPABILITY ANALYSIS

VISTA exhibits emerging capabilities in compositional, spatial, and semantic manipulation understanding. These capabilities support multi-object planning, spatially specified actions, semantic target identification, and stronger goal-conditioned execution by GoalVLA.

  • Compositional Understanding: VISTA sequentially generates goal images for multi-object instructions while maintaining spatiotemporal consistency in object positions.The results suggest potential for longer-horizon tasks and flexible composition of subtasks.
  • Spatial Understanding: VISTA correctly interprets spatial descriptions for placement and grasping targets, including placing an object near another object outside a plate.The example uses the instruction to put the Sprite near the mango.
  • Semantic Understanding: VISTA identifies manipulated objects and placement targets from semantic cues involving plate shape, plate color, and tabletop pictures.These examples require semantic understanding beyond simple object references.
  • Capability Boundary: Instruction-following remains limited after robotic fine-tuning, with hallucinations still occurring in generated results.The authors suggest that more diverse robotic data could improve generation quality.
  • Goal-Conditioned Execution: GoalVLA uses the world model’s instruction understanding to focus on action generation and goal-image following during real-world execution.The integrated system enables flexible composition of manipulation skills in the reported examples.

APPENDIX G VISUALIZATION AND ANALYSIS OF EXECUTION RESULTS

GoalVLA generally aligns the robot’s posture with synthesized goal images during novel-scenario execution, but failures arise when goal timing or spatial alignment is inaccurate, or when target positions exceed the training distribution.

  • Successful execution: GoalVLA aligns the robot arm posture with corresponding goal images and generates action chunks that reach specified positions in visually distracting novel scenarios.The visualizations compare each subtask’s final frame with its goal image.
  • Failure cases: Inaccurate goal-image timing can cause grasp failures when images depict the robot before grasping or after the object has been lifted.The policy may overfocus on matching the depicted pose and fail to adapt to the actual execution state.
  • Failure cases: Spatially misaligned goal images can shift the gripper into the object and cause grasp failure during execution.The reported examples involve leftward offsets in images from the left wrist camera.
  • Failure cases: GoalVLA struggles with target positions significantly outside the training distribution even when the generated goal images are accurate.The authors identify greater spatial diversity in training data as a direction for improving generalization.
Loading 2602.10983v2…