Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation

1 Introduction

• Existing WAMs predict likely short-term outcomes but lack explicit task-level organization across subtasks and an interface for specifying task progression.The challenge is to add task-level decision making while preserving world-centric prediction and physical generalization.Source: §1 · p. 2
• Visual subgoals connect long-term future selection to short-term action realization, addressing the difficulty of underspecified spatial outcomes in language subtask descriptions.The framework targets compositional tasks involving coordinated subtasks and generalization to unseen scenes, objects, and layouts.Source: §1 · pp. 1–2
• ViGAR predicts a visual subgoal from the current observation and global instruction, then jointly predicts visual trajectories and actions conditioned on that subgoal.Both stages use a shared generative world-model representation to draw on common physical knowledge.Source: §1 · pp. 2–3
• With the scene and instruction fixed, changing the global goal image can elicit a new subtask decomposition and reorganized action sequence without parameter updates.This supports task compositions unseen during training at deployment.Source: §1 · pp. 2–3
• 82.00% Clean and 67.02% Random success on RoboTwin Clean2Random surpass the strongest baseline by 12.86 percentage points in average success rate.Real-robot experiments also report performance on five compositional tasks and in-context learning on unseen global goals.Source: §1 · p. 3

2 Related Works

• VLA systems map visual observations and language to action trajectories, while compositional systems add explicit subtask information to condition a policy [3] [5] [13] [14] [16] [25] [37] [8] [16] [17] [22].The subtask information may be synthesized by a VLM or a pretrained world model.Source: §2 · p. 3
• World-action models jointly predict observations and actions but typically lack compositional planning; BagelVLA [15] and RxBrain [20] do not use predictions as explicit downstream subgoal conditions.Their subtask predictions guide actions implicitly or primarily serve as planning outputs.Source: §2 · p. 3
• Prior robotic in-context methods use demonstration videos or trajectories to convey procedures [10] [39], whereas ViGAR uses a global goal image to specify the terminal state.This leaves the subtask decomposition to the model.Source: §2 · p. 3

3.1 VLA and WAM for Robotic Manipulation

• A VLA policy predicts an action chunk from observations, proprioceptive state, language, and embodiment metadata; a WAM additionally models a joint distribution over future visuals and actions.The passage identifies the action chunk and visual future as short-horizon predictions.Source: §3.1 · pp. 3–4

3.2 Problem Formulation for Compositional Robot Tasks

• A compositional task is decomposed into subtasks, each with a text instruction and visual target image; execution advances to the next pair when the current subtask is complete.The task is initially specified by a global language instruction.Source: §3.2 · p. 4

3.3 In-Context Task Specification for Robotic Manipulation

• In-context manipulation specifies a new task at deployment by changing the instruction or inference-time context without updating policy parameters.Prior work often uses a demonstration video as context to prescribe the procedure.Source: §3.3 · p. 4

4 Method

• ViGAR separates subgoal planning from a world-action policy that predicts actions conditioned on generated subgoals, with both components initialized from the same pretrained world model.The planner generates a subgoal image; the policy jointly predicts future frames and an action chunk.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Subgoal Planner for Task-level Reasoning

• The planner predicts image subgoals from an observation and the global instruction, retaining the instruction as a semantic anchor rather than generating subtask text.Look-ahead redirection handles visually similar subtask boundaries, while end-effector-centered ROI weights emphasize manipulation regions during training.Source: §4.1 · p. 5

4.2 Subgoal-guided World-Action Model

• Conditioned on the current observation, proprioception, instruction, and predicted subgoal, the policy jointly predicts a short visual future and action chunk.The subgoal is VAE-encoded and fed directly to the generator branch, grounding it at the pixel level rather than through a coarser semantic embedding.Source: §4.2 · p. 6

4.3 In-Context Task Specification via a Global Goal

• At inference, a global goal image conditions the planner alone, allowing an unseen goal to induce a task decomposition without changing either component's parameters.The downstream policy remains unchanged because the global goal's effect is mediated through the predicted subgoal.Source: §4.3 · p. 7

5.1 Training Configuration

• The subgoal planner and policy are initialized from the pretrained Cosmos3-Nano checkpoint and adapted for image editing and robot-policy generation, respectively.The policy follows a standard world-model-to-robot-policy post-training recipe.Source: §5.1 · p. 7

5.2 Results on Simulation Benchmark

• ViGAR’s visual subgoals achieve the best performance across all reported prediction metrics on RoboTwin and real-robot experiments.The metrics include LPIPS, DINO cosine similarity, Change of IoU, and retrieval correctness.Source: §5.2 · p. 7, §5.3 · p. 8
• 82.00% Clean and 67.02% Random success rates make ViGAR the top method under both RoboTwin settings, with most of the gain coming from Random.The controlled Cosmos3-Nano-RoboTwin baseline without subgoal guidance reaches 22.46% under Random.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 Compositional Manipulation in Real-world Robot Deployment

• ViGAR achieves the best overall task progress score across five real-robot compositional tasks, outperforming both baseline policies.The comparison with Cosmos3-Nano-Policy associates the improvement with subgoal-guided action generation beyond the pretrained backbone.Source: §5.3 · p. 8, Figure 4 · p. 9

5.4 Steering Real-world Robot Manipulation via In-Context Learning

• 77.5% in-domain and 47.5% OOD average task progress scores show that global goal images can steer the tested real-robot placements.The evaluation holds the initial scene and language instruction fixed while the global goal image specifies the desired composition.Source: §5.4 · p. 9

5.5 Ablation Studies

• Removing either end-effector ROI weighting or subgoal look-ahead degrades Clean and Random performance; ROI weighting focuses supervision on manipulation regions, while look-ahead avoids completed subtasks.End-effector ROI weighting contributes most to the performance gain.Source: §5.5 · p. 9
• Generator-only goal injection has the highest average success rate; adding the reasoner route improves Clean performance but reduces Random robustness.These results motivate the geometric route used in ViGAR.Source: §5.5 · p. 9

6 Conclusion

• ViGAR factorizes manipulation into visual subgoal planning and subgoal-conditioned joint video-action generation, linking task-level decisions with modeled physical dynamics.The paper presents experiments on RoboTwin and five real-robot tasks as support for this design.Source: §6 · p. 10
• A global goal image provides inference-time steering that composes learned manipulation behaviors into tasks absent from training demonstrations without parameter updates.The conclusion characterizes explicit visual goals as a promising direction for compositional and steerable robotic manipulation.Source: §6 · p. 10
• The work addresses the gap between short-horizon world-action prediction and task-level planning for compositional manipulation.The proposed connection uses intermediate visual targets to relate task-level decisions to physical action generation.Source: §6 · p. 10

A Dataset

• The simulation data uses 2,500 Clean RoboTwin expert rollouts, annotated by pairing observations with target-stage expert images, instructions, proprioception, and action trajectories.Task structure determines the subtask annotations.Source: A Dataset · p. 14
• Nineteen RoboTwin tasks are segmented into multiple subtasks, while 31 remaining tasks use the trajectory terminal frame as a single subgoal.For multi-subtask tasks, each subtask’s endpoint frame is its ground-truth subgoal image; preprocessing does not change the original rollouts.Source: A Dataset · p. 14
• The real-robot dataset contains approximately 900 hours of AgiBot A2 teleoperation across 752 tasks, more than 200 objects, and over 10 backgrounds.A human-in-the-loop pipeline manually segments trajectories into subtasks aligned with each task’s semantic structure.Source: A Dataset · p. 14

B Implementation Details

• The planner adapts Cosmos3 image-to-video generation for image editing, freezes the reasoner branch, and updates generator-side parameters during training.The current observation is encoded as a conditioning latent, while the target subgoal image is encoded as a target latent.Source: B Implementation Details · p. 14
• The Base planner uses standard flow matching throughout, while ROI switches to end-effector region-weighted training after 30k steps of standard training.Both variants train for 100k steps; ROI weights the projected end-effector regions at a 4:1 foreground-to-background ratio.Source: B Implementation Details · pp. 14–15
• The policy predicts chunks of 48 actions and 12 video frames per control query, training on a mixture of predicted and ground-truth subgoals.Semantically inconsistent or visually distorted predicted subgoals are replaced with ground-truth targets during training.Source: B Implementation Details · p. 15
• At deployment, the planner recomputes a subgoal from the current observation at each control query, and the policy predicts one action chunk without an external completion detector.This receding-horizon loop implicitly advances the subtask as the scene evolves.Source: B Implementation Details · p. 15
• Numerically lossless inference optimizations accelerate policy inference by 2.5× and subgoal image generation by 11.2× on a single NVIDIA B20Z GPU.The optimization stack includes compilation, fused projections, batched classifier-free guidance, and condition caching; TeaCache is also used for the planner.Source: B Implementation Details · p. 15

C.1 Visualization of Real-world Robot Task Settings

• Figure 6 displays task progress for Pile Paper Cups, Stack Colored Bowls, Stamp over Letters, Tidy-up Desktop, and Take-out Steam Buns.Source: Figure 6 · p. 16
• Figure 7 pairs global goal images with task-progress columns for Fruit Arrangement and Desktop Item Storage.Source: Figure 7 · p. 16

C.2 Visual Subgoal Prediction Quality

• The planner is evaluated independently of the action policy using the same reference observations and targets on RoboTwin and the real-robot benchmark.Source: §C.2 · p. 17
• Subgoal prediction is assessed through LPIPS perceptual distance, DINO-cos semantic alignment, Change-IoU spatial-change overlap, and RetAcc target-subtask retrieval.Source: §C.2 · p. 17
• The four metrics cover perceptual fidelity, semantic alignment, spatial-change consistency, and subtask-level correctness; results appear in Table 1b.Source: §C.2 · p. 17

• Real-robot evaluation reports task progress score and stage-wise success rate over 20 independent trials per task and method.Source: §C.3 · p. 17
• Stage k is evaluated only on trials requiring at least k stages, and s_i counts stages completed before the first failure.Source: §C.3 · p. 17
• When all trials share K, TPS averages SR_k across K stages; Tables 4 and 5 report task progress and stage-wise success for five long-horizon and two in-context tasks.Source: §C.3 · p. 17, Table 4 · p. 19, Table 5 · p. 19
• Figure 8 visualizes progress for Pile Paper Cups, Stack Colored Bowls, Stamp Over Letter, Tidy-up Desktop, and Take-out Steam Buns.Source: §C.3 · p. 17, Figure 8 · p. 18
• Without subgoal look-ahead, ViGAR tends to regress to completed subtasks; training with look-ahead alleviates this issue.Source: Figure 9 · p. 18