Agentic World Action Models

1 INTRODUCTION

• Predicted video and action chunks do not by themselves specify how to decompose long-horizon tasks into reusable skills, while agentic robotics often lacks predictive grounding in physical dynamics.The introduction also notes that predicting the world and actions can be expensive and that the degrees of freedom to model can be difficult to bound.Source: §1 · p. 2
• AW-0 combines code-based planning with world-action prediction, using a code expert to generate modular skills with explicit parameters and control logic.Code guides synchronized video and control generation within a mixture-of-transformers architecture.Source: §1 · p. 2, Figure 2 · p. 3
• Evaluations across RoboTwin 2.0 and physical bimanual setups show substantial performance gains over standard VLA and WAM baselines.The framework also introduces a hybrid execution policy that switches between programmatic skill invocation and direct action chunks.Source: §1 · p. 2
• AW-0’s contributions include an automated pipeline for program annotation and skill discovery, a structured skill-program dataset, and hybrid execution between skills and action chunks.The pipeline maps code segments onto robot and human video trajectories.Source: §1 · p. 2

2 AGENTIC WORLD ACTION MODEL

• AW-0 models discrete skill programs, future video latents, and continuous action chunks jointly, conditioned on current observations, language, robot state, and program memory.The program describes structured skill plans, video latents represent future scene dynamics, and action chunks cover a horizon of robot motion.Source: §2 · p. 3

2.1 MIXTURE OF TRANSFORMER ARCHITECTURE

• Three experts handle skill planning, future visual prediction, and continuous control, retaining expert-specific layers while communicating through masked joint attention.Attention is bidirectional within each expert and blockwise causal across experts in code, video, then action order.Source: §2.1 · pp. 3–4

2.2 JOINT DENOISING GENERATION

• Joint denoising lets code planning context inform video and action generation, while action prediction also reads video representations.Video and action use continuous flow-matching corruption, whereas code uses independently sampled absorbing masks on program tokens.Source: §2.2 · p. 4
• Predictive-guidance resampling broadens the video noise distribution with a uniform component, while objective-specific weights balance video learning and predictive guidance.Video and action use flow-matching losses; code uses masked-token cross-entropy.Source: §2.2 · p. 4

2.3 AGENTIC CODING IN WAM

• An annotation pipeline creates task-general skill libraries and cached program labels; decoded program slices condition action generation, with memory tracking prior slices and optional hybrid execution.The pipeline reuses cached programs when contact and motion match, otherwise generating new annotations; by default, decoded slices are plans rather than executed code, while hybrid mode arbitrates code execution and direct action chunks.Source: §2.3 · pp. 4–5, Figure 3 · p. 5

2.4 TRAINING RECIPE

• Annotated data support joint code, video, and action pretraining across embodiments and scenarios, followed by task-specific joint fine-tuning; Table 1 reports success rates for clean and random settings.Figure 4 presents average scores from Table 1.Source: §2.4 · p. 6, Table 1 · p. 6, Figure 4 · p. 6

3.1 SIMULATION EXPERIMENT

• 50.7% average success across RoboTwin 2.0 settings; AW-0 reaches 80.8% in the clean setting, 16.0 points above WLA.Under visual randomization, AW-0 scores 20.6%, below π0.5 (27.7%) but above DreamZero (17.6%), EgoScale (2.9%), and WLA (3.6%).Source: §3.1 · pp. 6–7
• Hybrid execution raises random-setting success from 20.6% to 29.2%, but reaches 63.1% in the clean setting, 17.7 points below default AW-0.Hybrid execution retains 46% of its clean-setting success under randomization, versus 25% for the default mode.Source: §3.1 · p. 7

3.2 REAL-WORLD EXPERIMENT

• AW-0 achieves the highest average rubric score across both real-world embodiments, outperforming the strongest baseline by 25.1 points.The evaluation covers eight tasks, four per platform, with 3 × 16 rollouts per task.Source: §3.2 · p. 7, Table 2 · p. 7
• On Fold and Stack Towels, real-world hybrid execution scores 90.6%, compared with 84.4% for action-chunk execution.The comparison uses the same post-trained model and evaluation conditions.Source: §3.2 · p. 7

3.3 ABLATION STUDY

• C →V →A outperforms V →C →A because both video and action generation attend to code, letting semantic plans shape prediction throughout denoising.The compared variants use the same pretraining data, training budget, and optimization settings as AW-0.Source: §3.3 · p. 8, Figure 5 · p. 8
• Dedicated ViT conditioning improves performance on all eight tasks, including Pack Backpack (64.6% vs. 39.6%) and Solve Rubik’s Cube (72.1% vs. 53.9%).The comparison is against reused Wan2.2 VAE features for conditioning the code expert.Source: §3.3 · p. 8, Table 3 · p. 8
• Full annotated programs outperform comment-free code and language-only subgoals on the shared tasks, while removing skill calls reduces performance on all eight evaluated tasks.On Deal Poker, removing skill calls lowers the score from 58.3% to 42.0%.Source: §3.3 · p. 9, Figure 7 · p. 9
• Joint denoising outperforms AR on seven of eight tasks and takes 0.23 s versus 0.55 s, a 2.39× speedup.Against AW-0 without code, it adds 0.04 s (21.1%) while improving performance on all eight shared tasks.Source: Figure 6 · p. 8, Figure 5 · p. 8, §3.3 · p. 9

4 RELATED WORK

• WAMs predict dynamics alongside actions, while agentic systems structure reusable skills in programs; AW-0 places code inside the model as a planning interface.The generated code can also be executed at test time to complement the model’s own actions.Source: §4 · pp. 9–10

5 CONCLUSION

• AW-0 integrates code-based skill planning with predictive dynamics and continuous control, with experiments showing gains in language following, semantic reasoning, skill execution, and hybrid robustness.The paper presents this as a foundation for work on skill reuse and interaction between programmatic planning, world prediction, and execution.Source: §5 · p. 10

AI USE STATEMENT

• Gemini 3.1 Pro and Claude Opus 4.8 generated demonstration-aligned program labels and reusable skill libraries; Claude Opus 4.8 also generated simulation programs.GPT-6 Astra was used for language polishing and stylistic refinement, as well as part of figure preparation.Source: AI USE STATEMENT · p. 10

• Pretraining datasets follow their respective licenses and data-use terms, while the team collected all post-training demonstrations for the evaluated tasks.Source: ETHICS STATEMENT · p. 10

• Appendix A covers architecture, parameter settings, and computational resources, while Appendix B details pretraining-dataset composition.Source: REPRODUCIBILITY STATEMENT · p. 10
• Section 2.3 presents the annotation pipeline, with prompts and pretraining-model performance analysis in Appendices K.2 and F.1.Source: REPRODUCIBILITY STATEMENT · p. 10
• Standardized downstream data-generation criteria and methods are provided to facilitate extension to further tasks.Source: REPRODUCIBILITY STATEMENT · p. 10

A MODEL AND TRAINING DETAILS

• AW-0 uses three 28-layer modality-specific experts with masked joint attention: code conditions on visual observations and language, video attends to code, and action attends to both.The experts project queries, keys, and values into a shared 2,048-dimensional attention space.Source: A MODEL AND TRAINING DETAILS · p. 17
• Robot and human demonstrations share a 64-dimensional state–action representation, with absolute states, chunk-relative end-effector pose deltas, embodiment-specific targets, and percentile normalization.Unused embodiment dimensions are zeroed and left unnormalized.Source: A MODEL AND TRAINING DETAILS · p. 17
• Pretraining mixes pretrained-density and uniform video-noise sampling, uses uniform-reference action and code losses, and masks instruction and program inputs in 15% of batches; downstream runs fine-tune for 20 epochs on 100 demonstrations per task.The code expert freezes initialized SDAR components while training MoT attention projections and modulation parameters; video/action initialization and other trainable components are also specified.Source: A MODEL AND TRAINING DETAILS · pp. 17–18

B.1 PRETRAINING DATASET STATISTICS

• The pretraining union contains 248,642 episodes and 2,913.2 hours: 73,101 YAM episodes across 279 tasks and 175,541 egocentric-human episodes across 327 tasks.YAM data totals 1,919.3 hours and human data 993.9 hours; windows pair episode instructions with aligned program slices.Source: §B.1 · pp. 18–19

B.2 SKILL LIBRARY DEFINITIONS

• The annotation pipeline produces reusable Python-function skill libraries with 23 YAM skills and 27 egocentric-human skills, turning recurring subtasks into named calls over primitive APIs.Examples include folding an edge, opening a closure, pouring into a target, and stirring contents.Source: §B.2 · p. 19

B.3 VISUALIZATIONS OF YAM SKILLS

• YAM visualizations pair representative key frames and task/subgoal annotations with program slices and skill functions, including a blanket-folding example that invokes fold_edge_over.The library call pinches an edge with both arms, carries it across the item, and releases it.Source: §B.3 · p. 21

Skill IV: adjust in place

• The adjust-in-place example maps the subgoal of adjusting a bookmark in a book to a skill that grasps it without lifting, nudges it, and releases it.Its program slice calls adjust_in_place on the bookmark with a 0.05-unit x-direction delta.Source: Skill IV: adjust in place · p. 23

B.4 VISUALIZATIONS OF EGOCENTRIC HUMAN SKILLS

• Egocentric-human visualizations likewise pair key frames, task/subgoal annotations, program slices, and skill functions; examples include bracing a photo album while opening its cover and moving a spoon over a bowl to pour.The skill library also includes stirring contents, described as sweeping an already-held tool inside a vessel.Source: §B.4 · p. 23, Skill I: open closure · p. 23, Skill II: pour into target · p. 24

Skill III: stir contents

• The stir-contents skill braces and takes a tool, places it inside a vessel, and rotates it twice without releasing its grip.The episode program invokes the reusable routine with a stirring rod and beaker.Source: Skill III: stir contents · p. 24

Skill IV: wipe surface back and forth

• The wipe-and-fold examples encode repeated back-and-forth surface strokes and a braced fold that carries an edge across an item, releases it, and presses a crease.The examples apply wiping to pants with a lint roller and folding to long pants.Source: Skill IV: wipe surface back and forth · p. 25

Skill VI: wrap around target

• The wrap-around skill traces a circuit over, down, underneath, and back up a target; the curling-hair example repeats the routine twice.The episode program applies it to a curling iron.Source: Skill VI: wrap around target · p. 26

B.5 SKILL DISTRIBUTION IN THE EMBEDDING SPACE

• Slices assigned to different skills occupy largely separate embedding-space regions in both the YAM and egocentric human libraries, suggesting skill labels can condition action generation.Figures 18 and 19 show the 18 most common YAM skills and 16 most common egocentric human skills, respectively.Source: §B.5 · p. 26, Figure 18 · p. 26, Figure 19 · p. 27

C BASELINE IMPLEMENTATION DETAILS

• Baseline comparisons reproduce π0.5, DreamZero, EgoScale, and WLA using shared evaluation protocols, with task-specific policy training and model-specific action or video configurations.RoboTwin policies use 2,500 clean demonstrations across 50 tasks; separate policies are trained for each of eight real-world tasks, with three RGB views, robot state, and language instructions.Source: C BASELINE IMPLEMENTATION DETAILS · pp. 27–28

D REAL-WORLD EXPERIMENTAL SETUP

• The real-world setup uses YAM parallel-gripper arms and Tianji–Sharpa dexterous hands, with three RGB views, robot state, task instructions, and teleoperated demonstrations.YAM demonstrations use paired master and puppet arms; Tianji–Sharpa demonstrations use Manus gloves, PICO trackers, and Apple Vision Pro, with recorded observations, states, action targets, and task instructions.Source: D REAL-WORLD EXPERIMENTAL SETUP · pp. 28–29

E REAL-WORLD TASK PROGRESS

• Eight real-world tasks span the YAM and Tianji–Sharpa platforms, with each task graded by additive rubrics that award partial credit for completed sub-steps.Task progress is illustrated with key frames from one training episode aligned to color-coded blocks of its code label.Source: E REAL-WORLD TASK PROGRESS · p. 29

E.1 YAM TASKS

• YAM fruit arranging requires moving instructed fruit between a bowl and basket and matching the requested final state without drops.Source: §E.1 · p. 30
• YAM backpack packing scores zipper movement, opening the bag, placing an item inside, and closing the zipper.Source: §E.1 · p. 31
• YAM bulb replacement distinguishes a dead-bulb replacement condition from a working-bulb condition, with additive rubric steps for each.Source: §E.1 · p. 32
• YAM towel folding scores folding three towels, stacking them, and matching the instruction’s color order.Source: §E.1 · p. 33

E.2 TIANJI–SHARPA TASKS

• Tianji–Sharpa fruit arranging uses tongs to move named and unnamed fruit between a bowl and tray, then return the tongs.Source: §E.2 · p. 34
• Tianji–Sharpa poker dealing scores transferring a card stack, producing one protruding card, and inserting that card into a slot.Source: §E.2 · p. 35
• Tianji–Sharpa scoreboard flipping scores holding the board, flipping a digit card the requested number of times, and pressing the button at the target score.Source: Figure 27 · p. 35, §E.2 · p. 36
• Tianji–Sharpa cube solving randomizes the scrambled face and placement angle, then scores a quarter-turn, solved state, and release on the table.Source: §E.2 · p. 36, Figure 28 · p. 36

F.1 PRE-TRAINED CODE GENERATION QUALITY

• 89.7% teacher-forced token accuracy and an 87.5% free-generation parse rate coexist with entity F1 of 0.534 and primitive F1 of 0.365.Shuffling visual observations increases code loss by 1.774; 29.2% of programs contain skill-style helper calls.Source: §F.1 · p. 37, Table 10 · p. 37

F.2 POST-TRAINED CODE GENERATION QUALITY

• Online deployment examples show matching skill selection and arguments for zipper grasping and cube holding, although the Rubik’s Cube output contains a comment-formatting error that prevents direct execution.The backpack example also matches the reference arm assignment and numerical arguments.Source: §F.2 · pp. 38–39, Figure 31 · p. 39, Figure 32 · p. 39

G.1 IMPACT OF CODE EXPERT ON LANGUAGE FOLLOWING CAPABILITY

• For fruit arrangement, AW-0 predicts interaction with the mango, while Subgoal ambiguously targets apple or mango and AW-0 w/o code targets the apple.The compared rollouts use similar but non-identical initial observations.Source: §G.1 · p. 40

G.2 IMPACT OF CODE EXPERT ON SKILL ROBUSTNESS

• For card insertion, AW-0 predicts more accurate alignment and entry into the slot, while Subgoal and AW-0 w/o code produce less coherent sequences with blur and distortion.The passages suggest explicit skill calls and object arguments help predict the hand-card-receptacle motion.Source: §G.2 · p. 40

G.3 HYBRID ACTIONS EVALUATION DETAILS

• The qualitative example follows one towel-folding episode from the upper row into the lower row.Source: §G.3 · p. 40, Figure 35 · p. 41
• In the towel-folding episode, direct actions handle dexterous folding motions, while code execution handles structured motions such as approaching and stacking.The caption marks code execution in green and direct actions in purple across a single episode.Source: §G.3 · p. 41, Figure 35 · p. 41
• Code-guided stacking supplies guidance on stacking order and motion direction, which the passage links to improved semantic reasoning for the system.Source: §G.3 · p. 41

H FAILURE CASE ANALYSIS

• Representative failures include a near-rim fruit grasp catching the basket wall, a backpack tipping during zipper pulling, and a bulb remaining misaligned with its socket.The passages identify surrounding geometry, stabilization during deformation, and positional plus angular alignment as relevant challenges.Source: H FAILURE CASE ANALYSIS · pp. 41–42
• Other failures include tongs poking rather than grasping a pear, cards slipping during handover, and a fingertip losing contact before a score card flips.The examples point to tool opening and approach coordination, stable receiving grasps, and sustained contact during the flip.Source: H FAILURE CASE ANALYSIS · pp. 42–43

I HYBRID ACTIONS ALGORITHM

• Code slices execute only after stage-order and retry, dependency, syntax, whitelist, and nonempty-statement gates; an attempted slice displaces the same-pass action chunk even if it raises an exception.When code is not attempted, the action chunk runs for the replan horizon.Source: I HYBRID ACTIONS ALGORITHM · p. 43, §J.1 · p. 44

J.1 ROBOTWIN PER-TASK BREAKDOWN

• Table 11 breaks down RoboTwin 2.0 success rates across 50 tasks in clean and random settings, with 50 rollouts per task per setting.Source: §J.1 · p. 43, Table 11 · p. 45

J.2 HYBRID ACTIONS EVALUATION DETAILS

• The default-versus-hybrid comparison does not isolate hybrid execution because the configurations use different demonstration-collection pipelines and execution modes.Hybrid data use CaP-based collection to pair demonstrations with executable programs; the default uses RoboTwin’s native motion planner.Source: §J.2 · p. 46
• 20.6% direct-action success rises to 29.2% with hybrid control in the random setting, while clean-setting success falls from 80.8% with direct action to 63.1% with code admitted.On successful episodes, code drives 14.5% of steps in clean and 35.0% in random settings.Source: §J.2 · p. 46
• Blocks ranking uses code for most pick-and-place cycles, whereas direct actions recover a failed bottle grasp before code resumes.The examples show code working smoothly on regular objects and motions, but failing to secure the less regular bottle grasp.Source: §J.2 · p. 46

K.1 LOW-LEVEL PRIMITIVES

• The parallel-gripper and dexterous-hand APIs share ten semantic contact verbs but use different low-level motion primitives.Gripper calls use end-effector poses; dexterous-hand calls use relative motion, and API calls declare their contact events.Source: §K.1 · p. 46

K.2 ANNOTATION AGENT PROMPTS AND COORDINATOR PROTOCOL

• The coordinator audits annotation traces, while skill admission requires generic, recurrent, appropriately abstracted, nonduplicate API functions; YAM candidates must also reconcile with observed contact events.A skill candidate is distilled from observed call sequences and becomes available to the annotator in the next round.Source: §K.2 · pp. 49–50

L LIMITATIONS AND FUTURE WORK

• Planning quality depends on annotation accuracy and skill coverage, and syntactic validity alone does not ensure that generated programs execute correctly.Programs may select an appropriate skill but supply incorrect arguments.Source: L LIMITATIONS AND FUTURE WORK · p. 50
• Hybrid execution improves robustness under RoboTwin randomization but reduces clean-setting performance, with outcomes also constrained by primitive reliability and validity-only arbitration.The real-world hybrid evaluation is limited to the YAM platform, and the system does not learn online from its interactions.Source: L LIMITATIONS AND FUTURE WORK · p. 50