PaperlayerView exact revision on arXiv

Source-linked AI summary

EVO-WAM: Evolving World Action Models through Video-Action Verification

Shiyang Zhou, Xionghao Wu, Wenbo Li, Shenghe Zheng, Jiyao Zhang, Songsong Yu, Yijun Yang, Jianhui Liu, Haoze Sun, Senqiao Yang, Li Jiang, Jingyong Su, Haoyang Huang, Zhuotao Tian

arXiv:2609.38057v1cs.CVcs.RO

TL;DR

Robot policies are difficult to adapt to unseen tasks without costly new demonstrations, and WAM-generated trajectories may be incomplete or pair successful video with inconsistent actions. EVO-WAM trains WAMs on their own generated trajectories after checking completion and action consistency, improving results across unseen simulation tasks and real-world composite tasks.

  • Problem

    Adapting robot policies to unseen tasks without additional expert demonstrations remains challenging, while WAM-generated videos may fail to show completion or pair successful-looking behavior with inconsistent actions.

    Source: §1 · p. 2

  • Method

    EVO-WAM enables autoregressive rollouts and iteratively trains WAMs on generated prefixes selected for task completion by a VLM and checked for video-action consistency by an IDM.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On seven unseen RoboTwin 2.0 tasks, average success rises from 26.9% to 68.0% for Cosmos3 and from 28.5% to 46.4% for DreamZero; on three unseen real-world composite tasks, Cosmos3 rises from 20.0% to 76.7%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations show that WAMs can improve on unseen tasks by learning from their own verified predictions without external action execution during self-training.

    Source: §5 · p. 10

    DreamZero’s gains were limited on emptycup placement and block stacking, possibly reflecting constraints on useful behaviors in its generated candidates.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Improving robot policies on new tasks without collecting additional expert demonstrations remains a central challenge in robot learning. World action models (WAMs) use broad video priors to jointly predict future videos and actions, offering a potential source of supervision for adapting to new tasks. However, generated videos may fail to depict task completion, and even visually successful videos may be paired with inconsistent actions that lead to execution failure. We propose EVO-WAM, a framework that adapts WAMs to unseen tasks by learning from their own generated video-action trajectories, without executing candidate actions in an external environment. First, we augment WAM training with state prediction and anchored multi-frame context to enable complete autoregressive rollouts without external execution feedback. Second, we identify reliable training experience by selecting task-completing prefixes with a vision-language model and verifying their video-action consistency with an inverse dynamics model. Third, we iteratively train the WAM on verified prefixes and generate new rollouts with the updated model. On seven unseen RoboTwin 2.0 tasks, EVO-WAM increases average success rates from 26.9% to 68.0% for Cosmos3 and from 28.5% to 46.4% for DreamZero, reaching approximately $2.5\times$ and $1.6\times$ their initial success rates. On three unseen long-horizon composite tasks in the real world, it improves Cosmos3's average success rate from 20.0% to 76.7%, a gain of 56.7 percentage points. Project Page: https://evo-wam.github.io/.

1 Introduction

EVO-WAM addresses the challenge of adapting robot policies to unseen tasks without new expert demonstrations by training WAMs on their own verified generated trajectories. It evaluates this approach across two WAM backbones, simulated tasks, and real-world composite tasks.

1 Introduction

  • Unseen-task adaptation is challenging because demonstrations are costly to extend, generated videos may not depict completion, and paired actions may not match visually successful behavior.Earlier approaches using separate world models or WAM replay still rely on external experience or task demonstrations for learning new tasks.Source: §1 · p. 2
  • EVO-WAM enables WAMs to autoregressively generate video-action rollouts and improve on unseen tasks through verification and iterative self-training, without new demonstrations or external action execution.State prediction and anchored multi-frame context support continuation without external execution feedback.Source: §1 · p. 2
  • On seven unseen RoboTwin 2.0 tasks, verified self-training raises average success from 26.9% to 68.0% for Cosmos3 and from 28.5% to 46.4% for DreamZero.On three unseen long-horizon real-world composite tasks, Cosmos3 improves average success from 20.0% to 76.7%.Source: §1 · p. 2
  • A VLM selects task-completing prefixes, while an IDM checks video-action consistency before generated trajectories are used for iterative self-training.The authors report gains across Cosmos3 and DreamZero on unseen RoboTwin tasks, Cosmos3 on real-world composite tasks, and different VLM sizes.Source: §1 · pp. 2–3

2 Preliminaries

WAMs jointly predict future video and robot actions, so their generated trajectories can provide training experience. But rollouts may be incomplete or pair successful-looking video with actions that cannot realize it.

2.1 World Action Models

  • WAMs predict future video and paired robot actions from visual observations, robot states, and task instructions; the video depicts anticipated behavior and the actions are intended to realize it.Unlike conventional VLA policies, this joint prediction can form trajectories without obtaining future observations from environment interaction or a separate world model.Source: §2.1 · p. 3

2.2 Self-Training with Generated Rollouts

  • Generated rollouts are not reliable supervision by default: videos may show failure, and actions paired with successful-looking videos may be inconsistent with the depicted behavior.These problems motivate checking both visual task completion and video-action consistency before self-training.Source: §2.2 · p. 3

3 Method

EVO-WAM repeatedly generates candidate trajectories, verifies task completion and video-action consistency, and trains the WAM on accepted prefixes. State prediction and anchored visual context let rollouts continue without external execution feedback.

3 Method

  • EVO-WAM repeats generation, verification, and training from a base WAM, using verified data to update the model for unseen-task performance.The setup uses an initial observation, robot state, and task instruction without additional expert demonstrations or external action execution.Source: §3 · p. 3

3.1 Enabling Autoregressive Rollouts

  • Predicting each chunk’s end state and retaining an initial-frame anchor alongside recent generated frames supplies state and visual context for autoregressive continuation.The anchor references objects and scene, while recent frames provide motion history; each subsequent chunk uses the predicted state and recent video context.Source: §3.1 · pp. 3–4

3.2 Video-Action Verification

  • The VLM first identifies task-completing prefixes, then the IDM reconstructs actions from generated video to assess consistency with the WAM-paired actions.Accepted prefixes become candidates for self-training after the video-action check.Source: §3.2 · p. 4, Figure 3 · p. 5

Task-completion verification.

  • Subgoals are checked chronologically, with endpoints fixed at the first VLM acceptance; prefixes are discarded if an endpoint is missing or consistency or endpoint-vote checks fail.Each endpoint needs at least two Accept judgments out of three, and verification does not resume at a later endpoint after a failure.Source: Task-completion verification · p. 5

3.3 Iterative Self-Training

  • Each round combines newly verified prefixes with retained earlier data and original training data, then fine-tunes the WAM to generate candidates for the next round.Retaining earlier data is intended to preserve trajectory diversity and limit shifts in the training distribution between updates.Source: §3.3 · p. 6

4 Experiments

Experiments show gains across two WAM backbones on unseen simulation tasks and across three real-world composite tasks, with additional tests of self-training rounds, verification, generalization, and retention.

4.2 Simulation Experiments

  • Across 43 base-model training tasks, the remaining seven RoboTwin tasks are reserved for self-training and evaluation, with four rounds of candidate generation and accumulated verified prefixes.Each round generates 2,800 candidates and uses 1K training updates; evaluation uses 100 Clean and 100 Randomized trials per task.Source: §4.2 · p. 6

4.2 Simulation Experiments; 4.3 Real-World Experiments

  • On seven unseen RoboTwin tasks, EVO-WAM reaches 68.0% (Cosmos3) and 46.4% (DreamZero), versus 31.6% and 27.2%; on three real tasks, Cosmos3 reaches 76.7% versus 20.0%.The simulation averages compare each EVO-WAM backbone with its respective baseline; the real-world comparison is against Cosmos3.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8

4.3 Real-World Experiments

  • The real-world tasks test bowl stacking, duck selection among distractors, and opening an air-fryer drawer before placing bread inside.In the duck example, one candidate fails visual-goal verification and another fails video-action consistency; the policy completes both placements after training on prefixes passing both checks.Source: §4.3 · pp. 7–8, Figure 6 · p. 8

4.4 Improvement over Multiple Rounds

  • The largest simulation gains occur early: Cosmos3 rises from 26.9% to 58.3% in Round 1, then temporarily declines in Round 3 before recovering to 68.0% in Round 4.The real-robot model reaches 76.7% in Rounds 2 and 4, with 73.3% in Round 3; later rounds bring smaller gains and occasional regressions.Source: §4.4 · p. 9, Table 3 · p. 9

4.5 Ablation Studies

  • VLM + IDM reaches 68.0% versus 43.7% for VLM-only verification in Round 4, while simulator verification reaches 72.7% but requires a target-task simulator.IDM-based verification also supports real-world tasks without such simulators.Source: §4.5 · p. 9, Table 4 · p. 9
  • With Qwen3.5-27B, Round 4 reaches 65.7% versus 68.0% with Qwen3.8-Flash-Next; on new scenes, EVO-WAM achieves 70.4% versus 24.9% for the baseline.The new-scene evaluation uses no further adaptation.Source: §4.5 · pp. 9–10
  • On 43 seen tasks, Round 4 EVO-WAMCosmos3 reaches 84.8% success versus 85.8% for the 34K Cosmos3 baseline.The paper characterizes this as a small seen-task decrease alongside substantial unseen-task improvement.Source: §4.5 · p. 10

5 Conclusion

EVO-WAM adapts WAMs to tasks unseen during base-model training by turning their own verified video-action predictions into training experience. The paper positions this as a distinct route to self-improvement without external action execution during adaptation.

5 Conclusion

  • EVO-WAM improves WAMs on tasks unseen during base-model training by learning from their own generated experience, without additional demonstrations or external action execution during self-training.The framework enables autoregressive rollouts and verifies both task completion and video-action consistency before using prefixes for training.Source: §5 · p. 10

A Related Work

  • For generated embodied trajectories, verification must check both task completion and consistency between visual outcomes and actions.The discussion contrasts this need with verification approaches for reasoning models and executable tasks.Source: A Related Work · p. 14
  • Unlike DreamerV3 and other world-model approaches described here, EVO-WAM uses a WAM’s joint video-action predictions as supervision for iterative policy updates without external execution during adaptation.The related-work discussion also distinguishes approaches that use deployment experience, reward or value feedback, or rollout experience for model-based planning.Source: A Related Work · p. 14
  • Unlike methods that distill an external video teacher or use WAM trajectories for previously learned tasks, EVO-WAM generates and verifies trajectories for tasks unseen during base-model training.Its verification selects visually successful, video-action-consistent prefixes from the WAM’s own rollouts without executing candidate actions externally.Source: A Related Work · p. 14

B.1 Data Accumulation

The section reports success across self-improvement rounds under two data-use configurations, including a latest-round and an accumulated-data series.

B.1 Data Accumulation

  • Both series use inverse dynamics models trained on the 43 seen RoboTwin tasks.Source: §B.1 · p. 14, Table 6 · p. 14
  • Figure 7 tracks success in adaptation scenes across self-improvement rounds for the two configurations described in the paper.Table 6 reports their latest-round and accumulated-data series through Round 4; the accumulated-data series reproduces the EVO-WAMCosmos3 sequence in Table 3.Source: §B.1 · p. 14, Table 6 · p. 14, Figure 7 · p. 15

B.2 Per-Task Evaluation Across Scene Sets

The section compares EVO-WAM’s performance in new scenes with a baseline and reports per-task adaptation-versus-new-scene results.

B.2 Per-Task Evaluation Across Scene Sets

  • 70.4% versus 24.9% success rates compare EVO-WAMCosmos3 with the 34K Cosmos3 baseline on the same new-scene test set.The comparison uses identical inference settings and success criteria.Source: §B.2 · p. 15
  • 82.0% versus 72.5% for placing bread in a basket and 47.0% versus 35.5% for stacking three blocks compare new scenes with adaptation scenes.These are Round 4 results for EVO-WAMCosmos3.Source: §B.2 · p. 15, Table 7 · p. 15
  • Table 7 reports final-model success rates across adaptation and new scenes, with 100 Clean and 100 Randomized trials per task in each scene set.The table covers seven tasks and 1,400 total trials; its average pools both conditions.Source: Table 7 · p. 15

B.3 Real-Robot Tasks and Scene Configurations

The real-robot evaluation covers three tasks across varied initial layouts, using ten trials per task and task-specific completion criteria.

B.3 Real-Robot Tasks and Scene Configurations

  • Each policy is evaluated on ten trials per task: three layouts for Stack Bowls and Place Ducks, and two layouts for Load the Air Fryer.The three-layout tasks use 3/3/4 trials across layouts; the air-fryer layouts use 5/5.Source: §B.3 · p. 15, Figure 8 · p. 16
  • Task success is the number of successful trials divided by ten, with object arrangements varying across layouts while task instructions and targets remain fixed.Figure 8 shows the initial configurations.Source: §B.3 · p. 16, Figure 8 · p. 16
  • Stack Bowls requires pink on blue, then both bowls on white, with the gripper releasing the completed stack.The three layouts permute the bowls’ starting positions.Source: §B.3 · p. 16
  • Place Ducks requires releasing the pink duck into the pink bowl and the blue duck into the blue bowl, despite a third distractor duck.The layouts vary duck positions while retaining the target bowls.Source: §B.3 · p. 16
  • Load the Air Fryer requires opening the drawer, transferring the bread from its plate, and releasing it inside.The two layouts place the bread on opposite sides of the air fryer.Source: §B.3 · p. 16

B.4 Additional Real-Robot Case Studies

The additional real-robot case studies illustrate how verification distinguishes task-completing rollouts from visually plausible but unsuitable trajectories, alongside the IDM design used for consistency checks.

B.4 Additional Real-Robot Case Studies

  • The case-study figures show Before, three imagined trajectories, and After executions; the imagined trajectories use four chronological wrist-camera frames.Before and After scenes use independently reset scenes with matching relative object layouts.Source: §B.4 · pp. 16–17
  • Rollout 3 passes verification for both the bowl-stacking and air-fryer examples, while Rollout 2 is rejected by the IDM in each case.The bowl rollout’s score is 0.008170 against a 0.008111 threshold; the air-fryer rollout’s score is 0.009082 against that threshold.Source: §B.4 · p. 17
  • In bowl stacking, Rollout 1 reverses the required order, while Rollout 2 follows it but fails the action-consistency check.Rollout 3 supplies a stacking example for subsequent self-training rounds.Source: §B.4 · p. 17
  • In the air-fryer case, Rollout 1 distorts the drawer and bread, whereas the verified sequence opens the drawer before transferring the bread.The After execution shows the drawer opening, bread transfer, and release.Source: §B.4 · pp. 17, 19, Figure 10 · p. 18

B.5 IDM Training Data and Model Design

  • The IDM reconstructs recorded actions from observed motion, the window-start state, and the instruction, then uses reconstruction error to check paired WAM actions against generated video.Its training pairs include successful and unsuccessful executions because both contain actions that produced their observed videos.Source: §B.5 · p. 19

B.6 Data Production and Training Configuration

The appendix specifies baseline and self-training schedules, training-data mixtures, and the verification-ablation and data-retention procedures.

B.6 Data Production and Training Configuration

  • RoboTwin baseline models train for 34K steps on 43 seen tasks, while self-training starts from 30K-step checkpoints and runs four rounds of 1,000 updates.The final RoboTwin checkpoints reach 34K steps.Source: §B.6 · p. 19
  • Real-robot self-training uses four rounds of 500 updates, and its samples comprise 50% generated prefixes, 45% recorded successful executions, and 5% recorded unsuccessful executions.Action projection layers use a learning rate of 5 × 10−5.Source: §B.6 · p. 19
  • For RoboTwin verification ablations, variants share a four-round update budget and evaluation setup while changing the action verifier or task-completion model.The action-verification comparison uses VLM only, VLM + IDM, or VLM + Simulator; the VLM comparison retains the same IDM checkpoint and threshold.Source: §B.6 · p. 19
  • RoboTwin schedules 2,800 candidates per round from 1,400 scenes, and accepted prefixes accumulate across rounds under fixed backbone-specific consistency thresholds.In Round 2, EVO-WAMCosmos3 had 2,799 candidates reach the recorded selection stage; its threshold is 0.00418487 and DreamZero’s is 0.00843549.Source: §B.6 · p. 20
  • Each retained trajectory contributes one prefix to the training pool, whose cumulative size is the sum of prefixes retained through that round.Table 10 also reports new prefixes and optimizer updates per round.Source: Table 10 · p. 20
  • Real-robot rounds retain 33, 159, 179, and 76 new prefixes, producing a cumulative pool of 447 after four rounds.The first round excludes one corrupted video from 34 automated acceptances; the real-robot consistency threshold is 0.00811050.Source: §B.6 · p. 20

B.7 Analysis of Generated Rollouts

The analysis illustrates why visually successful generated rollouts need action-consistency checks and describes how the verification pipeline selects training prefixes. Adding the IDM reduces false acceptances, while its score is calibrated on recorded data and fixed across self-training rounds.

B.7 Analysis of Generated Rollouts

  • At step 120, the generated stapler placement fails simulator replay; its accepted prefix has IDM error 0.00632283, above the 0.00418487 threshold, and is rejected.The stapler remains on the table through the 194-action replay, illustrating why visual completion alone is insufficient.Source: §B.7 · p. 20, Figure 11 · p. 21

B.8 Verifier Reliability

  • 63 to 15 false acceptances with IDM, at the cost of lower recall; among VLM-accepted candidates, replay succeeded for 92/107 (86.0%) passing IDM versus 42/90 (46.7%) rejected by IDM.These labels reflect complete action-tape replay; simulator failures may also arise from physics artifacts and do not by themselves identify video–action inconsistency.Source: Table 11 · p. 21, §B.8 · p. 21

C.1 Task-Completion Verification

  • Visual verification scans subgoals in order, fixes each first accepted endpoint, and retains a complete prefix only after IDM and endpoint-vote checks pass.Each endpoint receives two additional VLM assessments; unresolved candidates are withheld from subsequent policy training.Source: §C.1 · p. 22

C.2 Inverse Dynamics Model

  • The IDM reconstructs actions from video windows, starting state, and instruction; its consistency threshold is calibrated on recorded data and remains fixed across self-training rounds.Training includes successful and unsuccessful recorded executions, since both provide video-action correspondences.Source: §C.2 · pp. 22, 25

C.3 Autoregressive Rollout Training

Autoregressive continuation combines the initial visual anchor and recent generated context with state inputs derived from model outputs. The qualitative comparison shows improved object persistence after occlusion when both context components are used.

C.3 Autoregressive Rollout Training

  • During autoregressive continuation, the model reuses its generated video latents and derives the next state input from its outputs, alongside the initial visual anchor and recent context.Cosmos3 predicts 64 video frames and 64 actions per chunk; DreamZero predicts 24 frames and 72 actions per chunk.Source: §C.3 · p. 25, Table 13 · p. 25

C.4 Qualitative Comparison of Visual Context

  • With the initial anchor and recent multi-frame context, the yellow block remains visible after the arm occludes it; without them, it disappears.The comparison uses matched timestamps in the two autoregressive rollouts.Source: §C.4 · p. 26, Figure 13 · p. 26

Related papers

Loading 2609.38057v1…