PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does World Modeling Actually Improve in Robot Action Learning?

Hanwen Cui, Yuanyuan Li, Qing Guo, Zhida Zhang, Chao Chen, Xi Chen, Jie Cao

arXiv:2610.34593v2cs.RO

TL;DR

Aggregate task success alone leaves unclear how predictive supervision changes execution behavior or whether benefits persist under task and environmental shifts. The paper conducts controlled loss ablations in two WAM families and adds auxiliary prediction to π0.5. Seen-task gains are substantial, but transfer is inconsistent; a lightweight predictor also improves π0.5 in the evaluated setting.

  • Problem

    Task-success improvements alone do not reveal how predictive supervision manifests in execution behavior or whether its benefits transfer across tasks and environments.

    Source: §1 · p. 1

  • Method

    The study controls future-video prediction loss within FastWAM and OpenWAM, analyzes trajectory clusters and distribution shifts, and tests auxiliary prediction for π0.5.

    Source: §1 · pp. 1–2

  • Results

    FastWAM and OpenWAM gain +24.38 and +72.91 percentage points under Seen+Clean, but gains do not reliably persist on unseen tasks or randomized environments.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Future-video prediction can provide effective action-learning supervision, and the evaluated π0.5 gains do not necessarily require a large pretrained video-generation branch.

    Source: §7 · p. 10

    The lightweight predictor still uses frozen Wan2.2 VAE latents, and branch differences prevent attributing performance differences to model size alone.

    Source: §6 · p. 9

Abstract

from arXiv · show

World Action Models (WAMs) combine future-video prediction with robot action learning, yet improvements in task success alone do not reveal how predictive supervision manifests in execution behavior or whether its benefits persist under task and environmental shifts. We conduct controlled future-video prediction loss ablations within FastWAM and OpenWAM, keeping model architecture, initialization, robot training data, and training steps fixed within each model family. Under clean conditions on seen tasks, predictive supervision increases success rates by 24.38 and 72.91 percentage points, respectively. Trajectory clustering further reveals higher success rates within several coarse trajectory clusters, including cases where cluster frequencies change little, indicating that the gains are not solely reflected in changes in the prevalence of trajectory patterns. However, these benefits do not consistently persist under distribution shifts: improvements on unseen tasks vary across models and tasks, while both models exhibit low success rates under the evaluated environmental randomizations, without a consistent advantage from predictive supervision. We further introduce future-video prediction as an auxiliary training objective for $\pi_{0.5}$, where a lightweight Transformer predictor used only during training increases success from 82.94% to 89.19%. These findings support future-video prediction as effective auxiliary supervision for action learning, while distinguishing improved execution on familiar tasks from reliable transfer and showing that such gains need not rely on a large pretrained video diffusion transformer (DiT) in the evaluated setting.

1. Introduction

The paper asks how future-video prediction supervision affects execution behavior and whether its benefits persist across tasks and environments. Controlled WAM comparisons show substantial seen-task gains, while analysis and auxiliary-prediction experiments examine the behavior and scope of those benefits.

2. Related Work

  • Prior WAM work combines future prediction and action learning, but aggregate success alone does not show how supervision affects execution or establish transfer across tasks and environments.Source: §1 · p. 1

1. Introduction

  • Controlled future-video-loss ablations in FastWAM and OpenWAM hold architecture, initialization, training data, and training steps fixed within each family.Source: §1 · pp. 1–2

2. Related Work

  • The π0.5 experiments test whether auxiliary future-video prediction can improve action performance without a large pretrained video DiT branch.Source: §1 · p. 2, §2 · p. 3

1. Introduction

  • Seen-clean success rises by 24.38 and 72.91 percentage points for FastWAM and OpenWAM, but gains vary on unseen tasks and neither model has a consistent advantage under environmental randomization.For π0.5, a training-only lightweight predictor raises success from 82.94% to 89.19%.Source: §1 · p. 2
  • Trajectory clustering finds higher within-cluster success in several coarse patterns even when their frequencies change little, so gains are not solely shifts in trajectory prevalence.Source: §1 · p. 2

2. Related Work

  • This study distinguishes training-time predictive supervision from using predicted future states to guide actions at inference, and evaluates the former with controlled comparisons and trajectory analysis.Source: §2 · pp. 2–3

3. Study Design

The study isolates the future-video prediction loss within each WAM family and evaluates it across task and environment conditions. It combines success-rate comparisons, coarse trajectory clustering, and a separate π0.5 auxiliary-prediction evaluation.

3.1. Controlled Variable: The Video Prediction Loss

  • Within each WAM family, the ablation changes only the future-video prediction loss weight while keeping architecture, initialization, module-freezing configuration, and training steps fixed.Source: §3.1 · p. 4

3.2. Tasks, Environments, and Evaluation Conditions

  • Seen tasks are included in training, Unseen tasks are excluded, Clean scenes match training, and Randomized scenes alter visual and scene configurations at test time.Source: §3.2 · p. 4

3.3. Evidence Types and Analysis Protocols

  • Success-rate comparisons cover ten Seen tasks under Clean conditions and a separate four-condition grid; trajectory clustering compares mode frequencies and within-mode success using numerical features, not verified strategies.The experiments use distinct task sets, so absolute success rates should not be compared across them.Source: §3.3 · p. 4, §3.2 · p. 4
  • The π0.5 configurations retain the action pathway and use either a lightweight Transformer or pretrained Wan DiT video branch only during training, removing it at inference.The video objective updates the shared vision-language pathway and prediction branch, not the action expert directly.Source: Figure 2 · p. 5

4. What Does Predictive Supervision Change?

The section tests whether future-video prediction improves familiar-task execution, how those gains appear in trajectory behavior, and whether they transfer under task or environmental shifts. Success rises substantially on clean, seen tasks, but transfer is inconsistent and environmental randomization yields no positive advantage.

4.1. Strong Gains on Familiar Tasks

  • FastWAM improves from 14.97% to 39.34% (+24.38 percentage points), while OpenWAM improves from 7.00% to 79.91% (+72.91 points) on ten Seen+Clean tasks.Both model families show substantial gains, although task-level gains are not uniformly positive.Source: §4.1 · p. 5, Table 1 · p. 6

4.2. Behavioral Evidence Beyond Aggregate Success

  • Within-mode success rises even when mode shares barely move: FastWAM’s stack blocks two M0 remains about 90% of trajectories as its success rises from 50.2% to 89.5%.Across tasks, predictive supervision does not produce a consistent shift in mode distribution, while improving within-mode success on multiple tasks.Source: §4.2 · p. 6, Table 2 · p. 7
  • Trajectory clusters are descriptive statistics, not evidence of a specific causal mechanism, and may partly reflect scene geometry.Source: §4.2 · p. 6

4.3. Limits of Transfer under Distribution Shift

  • Under Unseen+Clean, FastWAM gains 0.28 points and OpenWAM gains 23.38, with OpenWAM’s improvement concentrated in three tasks.Other unseen OpenWAM tasks improve little or decline slightly; the five Seen tasks in this experiment differ from the ten-task suite, so absolute success rates should not be compared across experiments.Source: §4.3 · p. 6, Table 3 · p. 7
  • Under Randomized conditions, both families reach at most 3.22% success and show small negative changes, with no observed positive advantage from predictive supervision.The severe floor effect means these negative differences do not establish that the no-prediction setting is more robust.Source: §4.3 · p. 6, Table 3 · p. 7
  • Seen-task execution gains and reliable transfer are distinct evaluation targets; substantial gains near the training distribution do not automatically confer transfer under evaluated environmental shifts.Source: §4.3 · p. 7

5. From Understanding to Design: Auxiliary Prediction for VLA

The paper tests whether training-time future-video prediction can benefit a conventional VLA policy, comparing lightweight and pretrained video-prediction branches while retaining π0.5’s action pathway. Both variants improve aggregate success on the evaluated tasks, but the comparison does not establish a reliable advantage for the lightweight branch over Wan or isolate model size as the cause.

5.1. Design Question and Auxiliary Prediction Variants

  • The study compares original π0.5, a lightweight Transformer video predictor, and pretrained Wan2.2-TI2V-5B, with the π0.5 action pathway unchanged.Both auxiliary branches are used during training and removed at inference.Source: §5.1 · p. 7

5.3. Results and Design Implications

  • 89.19% for the lightweight predictor versus 82.94% for original π0.5; Wan2.2-TI2V-5B reaches 86.97% (+4.03 points), and the lightweight variant’s 2.22-point lead over Wan is not statistically reliable.The confidence interval for the Wan-minus-lightweight difference includes zero.Source: Table 4 · p. 8, §5.3 · p. 8

5.2. Branch Architectures

  • The lightweight-versus-original comparison is relatively controlled, but Wan differs in architecture, initialization, conditioning, and loss weighting, preventing attribution to model size alone.The lightweight and original variants share initialization, training split, seed, optimizer, batch size, and training budget.Source: §5.2 · p. 8, §5.3 · p. 8

5.3. Results and Design Implications

  • Both auxiliary prediction variants improve π0.5 on the evaluated tasks, extending predictive supervision beyond dedicated WAM architectures.The lightweight branch uses video latents from a frozen Wan2.2 VAE as prediction targets, so the result does not show pretrained video representations are unnecessary altogether.Source: §5.3 · p. 8

6. Discussion

Predictive supervision improves familiar-task execution and changes success within similar trajectory patterns, but those gains do not reliably transfer. A lightweight auxiliary predictor also improves π0.5 in the evaluated setting, while several limits constrain interpretation and generalization.

Training-objective gains and transferability

  • FastWAM gains almost nothing on Unseen+Clean, OpenWAM’s cross-task gains concentrate in a subset of tasks, and both models have near-zero success under environmental randomization.Seen+Clean gains are +24.38 and +72.91 percentage points for FastWAM and OpenWAM, respectively.Source: §6 · p. 9

Behavior beyond aggregate success

  • On several tasks, within-cluster success rises even when coarse trajectory-mode frequencies remain nearly unchanged.The clustering results indicate gains within similar execution patterns, not only shifts in which modes policies select.Source: §6 · p. 9

Predictive supervision without a large generator

  • A randomly initialized six-layer lightweight Transformer predictor without a pretrained Wan DiT raises π0.5 success from 82.94% to 89.19%.The predictor is used as a training-time auxiliary objective; the current comparison does not isolate model size because architecture, initialization, conditioning, and loss weighting also differ.Source: §6 · p. 9

Limitations

  • The WAM controlled experiments use one training run per configuration, so rollout variation does not establish reproducibility across training seeds.The results support incremental trends only within the tested configurations.Source: §6 · p. 9
  • Low absolute success under environmental randomization makes it difficult to distinguish absent robustness benefits from insufficient model capability in those conditions.Both models have low success across all evaluated randomized conditions.Source: §6 · p. 9
  • Trajectory clusters use coarse numerical features that may reflect scene geometry, and the analysis does not identify internal control mechanisms.Action precision, force control, and specific kinematic indicators were not measured.Source: §6 · p. 9
  • The π0.5 comparison lacks systematic training-cost, memory, and latency measurements, and all experiments use bimanual tasks in RoboTwin 2.0 simulation.Generalization to other embodiments, real-world environments, or larger-scale training data remains unexamined.Source: §6 · pp. 9–10

7. Conclusion

Controlled experiments show that future-video prediction supervision improves familiar-task execution in FastWAM and OpenWAM, with trajectory-level analysis revealing behavioral changes beyond aggregate success. The gains do not reliably persist under task or environmental shifts; in π0.5, a training-only lightweight predictor also yields positive gains without necessarily requiring a large pretrained video-generation branch in the evaluated setting.

7. Conclusion

  • Future-video prediction substantially improves familiar-task execution in FastWAM and OpenWAM, but gains do not reliably persist on unseen tasks or under environmental randomization.For π0.5, a training-only lightweight predictor also produces positive gains without necessarily requiring a large pretrained video-generation branch in the evaluated setting.Source: §7 · p. 10

1. Detailed Evaluation Protocol

The evaluation isolates future-video prediction supervision within two WAM families and extends the comparison to a conventional VLA policy. Its four experiments cover seen-task success, trajectory behavior, distribution shifts, and auxiliary prediction designs.

1. Detailed Evaluation Protocol

  • Experiments 1–3 compare No-WM and WM within FastWAM and OpenWAM, while Experiment 4 compares prediction designs for π0.5 rather than a strictly architecture-matched pair.The protocols are designed to isolate future-video prediction supervision as much as possible in Experiments 1–3.Source: §1 · p. 12

1.1. Experiment 1: Seen+Clean Evaluation

  • Experiment 1 evaluates both WAM families on ten Seen+Clean tasks, using 3,200 rollouts per method for each family.OpenWAM uses ten fixed initial states per task and 32 rollouts per state; FastWAM also has 3,200 valid rollouts per method.Source: §1.1 · p. 12

1.2. Experiment 2: Behavior-Mode Analysis

  • Experiment 2 pools trajectories across model variants and scenes to measure behavior-mode frequency and success conditioned on each mode.Clusters are candidate modes from numerical trajectory features, not semantically validated strategies.Source: §1.2 · p. 12

1.3. Experiment 3: Distribution Shift

  • Experiment 3 evaluates five Seen and ten held-out Unseen tasks across Clean and Randomized environments, with 19,200 rollouts per model family across all four conditions.The four conditions are Seen+Clean, Seen+Randomized, Unseen+Clean, and Unseen+Randomized.Source: §1.3 · p. 12

1.4. Experiment 4: Extension to pi0.5

  • Experiment 4 compares original π0.5 with lightweight-predictor and Wan-based variants on the same ten Seen tasks; the prediction branches are training-only.The original and lightweight variants share initialization, training split, seed, and optimizer updates, whereas the Wan branch differs in several design choices and is an extension rather than a matched ablation.Source: §1.4 · p. 13

2. Detailed Results on Seen Tasks

The per-task Seen+Clean results show that gains vary by model and task: OpenWAM improves on every task, while FastWAM improves on most.

2.1. FastWAM

  • FastWAM improves on eight of ten tasks, with the largest gains on stack blocks three and place a2b right, but small degradations on put bottles dustbin and rotate qrcode.The size of the improvement varies substantially across tasks.Source: §2.1 · p. 13

2.2. OpenWAM

  • OpenWAM improves substantially on all ten Seen+Clean tasks; place mouse pad and stack blocks three rise from no successful No-WM rollouts to 57.50% and 85.62% success with WM.The two task-specific success rates are 57.50% for place mouse pad and 85.62% for stack blocks three.Source: §2.2 · p. 13

3. Detailed Behavior-Mode Analysis

Behavior-mode analysis finds higher success within coarse trajectory clusters, sometimes with little change in cluster frequency, while also showing that mode allocations can shift on some tasks.

3. Detailed Behavior-Mode Analysis

  • Across FastWAM and OpenWAM tasks, success often improves within the same coarse mode even when mode allocation changes little, though allocation shifts are not uniform across tasks.The clusters are descriptive candidate modes rather than definitive semantic strategy classes.Source: §3 · p. 13

3.1. FastWAM

  • In FastWAM place object stand, mode shares stay near 50%/50% as within-mode success rises from 67.8% to 77.7% and from 84.5% to 97.3%.In stack blocks two, the dominant mode changes from 90.4% to 90.0% while its success rate rises from 50.2% to 89.5%.Source: §3.1 · p. 13

3.2. OpenWAM

  • In OpenWAM open laptop, M0/M1 allocation shifts from 69.0%/31.0% to 47.1%/52.9%, while within-mode success rises from 22.7% and 22.9% to 78.4% and 94.4%.Source: §3.2 · p. 13

4. Detailed Results under Distribution Shift

Under distribution shifts, predictive-supervision gains depend on the model and task and weaken under environmental randomization. The strong Seen+Clean improvements therefore do not consistently transfer across the evaluated conditions.

4. Detailed Results under Distribution Shift

  • Across the four Seen/Unseen × Clean/Randomized conditions, task-level transfer varies substantially by model and task.The reported results compare five Seen tasks and ten Unseen tasks.Source: §4 · p. 13

4.1. FastWAM

  • FastWAM improves substantially on all five Seen+Clean tasks, but WM benefits are weaker and vary in sign under Seen+Randomized and both Unseen conditions.Source: §4.1 · p. 13

4.2. OpenWAM

  • OpenWAM improves greatly on all five Seen+Clean tasks, while Unseen+Clean gains concentrate in a subset of tasks and largely disappear under randomized environments.The cited Unseen+Clean examples are place container plate, open microwave, and press stapler.Source: §4.2 · pp. 13–14

5. Extension to pi0.5

The π0.5 extension tests future-video prediction as auxiliary training supervision, comparing lightweight and Wan-based predictors that are removed at inference. It reports higher aggregate success with both variants, while per-task rankings vary.

5.1. Architecture and Training Details

  • Both video branches condition on shared π0.5 representations; video loss updates shared vision-language components, while the private action expert is updated only by action loss.At inference, both video branches are removed.Source: §5.1 · pp. 14–15
  • The lightweight branch uses a randomly initialized six-layer, 512-dimensional, eight-head Transformer to predict two future latent blocks encoded from nine-frame windows with the frozen Wan2.2 VAE; the alternative conditions a pretrained Wan2.2-TI2V-5B DiT.The Wan-based variant uses an external video-loss coefficient of one from the first training step, together with the scheduler-dependent Wan objective weight, and has no step-based warmup.Source: §5.1 · pp. 14–15
  • The two video-auxiliary variants differ in architecture, initialization, conditioning, scheduler, and loss weighting, so their performance difference is not a strict causal ablation of model size or pretrained prior..Source: §5.1 · p. 16

5.2. Aggregate Results

  • 89.19% with the lightweight predictor versus 82.94% for original π0.5; both auxiliary variants show positive aggregate differences on the evaluated task set.The Wan2.2-TI2V-5B variant reaches 86.97%; confidence intervals for both variants versus original lie above zero.Source: §5.2 · p. 16

5.3. Per-Task Results

  • Task-level rankings vary: Wan has the highest point estimate on three listed tasks, while the lightweight predictor performs better on three others.The comparisons are task-dependent, and aggregate results do not establish that the larger pretrained model uniformly improves policy performance.Source: §5.3 · pp. 17–18, Table 10 · p. 18

Related papers

Loading 2610.34593v2…