Source-linked AI summary
Fast-WAM: Do World Action Models Need Test-time Future Imagination?
Tianyuan Yuan, Zibin Dong, Yicheng Liu, Hang Zhao
TL;DR
WAMs use future visual prediction for embodied control, but existing systems make it unclear whether test-time imagination is necessary. Fast-WAM decouples video co-training from future generation by predicting actions directly from learned world representations, and remains competitive while running at 190 ms latency. Controlled comparisons indicate that video co-training contributes more than explicit future generation at inference.
Problem
Existing WAMs entangle video prediction during training with explicit future generation during inference, leaving the necessity of test-time imagination unclear.
Method
Fast-WAM retains video co-training during training but skips future prediction at inference, directly generating actions from latent world representations.
Results
Fast-WAM remains competitive with imagine-then-execute variants across simulation and real-world tasks, while removing video co-training causes a much larger degradation.
Takeaways & Limitations
Video prediction may primarily improve world representations during training rather than provide value through explicit future generation at test time.
Abstract
from arXiv · showhide
World Action Models (WAMs) have emerged as a promising alternative to Vision-Language-Action (VLA) models for embodied control because they explicitly model how visual observations may evolve under action. Most existing WAMs follow an imagine-then-execute paradigm, incurring substantial test-time latency from iterative video denoising, yet it remains unclear whether explicit future imagination is actually necessary for strong action performance. In this paper, we ask whether WAMs need explicit future imagination at test time, or whether their benefit comes primarily from video modeling during training. We disentangle the role of video modeling during training from explicit future generation during inference by proposing \textbf{Fast-WAM}, a WAM architecture that retains video co-training during training but skips future prediction at test time. We further instantiate several Fast-WAM variants to enable a controlled comparison of these two factors. Across these variants, we find that Fast-WAM remains competitive with imagine-then-execute variants, while removing video co-training causes a much larger performance drop. Empirically, Fast-WAM achieves competitive results with state-of-the-art methods both on simulation benchmarks (LIBERO and RoboTwin) and real-world tasks, without embodied pretraining. It runs in real time with 190ms latency, over 4$\times$ faster than existing imagine-then-execute WAMs. These results suggest that the main value of video prediction in WAMs may lie in improving world representations during training rather than generating future observations at test time. Project page: https://yuantianyuan01.github.io/FastWAM/
1 Introduction
Fast-WAM separates training-time video modeling from test-time future imagination to test which factor drives WAM performance. It preserves video co-training while directly predicting actions, achieving competitive results with lower inference latency.
- WAMs model future visual observations alongside actions, motivated by the potential to capture physical dynamics and task-relevant temporal structure.
- Existing WAMs entangle video prediction during training with explicit future generation during inference, leaving the necessity of test-time imagination unclear.
- Fast-WAM retains video co-training but skips future prediction at test time, directly generating actions from latent world representations in one forward pass.
- Controlled comparisons show Fast-WAM remains competitive with imagine-then-execute variants, whereas removing video co-training causes a much larger performance drop.
- 190 ms latency makes Fast-WAM more than 4× faster than existing imagine-then-execute WAM approaches on real-world robotic tasks.
- The study identifies whether WAM gains primarily reflect video modeling during training or explicit future imagination during inference.
2 Related Work
Related work includes VLA policies and video-based robot policies, including WAMs that generate or jointly model future video and actions. Fast-WAM instead studies the separate roles of training-time video co-training and test-time future imagination.
- VLA policies map visual observations and language instructions to robot actions using pretrained vision-language backbones.
- Video-based robot policies use future visual prediction to model environment dynamics and infer actions.
- Recent WAMs either generate future visual trajectories before action prediction or jointly model future video and actions in a shared generative process.
- VPP and UVA reduce or bypass explicit test-time video synthesis, while Fast-WAM focuses on disentangling training-time video co-training from test-time future imagination.
3 Method
Fast-WAM decouples video co-training from explicit future generation, using latent world representations for direct action prediction at test time. Its controlled variants isolate the roles of video co-training and test-time imagination while reducing inference cost.
- Fast-WAM Design: Fast-WAM retains video co-training during training but predicts actions without explicitly generating future observations at inference.It uses a single forward encoding pass to obtain latent world features instead of sampling or denoising future video.
- Fast-WAM Design: Fast-WAM parameterizes the action distribution with a latent world representation produced from the current observation and instruction.The video backbone produces z(o, l), which conditions action generation through a direct-policy interface.
- Model Architecture: Fast-WAM uses a Mixture-of-Transformer architecture combining a video Diffusion Transformer backbone with an action expert for action-chunk generation.The architecture reuses pretrained video components and shares attention between video and action processing.
- Model Architecture: At inference, Fast-WAM removes the future-video branch and processes only clean first-frame tokens once, avoiding future-video denoising.This lowers inference cost relative to imagine-then-execute WAMs.
- Model Architecture: A structured attention mask prevents action tokens from attending to future video tokens while grounding both branches in the same visual context.Clean first-frame tokens attend to no other tokens, preventing future information from leaking into the action branch.
- Training Objective: The joint training objective combines action flow matching and video co-training, with λ balancing the two losses.The same flow-matching formulation is instantiated for action chunks and future-video latent tokens.
- Controlled Variants: Controlled variants preserve aligned implementation choices while isolating video co-training from explicit future imagination.The study includes representative imagine-then-execute designs and a no-video-co-training control that removes only the video modeling objective.
4 Experiment
Experiments compare Fast-WAM with imagine-then-execute variants and a no-video-co-training ablation across simulation and real-world tasks. Fast-WAM preserves strong performance while reducing inference latency, whereas removing video co-training produces substantially larger degradation.
- Experiment setup: Fast-WAM is evaluated on LIBERO, RoboTwin 2.0, and real-world towel folding, using simulation and robotic manipulation benchmarks.The real-world task requires long-horizon reasoning about deformable-object dynamics.
- Overall comparison on simulation benchmarks: 91.8% success rate on RoboTwin is achieved without embodied pretraining, exceeding all baselines without embodied pretraining.Fast-WAM remains comparable to pretrained LingBot-VA at 92.2%.
- Overall comparison on simulation benchmarks: 97.6% average success rate on LIBERO is achieved without embodied pretraining, outperforming π0.5 and remaining competitive with pretrained WAM baselines.Pretrained LingBot-VA and Motus achieve 98.5% and 97.7%, respectively.
- Controlled comparison with Fast-WAM variants: Across both simulation benchmarks, Fast-WAM remains comparable to imagine-then-execute variants, while removing video co-training causes a much larger performance drop.On RoboTwin, success falls from 91.8% to 83.8% without video co-training; on LIBERO, it falls from 97.6% to 93.5%.
- Real-world performance and efficiency: 190 ms inference latency makes Fast-WAM more than 4× faster than existing imagine-then-execute WAM approaches.Fast-WAM-IDM requires 810 ms.
5 Conclusion
The paper concludes that Fast-WAM can retain video co-training while skipping future prediction at inference, achieving strong performance without embodied pretraining and running in real time. Controlled comparisons indicate that video prediction during training contributes more than explicit future generation at test time, while larger-scale pretraining and model scaling remain future directions.
- Conclusion: Fast-WAM retains video co-training while skipping future prediction at inference, generating actions from world-grounded latent representations.The architecture is evaluated across simulation benchmarks and real-world robotic tasks.
- Conclusion: Fast-WAM remains competitive with imagine-then-execute variants, whereas removing video co-training causes much larger degradation.This pattern holds across simulation benchmarks and real-world robotic tasks.
- Conclusion: Video prediction may provide its main value by learning better world representations during training rather than generating future observations at test time.The conclusion presents this as a supported interpretation of the controlled comparisons.
- Conclusion: Larger-scale pretraining data and model scaling are identified as important directions for future work.The paper does not resolve their effects on this design.
A.1 RoboTwin Detailed Results
This section presents RoboTwin’s per-task success rates under clean and randomized evaluation settings.
- A.1 RoboTwin Detailed Results: Table 3 reports per-task success rates on RoboTwin.
- A.1 RoboTwin Detailed Results: Results are reported separately for clean and randomized evaluation settings.