Source-linked AI summary
When to Trust Imagination: Adaptive Action Execution for World Action Models
Rui Wang, Yue Zhang, Jiehong Lin, Kuncheng Luo, Jianan Wang, Zhongrui Wang, Xiaojuan Qi
TL;DR
Current World Action Models execute fixed-length action chunks without verifying whether their imagined futures still match reality. This paper introduces FFDC-WAM for adaptive replanning, reducing RoboTwin forward passes by 69.10% and execution time by 34.02% while improving success rate by 2.54% over the short-chunk baseline.
Problem
Current WAM execution uses fixed-size action chunks despite varying prediction reliability across tasks and task phases.
Method
FFDC-WAM jointly compares predicted actions and visual dynamics with real observations and language, using mixture-of-horizon training for adaptive execution.
Results
69.10% fewer WAM forward passes, 34.02% less execution time, and 2.54% higher success rate than the short-chunk baseline were achieved on RoboTwin, while real-world success improved by 35%.
Takeaways & Limitations
FFDC-WAM supports longer execution in predictable phases and earlier replanning when future–reality consistency breaks down across simulation and real-world settings.
Takeaways & Limitations
FFDC’s binary supervision from successful, failed, and synthetically corrupted segments may not cover the full diversity of real-world execution deviations.
Abstract
from arXiv · showhide
World Action Models (WAMs) have recently emerged as a promising paradigm for robotic manipulation by jointly predicting future visual observations and future actions. However, current WAMs typically execute a fixed number of predicted actions after each model inference, leaving the robot blind to whether the imagined future remains consistent with the actual physical rollout. In this work, we formulate adaptive WAM execution as a future-reality verification problem: the robot should execute longer when the WAM-predicted future remains reliable, and replan earlier when reality deviates from imagination. To this end, we propose Future Forward Dynamics Causal Attention (FFDC), a lightweight verifier that jointly reasons over predicted future actions, predicted visual dynamics, real observations, and language instructions to estimate whether the remaining action rollout can still be trusted. FFDC enables adaptive action chunk sizes as an emergent consequence of prediction-observation consistency, preserving the efficiency of long-horizon execution while restoring responsiveness in contact-rich or difficult phases. We further introduce Mixture-of-Horizon Training to improve long-horizon trajectory coverage for adaptive execution. Experiments on the RoboTwin benchmark and in the real world demonstrate that our method achieves a strong robustness-efficiency trade-off: on RoboTwin, it reduces WAM forward passes by 69.10% and execution time by 34.02%, while improving success rate by 2.54% over the short-chunk baseline; in real-world experiments, it improves success rate by 35%.
1 Introduction
The paper formulates adaptive World Action Model execution as future–reality verification: trust longer rollouts when predicted and observed futures agree, and replan when they diverge. FFDC operationalizes this principle, improving the efficiency–robustness trade-off across benchmark and real-world experiments.
- Problem formulation: Adaptive WAM execution compares predicted visual futures with physical observations to determine whether remaining action rollouts remain trustworthy.This reframes execution as future–reality verification rather than fixed open-loop control.
- Method: FFDC is a lightweight verifier that jointly models predicted actions, predicted visual dynamics, real observations, and language instructions.Its temporally aligned causal attention detects task-critical mismatches and unreliable future execution.
- Adaptive execution: Adaptive trust enables long execution in predictable phases and short execution in difficult phases, making action chunk size an emergent consequence of prediction–observation consistency.Stable rollouts reduce computation, while detected mismatches trigger earlier replanning for improved robustness.
- Experiments: 69.10% fewer WAM forward passes and 34.02% lower execution time were achieved on RoboTwin, with success rate improving 2.54% over the short-chunk baseline.These results demonstrate a stronger robustness–efficiency trade-off than the short-chunk baseline.
- Experiments: 35% higher success rate was achieved in real-world experiments.The experiments validate the method beyond the RoboTwin benchmark.
2 Related work
WAMs jointly model future visual observations and actions, enabling several control-relevant distributions, but their predicted futures are often underused during inference. Adaptive execution methods address fixed open-loop rollout through corrective supervision or risk signals, yet generally do not compare imagined scene evolution with physical rollout.
- World action models: WAMs extend VLA policies by jointly generating future videos and actions, modeling forward, inverse, marginal action, and marginal image distributions.These distributions include p(o′ | o, a), p(a | o, o′), p(a | o), and p(o′ | o).
- World action models: Recent WAMs perform strongly in zero-shot control, cross-environment transfer, and cross-embodiment learning.Because pixel-level video decoding is expensive, some methods use future video prediction mainly during training and rely on latent features or skip explicit future rollout at inference.
- Adaptive action execution: Adaptive action execution methods mitigate fixed open-loop rollout through corrective supervision, recovery behaviors, uncertainty estimates, novelty, or diffusion loss.These signals can trigger expert intervention or corrective replanning during execution.
- Adaptive action execution: Existing adaptive execution methods generally cannot compare internal future expectations with actual physical rollout because they do not explicitly predict future scene evolution under planned actions.They are primarily designed for action-only policies or VLA models and rely on observations, predicted actions, uncertainty, entropy, or auxiliary confidence.
3 Method
FFDC-WAM combines low-frequency WAM macro-planning with high-frequency lightweight verification to adapt action-chunk execution based on consistency between imagined futures and real observations. Its FFDC verifier uses cached action-visual-language predictions and temporally causal attention, while mixture-of-horizon sampling improves long-horizon trajectory coverage.
- Framework: FFDC-WAM combines WAM macro-planning with lightweight high-frequency verification for efficient adaptive action execution.The framework leverages WAMs’ joint video-action modeling capability.
- Adaptive action execution: The verifier continues the current rollout when confidence et ≥ 0.5 and replans when confidence falls below 0.5.It evaluates the latest observation against predicted future actions, visual tokens, and the language instruction.
- FFDC verifier: FFDC verifies horizon-k candidate action segments using temporally aligned past and future visual tokens, future actions, real observations, and instruction-conditioned semantic tokens.The action-to-video frequency ratio r determines the temporal sampling of predicted visual tokens.
- FFDC verifier: Cached WAM predictions let FFDC encode only the latest real observation, while local-window causal attention preserves temporal causality and reduces computation without rerunning the full WAM.The verifier stores predicted action, visual, and semantic tokens as a KV cache and attends locally over aligned future tokens.
- Mixture-of-Horizon Training: Mixture-of-horizon sampling uniformly selects conditioning timesteps and clips action-video indices to episode length, expanding long-horizon trajectory coverage.Repeating final valid actions or frames enables late-stage states to serve as training starts and reduces bias toward early-episode prefixes.
- Verifier training: The verifier is trained on executable and failure-inducing segments, using demonstrations, successful and failed rollouts, and corrupted valid actions as negative samples.The binary target y ∈{0, 1} indicates whether a future action segment is executable.
4 Experiments
Experiments show that FFDC-WAM adaptively balances robustness and efficiency across RoboTwin tasks by executing long chunks when predictions remain reliable and replanning when confidence drops. Ablations further show that jointly modeling predicted futures, real observations, actions, and language is important for confidence estimation.
- RoboTwin evaluation: FFDC-WAM raises SR from 54.20% to 76.40% on Rand.hard and from 57.80% to 76.00% on Clean.hard.On easy tasks, completion time drops from 23.5s to 15.7s on Rand.easy and from 20.4s to 12.9s on Clean.easy while maintaining comparable SR.
- Qualitative execution behavior: One WAM inference completes move can pot with FFDC-WAM, versus three for Base-Motus, because consistently high confidence permits continued execution without replanning.The task has simple dynamics, allowing FFDC-WAM to use a long action chunk.
- Qualitative execution behavior: Seven WAM inferences are required by Base-Motus on hanging mug, whereas FFDC-WAM executes long transport chunks and replans frequently when confidence drops during precision-critical hanging.Executing the same long chunk without FFDC accumulates error and eventually fails.
- Input ablation: 76.4% average success rate and 20.5s average completion time are achieved by the full FFDC model, outperforming variants that remove any input.The inputs are language, predicted visual tokens, the real observation, and predicted actions.
- Input ablation: Removing predicted visual tokens lowers average success rate from 76.4% to 71.6%, the largest drop among the ablations.This supports jointly modeling the imagined future, current real state, intended action rollout, and task instruction for confidence estimation.
5 Conclusion
FFDC-WAM reformulates adaptive WAM execution as online future–reality verification, enabling execution to continue while imagined futures remain reliable and replanning when consistency breaks down. The approach advances reliability-aware control beyond fixed-horizon chunking, but its binary supervision may not cover the diversity of real-world deviations.
- Core approach: FFDC-WAM treats adaptive execution as a future–reality verification problem rather than fixed-horizon action chunking.Its FFDC verifier jointly models temporally aligned information during rollout.
- Adaptive execution: The robot continues executing when predicted futures remain reliable and triggers replanning when future–reality consistency breaks down.This makes adaptive trust in WAM imagination an emergent control strategy.
- Implication: FFDC-WAM moves WAM deployment beyond fixed-horizon chunking toward reliability-aware control that verifies imagined futures online.The conclusion frames online self-verification, rather than selecting one execution length, as central to effective deployment.
- Limitation: FFDC’s binary supervision uses successful, failed, and synthetically corrupted segments, which may not cover the full diversity of real-world execution deviations.Future work should use richer failure modes and more diverse real-world data.
A Technical appendices and supplementary material · A.1 Limitations
The current FFDC design is lightweight and effective, but its parameter-scale versus verification-capability trade-off requires further study. The method also uses a fixed FFDC execution-decision threshold of 0.5.
- A.1 Limitations: FFDC uses a lightweight mechanism to model predicted visual features, predicted actions, current observations, and language instructions.The design is described as efficient and effective in the reported experiments.
- A.1 Limitations: Further work is needed to explore the trade-off between FFDC parameter scale and verification capability.The limitation concerns how module size relates to verification performance.
- A.1 Limitations: FFDC-based execution decisions currently use a fixed detection threshold of 0.5.The supplied passage identifies this fixed threshold as part of the current method’s limitations.
A.2 Additional experimental results
Additional experiments report results separately for hard tasks and for easy tasks under random and clean environment settings. The reported metrics include success rate and duration in seconds.
- SR denotes success rate, while T denotes duration in seconds.
- Results are presented for hard tasks in Table 4.
- Easy-task results under the random environment setting are presented in Table 5.
- Easy-task results under the clean environment setting are presented in Table 6.