Source-linked AI summary
AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing
Jisong Cai, Long Ling, Shiwei Chu, Zhongshan Liu, Jiayue Kang, Zhixuan Liang, Wenjie Xu, Yinan Mao, Weinan Zhang, Xiaokang Yang, Ru Ying, Ran Zheng, Yao Mu
TL;DR
Existing world-action models use the same short temporal rhythm for world prediction and action execution, limiting how the video branch supports control. AHA-WAM separates slow world planning from fast action execution and achieves 92.80% average success across 50 RoboTwin 2.0 tasks.
Problem
Existing world-action models couple world prediction and action execution over short horizons, making the video branch model redundant, weakly informative adjacent-frame variations.
Method
AHA-WAM decouples a low-frequency video-DiT planner with reusable long-horizon context from a high-frequency action-DiT executor, using OVCR and horizon-adaptive offset training for asynchronous control.
Results
92.80% average success across 50 RoboTwin 2.0 tasks, with 78.3% average success across four real-world tasks.
Takeaways & Limitations
AHA-WAM shows learned visual dynamics can improve robot control without compromising control frequency.
Takeaways & Limitations
Optimal planner frequency, video horizon, and action chunk size may depend on task dynamics and embodiment, while scaling to long-horizon tasks remains insufficiently evaluated.
Abstract
from arXiv · showhide
World-action models have emerged as a promising paradigm for robot manipulation, jointly modeling visual scene dynamics and actions to inject physical priors into policy learning. However, existing world-action models couple world prediction and action execution at the same temporal resolution, forcing the world branch to model near-term frame variations that are redundant and weakly informative. We posit that strictly binding world prediction and action execution to the same temporal rhythm may underutilize the potential of the video branch for embodied control. Therefore, we propose AHA-WAM, an Asynchronous Horizon-Adaptive World-Action Model built on a dual Diffusion Transformer (DiT) architecture that reorganizes world-action modeling around this temporal asymmetry. AHA-WAM instantiates the video DiT as a low-frequency world planner that maintains rolling key-value memory over past observations and exposes reusable layerwise latent context encoding long-horizon scene evolution, while a high-frequency action DiT executes short action chunks in closed loop by querying this context through layerwise joint attention. To support asynchronous execution, we introduce horizon-adaptive offset training and Observation-Guided Video-Context Routing (OVCR), which together let the action expert exploit long-horizon world context while remaining responsive to real-time execution state without rerunning the video DiT. Experiments on RoboTwin and real-world manipulation tasks show that AHA-WAM achieves state-of-the-art performance without any robot-data pretraining, attaining 92.80% average success on RoboTwin and 78.3% success across 4 real-world tasks, while reaching 24.17 Hz closed-loop control with a 4.59x speedup over Fast-WAM.
1 Introduction
AHA-WAM addresses the temporal mismatch in world-action modeling by separating low-frequency long-horizon world planning from high-frequency closed-loop action execution. It further uses OVCR and horizon-adaptive offset training to keep reused planner context aligned with real-time execution without recomputation.
- Motivation: Existing WAMs bind world prediction and action execution to the same short horizon, limiting how effectively the video branch can support robot control.This forces the world branch to model near-term frame variations that may be redundant or weakly informative.
- Method: AHA-WAM separates a low-frequency video-DiT world planner from a high-frequency action-DiT executor that predicts short action chunks in closed loop.The planner maintains rolling KV memory and exposes reusable layerwise latent context encoding long-horizon scene evolution, amortizing computation across action steps.
- Method: OVCR constructs chunk-specific latent queries from latest observations to route and update layerwise video context before action prediction.Proprioceptive feedback enters the action DiT directly, while visual feedback enters indirectly through routed video context, avoiding planner recomputation.
- Results: 92.80% average success across 50 RoboTwin 2.0 tasks demonstrates state-of-the-art-level performance without robot-data pretraining.The evaluation compares AHA-WAM with strong VLA and WAM baselines.
- Results: 78.3% average success across four real-world tasks shows robust deployment performance beyond simulation.The tasks cover deformable manipulation, long-horizon organization, fine-grained tool use, and spatial generalization.
2 Related Work
Related work spans generalist robot manipulation policies, video-based world models for predictive visual dynamics, and dual-system or asynchronous policies that combine deliberative computation with reactive control. These approaches motivate AHA-WAM’s focus on integrating world modeling with action execution beyond slow-branch guidance alone.
- Generalist robot manipulation policies: Generalist robot policies learn broadly applicable manipulation skills from large-scale, diverse demonstrations.RT-1 and RT-2 established the vision-language-action paradigm by combining heterogeneous robot trajectories with web-scale vision-language knowledge for robotic control.
- World models for robot control: Video-based robot world models augment action-only imitation with predictive visual dynamics and include imagine-then-act approaches.UniPi, Seer, and Video Prediction Policy predict future visual states or representations before recovering actions through inverse dynamics or a policy conditioned on those predictions.
- Dual-system and asynchronous robot policies: Dual-system and asynchronous policies combine deliberative computation with reactive control, but their slow branches primarily provide guidance, feedback, or representations to action policies.Examples include RoboDual, Reactive Diffusion Policy, and AsyncVLA, which pair slower semantic or latent processing with faster action or tactile pathways.
3 Asynchronous Horizon-Adaptive World-Action Modeling
AHA-WAM reframes world-action modeling as asynchronous two-timescale generation: a low-frequency video DiT plans longer-horizon scene evolution, while a high-frequency action DiT executes shorter chunks. Layerwise planner context, OVCR, rolling memory, and horizon-adaptive offset training support efficient, state-aligned closed-loop control.
- Asynchronous Horizon-Adaptive World-Action Modeling: AHA-WAM decouples temporal roles: the video branch models a longer horizon h_v, while the action branch predicts executable chunks of horizon h_a, with h_a < h_v.This replaces the shared short-horizon rhythm used by existing formulations with a two-timescale generation problem.
- Model architecture: The dual-DiT architecture combines a video DiT world planner with a same-depth, lightweight action DiT executor for layerwise interaction.Both branches are conditioned on language, while visual observations are encoded through a pretrained VAE.
- Layerwise planner–executor coupling: One planner forward produces layerwise latent world-plan context that multiple subsequent action-DiT forwards reuse, amortizing video computation across high-frequency action updates.At deployment, the video DiT refreshes planner context without explicit future-frame decoding, and the action DiT generates closed-loop chunks from it.
- Observation-Guided Video-Context Routing: OVCR converts shared planner context into chunk-specific, observation-conditioned context without rerunning the video DiT or feeding dense visual tokens into the action branch.It aligns reused context with the latest visual evidence while proprioceptive state is provided directly to the action DiT.
- Horizon-adaptive offset training: Horizon-adaptive offset training randomly shifts the action-chunk grid by δ ∈ [0, h_a) within the video planning horizon, exposing the executor to asynchronous phase offsets.The video planner covers [τ, τ + h_v), while action chunks need not begin at the planner start.
4 Experiments
Experiments evaluate AHA-WAM across RoboTwin 2.0 simulation, component ablations, real-world deployment and generalization, with success and closed-loop latency as primary measures. AHA-WAM preserves strong manipulation performance while substantially increasing control frequency over Fast-WAM.
- RoboTwin 2.0: 92.80% average success was achieved on RoboTwin 2.0, including 93.40% clean and 92.20% randomized success without robot-data pretraining.AHA-WAM improves over Fast-WAM by 0.97 percentage points.
- Component ablations: 88.60% success for Naive-Async shows that asynchronous execution alone is insufficient compared with its 91.83% starting performance.Naive-Async directly reuses the latest planner context without rolling K/V memory or OVCR.
- Component ablations: 91.01% success with rolling K/V memory and 91.47% with OVCR demonstrate progressive recovery from stale or phase-misaligned planner context.Rolling memory stabilizes reused video context across refreshes, while OVCR provides observation-conditioned context adaptation.
- Real-world deployment: 78.33% real-world success across four tasks outperforms Motus at 21.67% and Fast-WAM at 68.33%, while matching π0.5 at 76.67%.Under generalization shifts, AHA-WAM ranks second in success and achieves the highest progress score of 35.00.
- Inference efficiency: 24.17 Hz closed-loop control reduces latency to 41.37 ms from Fast-WAM’s 190.00 ms and increases frequency from 5.26 Hz.AHA-WAM-Flash further reaches 56.95 Hz at 17.56 ms, yielding a 10.82× speedup.
5 Conclusion
AHA-WAM decouples slow video planning from fast closed-loop action execution through observation-guided context routing and horizon-adaptive offset training. The conclusion identifies temporal-allocation limitations while highlighting asynchronous WAMs as a design space for more predictive, deliberative, or physically grounded video branches.
- Conclusion: AHA-WAM decouples a low-frequency video DiT world planner from a high-frequency action DiT closed-loop executor.Observation-Guided Video-Context Routing adapts reused planner context to each current observation, while horizon-adaptive offset training handles arbitrary planner-executor phase offsets.
- Limitations and future work: The planner update frequency, video horizon, and action chunk size create temporal hyperparameters whose optimal allocation may depend on task dynamics and embodiment.Future work includes longer-horizon prediction, richer scene representations, and systematic evaluation on dedicated long-horizon benchmarks.
- Broader implications: Decoupling the video branch from the high-frequency control loop allows more expensive computation without directly increasing action latency.The video branch can become more predictive, deliberative, or physically grounded while the action branch preserves a fast closed-loop rate for deployment.
A Implementation Settings
AHA-WAM’s RoboTwin implementation combines large Video-DiT and Action-DiT branches with memory/context routing, while using flow-matching objectives and separately specified ODE-distillation settings. The appendix summarizes these architecture, optimization, and inference configurations in Tables 4 and 5.
- Model architecture: The instantiated model contains approximately 4.99B video-branch parameters, 1.02B action-DiT parameters, and 1.22B memory and context-routing parameters.During asynchronous inference, the action path receives the latest observation and proprioceptive state while querying the model context.
- Implementation settings: Table 4 summarizes AHA-WAM’s Video-DiT planner, Action-DiT executor, OVCR interface, optimizer, scheduler, batch size, epochs, and default inference settings.These settings are specified for the RoboTwin implementation.
- ODE distillation: Table 5 records the distillation target, teacher–student denoising schedules, and main optimization settings for ODE-distilled fast sampling.Dataset, checkpoint, logging, and other run-specific bookkeeping are omitted.
- Training configuration: Both video prediction and action prediction use flow matching with noise times sampled from a logit-normal distribution.When both branches are optimized jointly, the video loss and action loss are weighted equally.
- Training configuration: Horizon-adaptive offset training follows Section 3.3, with each branch using the corresponding asynchronous training configuration.The supplied passage truncates after stating that horizon-adaptive offset training is applied.
B RoboTwin Evaluation Details
The RoboTwin 2.0 evaluation uses 50 AgileX dual-arm manipulation tasks with multi-task training and task-averaged success rates across clean and randomized episodes. AHA-WAM is compared against representative VLA and WAM baselines, including variants that distinguish robot-data pretraining from architecture effects.
- Benchmark protocol: 50 RoboTwin 2.0 dual-arm manipulation tasks are evaluated with the AgileX embodiment, spanning diverse bimanual skills.The benchmark contains 50 tasks.
- Benchmark protocol: 2,500 clean and 25,000 randomized demonstrations train AHA-WAM, using 50 clean and 500 randomized demonstrations per task.The training follows the multi-task setting used by prior WAM baselines.
- Benchmark protocol: 100 clean episodes and 100 randomized episodes per task are evaluated, with results reported as task-averaged success rates.Each task uses both clean and randomized evaluation conditions.
- Baseline comparison: AHA-WAM is compared with representative VLA and WAM baselines, including Fast-WAM, Motus, LingBot-VA, π0, π0.5, and ABot-M0.Fast-WAM is the closest baseline because it also uses a video DiT inside a world-action model.
- Baseline comparison: Embodied-pretrained and Wan2.2-initialized variants are included when available to separate robot-data pretraining effects from model architecture.The comparison spans both unified world-action architectures and generalist VLA policies.
C Real-World Task Execution and Scoring Criteria
The evaluation uses four real-world manipulation tasks, each executed through three ordered subtasks and assessed by binary success and a 0–3 partial-progress score over 30 trials per model-task pair.
- C Real-World Task Execution and Scoring Criteria: Each task comprises three ordered subtask steps executed sequentially during a rollout.These steps directly define the partial-progress score used in Figure 4.
- C Real-World Task Execution and Scoring Criteria: 30 independent trials per model-task pair determine binary success rates and scores on a 0–3 scale.Success is binary, while the score measures partial task progress.
- C Real-World Task Execution and Scoring Criteria: Four tasks span rigid-object placement, deformable manipulation, long-horizon multi-object organization, and fine-grained tool use.Together, they cover complementary real-world control challenges.
D Inference Speedup Details
The inference-speedup analysis measures closed-loop action latency through Lchunk while asynchronously refreshing planner context. Optimization combines CUDA deployment, hot-path streamlining, and action-path ODE distillation, with prefill mode selected to avoid increasing Lchunk.
- Measurement protocol: Lchunk measures end-to-end single action-chunk latency, including image encoding, planner-context access, OVCR routing, and Action DiT denoising.Video-DiT prefill runs asynchronously, so Lchunk is the primary closed-loop action-frequency metric; prefill throughput is tracked separately.
- CUDA acceleration: CUDA acceleration compiles Action DiT, memory/context modules, and the VAE encoder into TensorRT engines while replaying the fixed denoising loop with CUDA Graphs.Selective torch.compile is also used to reduce Python dispatch and redundant kernel-launch overhead.
- Prefill compilation: The default prefill compilation mode is selected because reduce-overhead achieves the lowest prefill latency but increases Lchunk.Prefill optimization targets sufficiently fast asynchronous planner-context refresh.
- Hot-path streamlining: Hoisting chunk-invariant computations reduces Lchunk from 71.45 ms to 63.25 ms, while removing redundant recursive state traversals further reduces latency to 50.37 ms.The optimized path also reuses static input buffers for video-KV tensors.
- ODE distillation: ODE distillation freezes the video DiT and distills only the action denoising path, preserving the 10-step teacher’s planner-context and OVCR interface.Training uses teacher trajectories with anchors {0, 1, 2, 4, 8, 12, 16} and emphasizes high-noise starting states.
E Per-Task RoboTwin Success Rates
Table 12 reports AHA-WAM’s per-task RoboTwin 2.0 success rates alongside selected baselines, separating clean and randomized evaluation. All reported values are percentages.
- Per-Task RoboTwin Success Rates: Table 12 reports per-task success rates for AHA-WAM and selected baselines under separate clean and randomized RoboTwin 2.0 evaluations.All values are percentages.