Source-linked AI summary
Learning Transferable Dynamics Priors from Action to World Modeling
Ze Huang, Jiahui Zhang, Hairuo Liu, Chenxi Zhang, Ran Cheng, Li Zhang
TL;DR
Robot learning lacks clear evidence that action-conditioned world-model pretraining yields transferable dynamics priors across simulator-centric and policy-centric settings. The paper introduces A2World and adapts it into A2World-sim and A2World-policy, finding consistently stronger transfer across simulation benchmarks and real-robot experiments than text-conditioned or task-specific pretraining.
Problem
It remains unclear whether action-conditioned world-model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.
Method
The authors pretrain the multi-view diffusion world model A2World on real robot action annotations, then adapt it for simulation rollouts and instruction-conditioned action prediction.
Results
Action-to-video pretraining consistently produces a stronger transferable prior than text-conditioned or task-specific robot pretraining across simulation benchmarks and real-robot experiments.
Takeaways & Limitations
A2World’s learned prior supports both long-horizon simulator-based policy evaluation and instruction-conditioned control within the reported settings.
Abstract
from arXiv · showhide
We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation. Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning. Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.
1 Introduction
The paper pretrains A2World, a multi-view interactive diffusion world model, directly on real robot action annotations to learn transferable dynamics priors. It evaluates these priors through simulator-side and policy-side adaptations for robot learning.
- Pretraining: A2World is pretrained on openly available robot manipulation datasets using real robot action annotations and multi-view interactive diffusion modeling.The approach avoids auxiliary latent-action models that generate indirect pseudo-labels, promoting generalization across embodiments, camera setups, tasks, and motion patterns.
- Evaluation: The pretrained model is evaluated from two complementary perspectives central to robot learning.These perspectives adapt A2World for simulator-side modeling and policy-side learning.
- Simulation-side adaptation: On the simulation side, A2World is fine-tuned into a history-aware autoregressive world model, A2World-sim, and specialized as a task-specific model.This adaptation is intended to assess the value of action-conditioned pretraining for simulation-based robot learning.
2 Related works
Prior work uses action-conditioned robot world models to generate future manipulation videos and support VLA data augmentation, policy validation, and increasingly policy improvement. Related world-action models instead predict scene evolution before converting visual changes into control commands.
- Action-conditioned robot world models: Action-conditioned robot world models condition on low-level signals such as end-effector pose changes or joint-angle deltas to generate future manipulation videos.Applications include VLA data augmentation and safe, scalable policy validation without real-robot execution.
- Action-conditioned robot world models: Recent pipelines couple real-world world model simulation with policy improvement, moving beyond validation-only uses that replace standard-benchmark simulators.Earlier applications included learned-world-model substitution in standard benchmarks and policy validation without real-robot execution.
- World-action models: World-action models predict how scenes evolve from observations and then convert the predicted visual changes into control commands.They differ from approaches that predict control commands directly from a single observation.
3 Methodology
A2World learns action-conditioned dynamics priors by forecasting future multi-view observations from action chunks, using action injection and jointly consistent view modeling. The pretrained model is then adapted into history-aware simulation and instruction-conditioned policy models.
- A2World: A2World forecasts future observations from a conditioning frame and future action chunk, learning transferable action-to-dynamics priors.It is trained with EDM denoising score matching to recover clean latent video sequences from corrupted versions.
- A2World: Action chunks are encoded by an MLP and added to diffusion timestep embeddings consumed by every DiT block.Adaptive layer normalization uses the resulting embeddings to produce dynamic modulation.
- Multi-view generation: A2World jointly generates multi-view videos by temporally concatenating view latents, adding learnable camera embeddings, and applying cross-view attention.The design provides explicit camera identity while encouraging spatiotemporal consistency and preserving view-specific details.
- A2World-sim: A2World-sim transfers the pretrained prior into a history-aware autoregressive world model for sequential long-horizon rollouts under actions.Pose-guided history sampling selects frames along weighted relative-action arc length, while history tokens enter cross-attention and self-attention memory paths.
- A2World-policy: A2World-policy jointly predicts future observations and actions from an initial frame and language instruction in a single generative process.A pretrained T5 encoder supplies instruction context, while video and action tokens share self-attention with modality-specific AdaLN and MLP branches.
4 Experiments
Experiments show that A2World learns action-responsive, counterfactual dynamics and transfers them to both simulator-based evaluation and policy learning. A2World-sim achieves strong rollout quality and real-world agreement, while A2World-policy performs strongly on standard and out-of-distribution tasks.
- Qualitative rollout evaluation: A2World generates action-conditioned multi-view rollouts that support different grasps, failed attempts, full-DoF control, and unseen scenes or camera setups.These qualitative results indicate responsiveness to action inputs beyond success-only video generation and demonstrate counterfactual controllability across DROID, RoboCoin, RoboMind, and VIOLA.
- Rollout quality evaluation: A2World-sim consistently achieves the best rollout quality across LIBERO and real-robot data, improving both appearance metrics and action-faithfulness.On long-horizon generation, baselines drift after approximately 6s and collapse, whereas A2World follows actions more faithfully, completes the task, and maintains visual quality.
- Simulator-based policy evaluation: 0.916 Spearman ρ, 0.965 Pearson r, and 0.930 R2 show strong agreement between A2World-sim and the real world across N = 8 policies.Success rates were estimated from approximately 25 real rollouts and 64 simulator rollouts per policy, with outcomes manually verified.
- Evaluation on LIBERO: 98.6% overall success rate makes A2World-policy the best-performing method on LIBERO under the standard four-suite protocol.The policy is initialized from pretrained A2World weights and fine-tuned on downstream robot data.
- OOD policy evaluation: 88.5% average success rate is achieved by A-pre on LIBERO-Plus Spatial after fine-tuning on LIBERO, outperforming the compared initialization and pretraining alternatives.The comparison includes C-init, T-pre, A-pre, and P-pre under visual, linguistic, and dynamics shifts.
- Ablations: Pose-guided history sampling consistently improves video generation quality, while full joint training reaches a stronger video-consistency and action-quality frontier than freezing the video branch.The video-frozen variant achieves 86.2% in Tab. 7 while keeping the action branch and shared transformer layers trainable.
5 Conclusions
The conclusion presents action-conditioned world modeling as a scalable approach for learning transferable dynamics priors. A2World uses causal action supervision and can be adapted for long-horizon rollout and policy learning, while video prediction quality correlates with action-generation quality.
- Policy learning: Improving video prediction consistently correlates with better action generation during A2World-policy training.This links video-modeling progress with policy-action prediction performance.
- Core conclusion: Action-conditioned world modeling learns transferable dynamics priors by using actions as causal supervision during large-scale action-to-video pretraining.The learned interaction knowledge is described as reusable across tasks and environments.
- Core conclusion: A2World instantiates this approach as a multi-view diffusion world model pretrained on diverse real-robot data.Its learned prior is adapted to A2World-sim for long-horizon rollout and to A2World-policy for action prediction.
6 Supplement implementation details
The supplement details dual-arm pose-guided history sampling and A2World-policy’s real-robot training configuration. The sampling preserves motion coverage through pose-induced arc length, while joint video-action diffusion uses modality-specific noise scaling and weighted denoising.
- Pose-guided history sampling: Dual-arm history sampling computes motion distance from weighted translational and rotational changes for both arms, then samples frames uniformly in cumulative arc-length space.The method retains anchor frames and selects indices nearest uniformly spaced arc-length targets under a fixed history budget.
- A2World-policy settings: A2World-policy uses video at 10 fps and actions at 30 Hz over a 60-step action horizon in real-robot experiments.Its backbone and video diffusion configuration largely follow A2World, with only a few policy-specific settings.
- A2World-policy settings: Joint video-action diffusion samples shared base noise σbase, scales modalities with mv = 6 and ma = 0.5, and applies 0.05 high-σ augmentation on video.Action noise remains coupled to the shared σbase.
- A2World-policy settings: The weighted joint denoising objective sets λa = 1, and the final backpropagated loss is scaled by 10.The objective uses the noise-dependent weighting function w(σ) for both modality-specific reconstruction terms.
7 Supplement experimental details
Supplementary experiments show that A2World supports fine-grained, coherent action-conditioned rollouts, while A2World-sim preserves long-horizon closed-loop consistency for real-world policy evaluation. Additional analyses define the action-quality metrics and report qualitative advantages for A2World-policy on contact-rich real-robot tasks.
- Qualitative controllability: A2World faithfully follows scripted dual-arm pose commands of increasing magnitudes while maintaining coherent multi-view rollouts.Commands include 20/30/40cm left-arm translations and 10/20/30° right-arm rotations.
- World model as real-world simulator: A2World-sim preserves spatiotemporal coherence over extended horizons during closed-loop π0.5 rollouts generated step-by-step from model observations.The supplementary rollout spans 200 frames, corresponding to 20s at 10fps, across five real-world tasks.
- Out-of-distribution simulator evaluation: Under OOD evaluation from LIBERO to LIBERO-Plus Spatial, action-conditioned A2World pretraining improves action-faithfulness metrics over DreamDojo.In an unseen blue-background scene, DreamDojo tends to drift toward training-domain appearance.
- Definition of the axes in Fig. 11: The Fig. 11 action-quality score averages normalized translation, rotation, and gripper-state errors, with larger values indicating better relative checkpoint quality.Errors are computed in physical units after inverse dataset normalization; the score is intended for within-figure trend comparison, not absolute cross-experiment comparison.
- Qualitative real-robot results: On the long-horizon Put chain in the box task, A2World-policy more consistently completes the full sequence than π0.5 and LingBot-VA.Baselines often fail to place the chain or cannot finish closing the box.