Source-linked AI summary
World Action Models are Zero-shot Policies
Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi "Jim" Fan, Joel Jang
TL;DR
VLAs remain limited in generalizing to novel environments, tasks, and physical motions because they lack representations of precise action execution. DreamZero jointly predicts future video and actions with a pretrained video diffusion backbone, achieving over 2× improvement over state-of-the-art VLAs while enabling real-time control and cross-embodiment transfer.
Problem
VLAs generalize across language instructions and objects but remain limited on novel environments, motions, and skills requiring precise spatial, geometric, dynamic, and motor-control representations.
Method
DreamZero is a World Action Model that jointly predicts future video and actions using a 14B autoregressive video diffusion transformer trained on diverse heterogeneous robot data.
Results
DreamZero achieves more than a 2× improvement in average task progress on environment and task generalization benchmarks over state-of-the-art pretrained VLAs.
Takeaways & Limitations
DreamZero extends robot-policy generalization across environments, tasks, and embodiments while supporting 7Hz closed-loop control and video-only cross-embodiment transfer.
Takeaways & Limitations
Despite 7Hz control, DreamZero remains computationally expensive because of its large parameter size and iterative video denoising.
Abstract
from arXiv · showhide
State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.
1. Introduction
DreamZero addresses VLAs’ limited generalization to novel motions, tasks, and environments by jointly modeling video futures and actions. The resulting WAM supports heterogeneous-data learning, broad generalization, real-time control, and cross-embodiment transfer.
- Motivation: VLAs generalize across language instructions and objects but remain limited on novel environments, motions, and skills.They lack representations of precise spatial awareness, geometry, dynamics, and motor control.
- Approach: DreamZero is a 14B World Action Model that jointly predicts visual future states and motor actions from language and observations.Built on a pretrained image-to-video diffusion backbone, it shifts action learning toward inverse dynamics aligned with predicted visual futures.
- Data and architecture: DreamZero learns effectively from heterogeneous useful-behavior trajectories rather than requiring multiple repeated demonstrations per task.The model analysis also reports that diverse training data is essential for generalization and that autoregressive architectures improve motion smoothness and modality alignment.
- Generalization: More than 2× improvement in average task progress over state-of-the-art pretrained VLAs is observed on environment and task generalization benchmarks.Environment generalization remains after task-specific post-training, with 10% higher average task progress than state-of-the-art VLAs.
- Cross-embodiment transfer: Over 42% relative improvement on unseen tasks comes from 10–20 minutes of video-only demonstrations by humans or another robot.A further 30 minutes of play data enables adaptation from AgiBot G1 to YAM while retaining zero-shot generalization.
- Efficiency: 38× inference speedup enables DreamZero to perform real-time closed-loop control at 7Hz.The optimizations combine algorithmic changes, system parallelism and caching, quantization, and CUDA kernel tuning.
2. Related Work
Prior work extends vision-language models to robot actions, but generalization remains limited for novel physical skills and environments. World Action Models address this by coupling future-state video prediction with action prediction, while DreamZero emphasizes scale, diversity, autoregression, and cross-embodiment transfer.
- Vision-Language-Action Models: VLAs inherit linguistic priors for diverse instructions and objects but remain limited in generalizing to novel environments, skills, and motions.
- DreamZero: DreamZero shows non-trivial performance on Genie Sim 3.0 despite training on approximately 500 hours of real-world data and no explicit training on its 10k simulation hours.
- Vision-Language-Action Models: Existing VLA task generalization relies on large libraries of language-conditioned motion primitives, which cannot practically cover the full range of physical interactions.
- World Action Models: World Action Models jointly model future world states and actions, using pretrained video diffusion representations to capture visual dynamics for policy learning.
- DreamZero: DreamZero differs from prior WAMs through systematic exploration of data diversity and scale, autoregressive long-horizon modeling, and generalization across tasks, environments, and embodiments.
3. DreamZero
DreamZero is an autoregressive World Action Model that jointly predicts video futures and actions from language, observations, and proprioception. Its closed-loop architecture and inference optimizations align video-action prediction with real-time robotic control.
- Model Architecture: DreamZero jointly predicts future video and actions from language instructions, proprioceptive state, and visual observation history.
- Model Architecture: A single end-to-end model jointly denoises video and action modalities to improve their alignment rather than using separate video and inverse-dynamics models.
- Model Architecture: Autoregressive video generation uses visual history and KV caching, while replacing predicted frames with ground-truth observations prevents compounding errors during closed-loop inference.
- Real-Time Control: Diffusion-based WAMs face a 5.7-second naive latency caused by iterative denoising, the 14B DiT backbone, and sequential execution, making closed-loop control infeasible.
- Real-Time Control: DreamZero-Flash biases video toward noisy states while keeping action noise uniform, reducing diffusion steps from four to one and latency from approximately 350ms to 150ms with minimal performance loss.
4. Experimental Setup
DreamZero is evaluated on diverse robot data, embodiments, tasks, and out-of-distribution environments. The setup emphasizes heterogeneous pretraining, matched VLA comparisons, and separate seen-task and unseen-task protocols.
- Pretraining: Experiments use two embodiments: the AgiBot G1 mobile bimanual manipulator and the Franka single-arm robot, with YAM and human data for cross-embodiment evaluation.
- Baselines: DreamZero is compared with GR00T N1.6 and π0.5 using from-scratch and officially pretrained initializations, identical training data, and comparable compute budgets.
- Pretraining: AgiBot pretraining contains approximately 500 hours across 22 environments, with episodes averaging 4.4 minutes and approximately 42 subtasks.
- Pretraining: The AgiBot data strategy prioritizes task diversity and real-world utility over task-specific repetition, supporting learning from heterogeneous demonstrations.
- Evaluation Protocol: Evaluation defaults to unseen environments and unseen objects, testing out-of-distribution generalization because training and evaluation sites differ geographically.
- Evaluation Protocol: AgiBot evaluation covers seen and unseen tasks across multiple environments and objects, while DROID evaluates 20 seen and 20 unseen tasks with 80 total rollouts per checkpoint.
- Post-Training: Fine-tuning evaluates shirt folding, fruit packing, and table bussing using randomized object positions, combinations, and task-specific post-training.
5. Experimental Results
DreamZero generalizes more effectively than VLA baselines across unseen environments, tasks, and embodiments, while maintaining real-time inference through targeted optimizations.
- Main Results: 62.2% average task progress on AgiBot G1 exceeds the best pretrained VLA baseline’s 27.4% on unseen environments.DreamZero learns effectively from heterogeneous data, while from-scratch VLAs achieve near-zero progress.
- Unseen Tasks: 39.5% average task progress on 10 unseen tasks exceeds pretrained VLAs’ 16.3% on AgiBot G1.DreamZero also reaches 49% task progress and 22.5% success rate on DROID-Franka, versus 31% and 12.5% for GR00T N1.6.
- Post-Training Evaluation: DreamZero retains environment generalization after post-training and matches or outperforms VLA baselines across shirt folding, table bussing, and fruit packing.The reported comparison includes significant outperformance on fruit packing.
- Ablations: 50% task progress for diverse data versus 33% for repetitive data supports data diversity as a driver of generalization.The comparison uses 500 hours in each training condition.
- Ablations: 50% task progress for the 14B model versus 21% for the 5B model shows stronger scaling with model size.The smaller model is prone to visual hallucinations that propagate to erroneous actions.
- Ablations: Autoregressive WAMs produce smoother motions than bidirectional variants and infer 3–4× faster through KV caching.Task progress is similar between the architectures.
6. Discussion and Future Work
The discussion identifies unresolved limits in scaling, data breadth, computational cost, temporal horizon, precision, and embodiment design.
- Scaling Laws: Scaling laws for WAMs across model size, dataset size, and training compute remain insufficiently established.The paper leaves deeper investigation of optimal configurations for future work.
- Human Data: Human-video transfer experiments remain limited to 12 minutes of small-scale in-lab egocentric data.The paper proposes large-scale in-the-wild human video as a future direction.
- Inference: 7Hz control requires 2 GB200s and remains computationally expensive relative to VLAs reaching over 20Hz on consumer GPUs.The cost comes from DreamZero’s large parameter count and iterative video denoising.
- Long-Horizon Reasoning: DreamZero’s visual memory is short-horizon at 6 seconds, limiting robust long-horizon execution without extended context or a planner.The paper discusses modular or unified System 2 approaches as possible remedies.
- High-Precision Tasks: Tasks requiring sub-centimeter precision remain constrained because broad diverse pretraining may underrepresent dense fine-assembly demonstrations.Examples include key insertion and fine assembly.
- Embodiment Design: Higher-DOF embodiments may require more play data because their visual-future-to-motor-command mappings grow combinatorially with kinematic complexity.The accuracy of implicit inverse dynamics models remains difficult to quantify.
A. Comparison with Alternative World Model Architectures
Alternative world models typically predict forward dynamics and require separate inverse dynamics or test-time planning, whereas WAMs jointly predict visual futures and actions.
- Alternative Architectures: Latent-space and point-cloud world models predict future states in abstract or 3D representations rather than directly in pixel-space video.These representations can improve efficiency or embodiment-agnostic learning, respectively.
- Deployment Differences: Alternative approaches require separate inverse dynamics models or explicit planning and search procedures at deployment.PointWorld, for example, uses explicit optimization such as MPPI sampling to generate action trajectories.
- WAM Formulation: WAMs jointly model visual observations and actions, directly producing action trajectories aligned with predicted visual futures without test-time optimization.This formulation is expressed as p(o_t:t+H, a_t:t+H|o_0:t, c).
- Deployment Differences: 7Hz real-time closed-loop control is enabled by direct action generation while inheriting spatiotemporal priors from video pretraining.The paper contrasts this frequency with the difficulty search-based methods would face at deployment.
- Autoregressive Design: Autoregressive WAMs preserve language-video correspondence and native frame rate by conditioning on video context instead of subsampling.Bidirectional WAMs face distorted frame rates when observations arrive mid-task.
C. Model and Training details
DreamZero trains a joint video-action diffusion model with chunk-wise attention and closed-loop inference. Its implementation combines cached context, ground-truth observation updates, and systems optimizations for efficient control.
- Training: K=2 latent frames per chunk empirically outperforms K=1, with M=4 chunks used by default.Shorter trajectories may use fewer than four chunks.
- Training: DreamZero trains a joint video-action diffusion model using trajectory-level updates and attention masking over clean context from previous chunks.The model predicts noisy video and action chunks while conditioning on earlier trajectory context.
- Inference: During inference, DreamZero jointly denoises video and action chunks, then asynchronously executes the predicted action while updating the cache with real observations.The inference procedure pre-fills a KV cache, performs autoregressive denoising, executes filtered actions, and injects ground-truth observations.
- System optimization: CFG parallelism distributes conditional and unconditional score estimation across two GPUs, nearly halving diffusion-step latency without affecting model quality.The optimization addresses the computation bottleneck of DiT inference.
- System optimization: DiT caching reuses velocity predictions when successive predictions are sufficiently similar, reducing average DiT steps from 16 to 4 with minimal degradation.The cache uses a cosine-similarity threshold to skip redundant forward passes.
D.2. Implementation-level Optimizations
DreamZero-Flash decouples video and action denoising schedules, while implementation optimizations reduce inference overhead and stabilize generated action chunks. The training data emphasizes diverse environments and coarse-grained tasks rather than repetition.
- Implementation-level optimizations: Torch compilation with CUDA Graphs reduces CPU and memory overhead through full graph capture and operator fusion across five model components.Compilation is applied to the diffusion transformer, scheduler, text encoder, image encoder, and VAE.
- Implementation-level optimizations: Mixed-precision quantization uses NVFP4 for weights and activations while retaining FP8 for sensitive QKV and Softmax operations.FP16 accumulation is maintained for nonlinear operations, with negligible impact on generated quality.
- Model-level optimizations: DreamZero-Flash biases video timesteps toward higher-noise states while keeping action timesteps uniform.This exposes the model to noisy visual context during action prediction.
- Model-level optimizations: The decoupled schedule yields expected video timestep 0.125 versus 0.5 in the coupled setting.The schedule aligns training with rapid action denoising while video remains partially noisy.
- Implementation-level optimizations: Action chunks are upsampled, smoothed with a Savitzky-Golay filter, and downsampled to suppress high-frequency denoising noise while preserving trajectory shape.The filter uses window size 21 and polynomial order 3.
- Data collection: Data collection spans 22 real-world environments, prioritizing diversity over repetitive demonstrations and using coarse-grained tasks that expand as existing tasks are deprecated.Examples include homes, restaurants, supermarkets, offices, warehouses, laboratories, and hotels.
E.2. Multi-Task Episode Structure
DreamZero uses multi-task episodes to increase behavioral diversity and encourage smooth transitions between tasks. Combined with heterogeneous environments and expanding task coverage, this supports skills that transfer across tasks and environments.
- Multi-Task Episode Structure: Each episode contains three consecutive tasks, producing an average of 42 subtasks per episode.Example transitions include clearing dishes, wiping a table, and organizing condiments.
- Multi-Task Episode Structure: The three-task structure maximizes within-episode diversity and encourages the model to learn smooth task transitions.This differs from conventional single-task dataset organization.
- Multi-Task Episode Structure: The resulting heterogeneous dataset is designed to produce generalizable skills rather than narrow task-specific policies.The paper describes transfer across environments and tasks as the intended outcome.
- Multi-Task Episode Structure: Data collection covers 22 environments including homes, offices, laboratories, restaurants, supermarkets, warehouses, coffee shops, and hotels.The environment diversity is intended to support generalization to unseen environments without task-specific fine-tuning.
F. AgiBot Evaluation Details
The AgiBot evaluation reports results for both seen and unseen tasks using four robots. Each task is evaluated with two rollouts that vary objects, locations, and the robot arm.
- F. AgiBot Evaluation Details: AgiBot evaluation covers both seen tasks and unseen tasks, reported in separate tables.The setup includes initial frames and instructions for four robots.
- F. AgiBot Evaluation Details: Each task receives two rollouts varying objects, locations, and the robot arm used.The displayed evaluation mostly uses the left arm.
G. DROID Evaluation Details
The DROID evaluation includes seen and unseen task setups, with two rollouts per task that vary object locations. Tasks cover familiar instructions and tasks involving unseen verbs.
- Two rollouts per task vary object locations in the DROID evaluation.
- The DROID setup is separately identified for seen tasks and unseen tasks in evaluation tables.
- Seen tasks on DROID: Seen-task evaluation includes object manipulation instructions such as placing bowls, moving grapes, inserting bread, and transferring cups.
- Tasks with unseen verbs on DROID: The evaluation also includes tasks with unseen verbs, including moving cups or bowls, picking up pencils or markers, and relocating gloves.
H. Failure Case Analysis
DreamZero’s failures arise when generated visual plans contain incorrect task ordering or omit the requested action, and execution follows those plans. The examples indicate that improving language following and visual planning could improve action execution.
- Failure-case task set: The evaluated unseen-task instructions include actions such as revealing hidden objects, baking, frying, matching, maneuvering, and weaving.
- DreamZero generated an incorrect plan for drawing on a whiteboard, and the AgiBot execution passed the marker between arms instead of drawing.
- DreamZero generated a plan that picked up bread before opening the oven, and the DROID rollout followed this order until becoming stuck at the oven.
- The failure analysis links these execution errors to limitations in language following and visual planning capability.