Source-linked AI summary
ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training
Xionghao Wu, Yijun Yang, Shiyang Zhou, Haoze Sun, Jianhui Liu, Songsong Yu, Jiyao Zhang, Wenbo Li, Bo Wang, Guoqing Ma, Lin Song, Renjie Liao, Shenghe Zheng, Wei Tang, Xiaojuan Qi, Yanwei Li, Yuan Zhang, Zhuotao Tian, Haoyang Huang, Nan Duan
TL;DR
Robotic manipulation needs broad physical experience for generalization, but action-labeled robot trajectories are costly and limited, motivating scalable use of action-free egocentric video. ZimaBlue trains a World Action Model through video pre-training, cross-embodiment action alignment, and target-robot specialization, then uses a Slow-Fast system for real-time control. In zero-shot real-robot evaluations, scaling to 120,000 hours of video raises success from 36.1% to 77.8%, with strong benchmark performance and especially pronounced gains on unseen tasks.
Problem
Action-labeled robot trajectories are expensive and limited in diversity, while scalable egocentric videos lack action labels needed for standard robot-control training.
Method
ZimaBlue combines causal embodied video pre-training, multi-embodiment video-action mid-training with unified state-action representation, target-robot post-training, and an asynchronous Slow-Fast control architecture.
Results
77.8% zero-shot success is achieved with 120,000 hours of video, compared with 36.1% from target-robot post-training alone, alongside strong performance across manipulation benchmarks.
Takeaways & Limitations
Large-scale embodied video provides an effective and economical scaling axis for generalizable robot learning, particularly when task and environment distributions shift.
Takeaways & Limitations
On Perturbed air-fryer tasks, the full configuration achieves only 5/10–7/10 despite reaching 87.9% on Standard, indicating remaining progress and local interaction failures.
Abstract
from arXiv · showhide
Robotic manipulation faces a fundamental scaling challenge: robust generalization demands broad physical experience, yet action-labeled robot trajectories are expensive to collect and inherently limited in diversity. Egocentric videos offer a far more scalable source of embodied experience, capturing object interactions, contact dynamics, tool use, and long-horizon behaviors across diverse environments. The central challenge is how to convert this abundant but action-free experience into effective robot control. We introduce ZimaBlue, a scalable framework for learning generalizable World Action Models (WAMs) from large-scale video. ZimaBlue follows a three-stage training curriculum: it first performs causal embodied video pre-training on large-scale human and robot egocentric videos, then grounds the learned visual dynamics in heterogeneous robot trajectories through video-action mid-training with a unified action representation, and finally specializes the model to a target robot for deployment. To make generative WAMs practical for real-time control, ZimaBluefurther adopts an asynchronous Slow-Fast dual-system architecture, where a high-capacity Slow world model provides generalizable spatiotemporal representations and a lightweight Fast branch enables 30 Hz action prediction on NVIDIA RTX 4090. On real-robot zero-shot evaluations, scaling from target-robot data alone to over 120,000 hours of embodied video improves success from 36.1% to 77.8%. ZimaBlue further delivers strong performance across multiple benchmarks, with particularly pronounced gains on unseen tasks.
1 Introduction
ZimaBlue addresses limited action-labeled robot data by scaling causal embodied video pre-training, aligning it with heterogeneous robot trajectories, and specializing the resulting model to a target robot. Its zero-shot success rises as training data expands, while a Slow-Fast design supports real-time closed-loop control.
- Motivation: Egocentric videos provide scalable embodied experience covering affordances, tool use, contact events, failure recovery, and long-horizon task structure without action labels.WAMs can first learn causal visual dynamics from video, then align them with robot states and actions using less action-labeled data.
- Approach: ZimaBlue uses a three-stage curriculum: causal embodied video pre-training, multi-embodiment video-action mid-training, and target-robot post-training.A unified action representation standardizes states and actions across heterogeneous platforms.
- Scaling results: 77.8% zero-shot success is achieved with 120,000 hours of video, up from 36.1% with target-robot post-training alone.Intermediate configurations reach 46.1% with 6,000 hours and 66.9% with 60,000 hours of video.
- Benchmark evaluation: ZimaBlue improves generalization on unseen manipulation tasks and performs strongly across LIBERO-Plus, RoboTwin 2.0, and RoboCasa365.On RoboCasa365, it is strongest among compared methods except Xiaomi-Robotics-1, while gains are especially pronounced on unseen tasks.
- Real-time control: The Slow-Fast architecture combines a high-capacity world model for generalization with a lightweight action module for faster control.Diffusion-step distillation and CUDA graph optimization contribute to a 33 ms control loop on NVIDIA RTX 4090.
2 Related Work
Related work spans vision-language-action policies, world models, joint video-action models, and dual-system architectures. These approaches motivate ZimaBlue by exposing limits in physical dynamics modeling, computational efficiency, and coordination between predictive reasoning and reactive control.
- Vision-language-action models: VLA models unify language-conditioned robot control across heterogeneous demonstrations but remain primarily action-centric.Their downstream control is typically learned from comparatively limited robot trajectories.
- World models: World models predict environment dynamics using latent, 3D, pixel, or video-space representations for planning, reinforcement learning, or future-observation prediction.Video-based models directly predict future visual observations.
- Video-action models: Imagine-then-act systems generate future visual states before recovering actions, but can suffer open-loop drift, observation mismatch, and additional test-time latency.Joint video-action architectures instead model visual dynamics and actions within one generative architecture.
- Deployment challenges: Existing WAMs face deployment challenges because video generation is computationally expensive and visual prediction and motor correction operate at different timescales.Some architectures bind both processes to the same short temporal horizon, limiting control frequency.
- Dual-system architectures: Dual-system architectures pair a slower expressive pathway with a faster reactive pathway for real-time robotic control.AHA-WAM applies this principle through low-frequency video planning, high-frequency action execution, and reused planner context.
3 Model Architecture
ZimaBlue uses a unified semantic state–action interface to align heterogeneous robot embodiments, then separates high-capacity world modeling from low-latency action generation through a Slow-Fast architecture.
- Unified Representation: A 100-dimensional semantic state–action space maps incompatible robot control interfaces into physically consistent coordinates across embodiments.State describes the current configuration, while action represents a future control chunk.
- Unified Representation: Each embodiment activates only its native-interface slots, while undefined coordinates are zero-filled and excluded using validity masks.For example, DROID activates 17 valid coordinates covering one end-effector, gripper, and seven arm joints.
- Unified Representation: End-effector actions are represented relative to the proprioceptive state at the first frame of each action chunk rather than in absolute coordinates.Relative translation and rotation are defined from the chunk anchor, and joint targets use the same relative principle.
- Unified Representation: Relative translation is normalized using robust percentile bounds, while the 6D rotation representation remains unscaled and is restored to physical units at deployment.The inverse transformation is applied before relative targets are composed with the current robot state.
- Slow-Fast Dual-System: The 5B Slow DiT models causal video dynamics, while the 0.5B Fast DiT uses Slow-branch representations with current observations and state to predict control actions.This separates high-capacity world modeling from high-frequency action generation.
- Slow-Fast Dual-System: The Fast branch cross-attends to layer-wise video K/V caches computed by the Slow branch, bridging visual dynamics representations with reactive action prediction.By default, the Fast tower uses caches from the first 12 Slow layers and directly generates final deployment actions.
4 Training Pipeline
ZimaBlue uses a three-stage data pyramid that progresses from broad, action-free embodied video learning to cross-embodiment action grounding and target-robot specialization. Its video-centric curriculum uses heterogeneous views and causal training to connect future visual dynamics with robot actions.
- Stage I: Video Pre-training: Stage I adapts a video-generation prior into causal embodied dynamics using heterogeneous human and robot videos without action annotations.The curriculum includes broad video pre-training followed by a curated manipulation-centric continuation.
- Stage III: Post-training: Stage III specializes the aligned representations for target embodiments and deployment benchmarks.The data pyramid progressively adapts broad visual knowledge toward deployment-specific embodiment.
- Data Interface: The unified three-view canvas preserves multi-view robot observations while allowing single-view human videos to share the same visual interface.Missing views are padded, and available views are placed in canonical rows.
- Training Objective: Causal video training uses clean context, noisy future blocks, validity masks, and block-causal teacher forcing to train parallel future prediction without removing rollout dependencies.Action supervision is disabled during video pre-training, while temporal and view validity determine which visual tokens contribute to the loss.
- Stage II: Video-Action Mid-training: Stage II jointly predicts future video latents and action chunks from heterogeneous robot trajectories, grounding visual dynamics in cross-embodiment control.The training interface standardizes multiple embodiments while preserving embodiment-specific sensorimotor modalities.
- Training Objective: The world-action objective couples visual and action prediction so the model learns visual transitions together with the controls required to produce them.At inference, clean target video and action tokens are omitted and their noisy counterparts are progressively denoised.
5 Acceleration Schemes
ZimaBlue combines asynchronous Slow-Fast inference with diffusion step distillation to make generative world-action models suitable for high-frequency control. Slow preserves long-horizon visual-action modeling, while Fast refines actions using current observations and cached Slow guidance.
- Asynchronous Slow-Fast inference: Slow updates long-horizon world-model guidance asynchronously, while Fast generates high-frequency actions without waiting for each Slow rollout.The concurrent streams let Fast continuously refine actions while receiving updated Slow guidance.
- Asynchronous Slow-Fast inference: The Slow stream rolls out future visual dynamics from observations, proprioception, and language, exporting layer-wise video K/V caches for repeated Fast requests.Fast accesses updated caches without interrupting the ongoing control loop.
- Asynchronous Slow-Fast inference: Fast predicts an action chunk from the latest observation, state, Slow cache, and noisy action chunk, replacing only future unexecuted actions during updates.Already committed actions remain unchanged during asynchronous replacement.
- Asynchronous Slow-Fast inference: RTC anchors consecutive Fast predictions with a prefix from the previous action sequence and predicts a corrected suffix under new observations and Slow guidance.The prefix supports continuity, while the suffix enables online correction.
- Diffusion step distillation: DMD distills the task-specialized Slow branch and then the Fast action branch from eight to two DiT evaluations per rollout or request.The Slow distillation preserves joint future-video and action prediction before the Fast branch is distilled with frozen Slow K/V guidance.
- Diffusion step distillation: DMD estimates normalized teacher–student distribution differences using valid dimensions and stop-gradient targets, while two-time-scale optimization updates the fake-score model more frequently.The two-stage procedure retains coupling between predicted visual dynamics and actions; deployment keeps only the distilled Slow and Fast students.
6 Experiments
The real-robot experiments evaluate ZimaBlue variants on zero-shot Standard and Perturbed manipulation tasks under fixed deployment and measurement conditions. Scaling heterogeneous robot and egocentric video data produces a monotonic increase in suite-level success, reaching 77.8% for the full model.
- Evaluation setting: 12 zero-shot real-robot tasks are divided into eight Standard tasks and four Perturbed tasks to test individual video-pretraining contributions.The Perturbed suite introduces dynamic lighting, background distractors, and novel tabletop appearances.
- Evaluation setting: Experiments use a 7-DoF Franka arm with DROID post-training and held-out task–scene configurations without task-specific demonstrations.Deployment receives two external RGB views, one wrist-camera view, proprioception, and language.
- Evaluation setting: Each task is evaluated over 10 trials using binary success, except Toys, which contributes 30 placement outcomes; suite scores are task-macro averages.Camera placement, instructions, resets, decoding, and safety limits are fixed across methods.
- Scaling comparison: The comparison scales from a baseline through multi-embodiment video-action data and 60K or 120K hours of egocentric video under identical DROID post-training.All variants use the same reported Slow and Fast training schedules.
- Results: 77.8% overall success is achieved by the full model, rising monotonically from 36.1% for the Baseline configuration.The full model reaches 87.9% on Standard and 57.5% on Perturbed suites, outperforming π0.5 and DreamZero by a large margin.
- Task examples: The listed task labels include Cup Selection, Bread Transfer, Block Stacking, Toy Collection, Microwave Closing, and Toaster Activation.These labels appear among the real-robot suite materials.
1. Bowl Stacking
The supplied passages contain fragmented task-label text associated with manipulation evaluations, including cup selection, bread transfer, block stacking, and placing items on a plate.
- Task labels: Bread Transfer is described with moving bread from a toaster to a plate.The text provides the task action but no performance result.
5. Air-fryer Opening
The supplied passages identify toaster activation and bowl stacking among the real-robot task labels, with Standard-suite figure context but no task-specific results.
- Task labels: The task materials separately describe pushing down the toaster switch.This fragment specifies an interaction but gives no outcome value.
- Figure context: The figure context identifies a Standard real-robot suite.The supplied fragment does not state a result for the depicted tasks.
- Task labels: Bowl Stacking appears in the listed task labels alongside Bread Placement and asterisk-marked Toaster Activation.No comparison or task-specific result is provided.
9. Air-fryer Opening∗
The evaluation compares ZimaBlue across standard and visually perturbed real-robot suites, benchmark settings, and architectural variants. Results show strong performance gains from scaled video pre-training, dual-system control, and broader benchmark evaluation, while perturbed tasks remain more difficult.
- Task-level gains: 9/10 air-fryer-opening success is reached after adding multi-embodiment action-labeled data, compared with 6/10 previously.The largest gains from cross-embodiment supervision occur on contact-rich articulated-object tasks.
- Video scaling: 22.5 percentage points is the Perturbed-suite gain when video pre-training scales from 60K to 120K hours.The same scaling produces a 5.0-point Standard-suite gain, with improvements across all four Perturbed tasks.
- Dual-System: 57.5% Perturbed task-macro success is obtained with the Dual-System, up from 30.0% for the Slow-only system.The improvement is associated with tasks requiring frequent visual feedback and recovery from changing scene states.
- Acceleration: 33.0 ms latency and 75.0% overall task-macro success are achieved after distillation and compilation.The accelerated model is 13.6× faster overall, while retaining performance within 2.8 percentage points of the undistilled Dual-System.
- Simulation benchmarks: 92.0% LIBERO-Plus success is achieved with task-specific fine-tuning, while zero-shot performance reaches 86.7%.Camera viewpoint changes remain a key bottleneck in the zero-shot setting, improving from 58.1% to 95.4% with fine-tuning.
7 Conclusion and Future Work
ZimaBlue frames embodied video as a scalable source for generalizable robot learning and combines it with cross-embodiment alignment, target-robot adaptation, and decoupled reasoning and control. Future work emphasizes broader evaluation, larger data and models, stronger reasoning, and in-context skill acquisition.
- Conclusion: ZimaBlue scales generalizable robot learning through egocentric video, cross-embodiment video-action alignment, and target-robot post-training.Its Slow-Fast dual system separates high-level world reasoning from real-time closed-loop control.
- Future work: 120,000 hours of video are described as only a glimpse of robotics scaling potential.Proposed directions include richer benchmarks, larger video and model scales, stronger reasoning, and in-context learning.
8 Authors
The paper identifies its authors and describes the real-robot task suites and shared evaluation protocol. Target-robot adaptation freezes the Slow System before training the Fast System across the ZimaBlue variants.
- Authors: The author list includes Xionghao Wu, Yijun Yang, Shiyang Zhou, Haoze Sun, Jianhui Liu, Songsong Yu, Jiyao Zhang, and Wenbo Li.The contribution listing assigns roles across data, pre-training, mid-training, post-training, dual-system design, acceleration, deployment, guidance, and project leadership.
- Task protocol: 12 real-robot tasks are held out from DROID post-training, with end-of-episode success required except for per-object Toys scoring.The task definitions cover stacking, transport, appliances, and toy placement.
- Task protocol: The Perturbed suite evaluates appliance tasks under glare and flashing illumination and bread/bowl tasks under tablecloth backgrounds with unseen clutter.These conditions are distinct from the controlled Standard suite.
- Adaptation: The Slow System is post-trained first and then frozen while the Fast System is trained for target-robot deployment.This two-stage adaptation is shared across the four pre-training initializations.
A.5 Visually Perturbed Suite Results
The Visually Perturbed suite isolates robustness to controlled environmental shifts, including unseen scenes, clutter, distractors, glare, flashing lights, and changed appliance location. Results are reported as successful rollouts over ten attempts for competing methods and ZimaBlue variants.
- Perturbed conditions: Four Perturbed tasks group evaluation by unseen visual conditions affecting appliances, bread, and bowls.Bread and bowl tasks combine an unseen scene, cluttered background, and unseen distractor objects; appliance tasks add dynamic illumination and clutter.
- Evaluation results: Each Table 10 entry reports successful rollouts over ten attempts for π0.5, DreamZero, and four ZimaBlue variants.VA, V+VA, and SV+VA denote progressively expanded video-action pre-training configurations.
- Evaluation protocol: The Visually Perturbed score is reported separately from the Standard score to isolate robustness under controlled visual shifts.Failure analysis distinguishes instruction or perception errors, grasp and contact failures, accumulated pose error, and other task failures.
B.1 LIBERO-Plus Detailed Analysis
The LIBERO-Plus analysis evaluates ZimaBlue across four suites, seven perturbation categories, and 10,030 episodes, combining aggregate results with fine-grained task and attention analyses. SFT improves suite-level performance overall, while category-level trade-offs and the Slow-Fast attention design clarify remaining robustness boundaries.
- Evaluation setup: Evaluation covers 10,030 episodes across four suites and seven perturbation categories, with category-macro scores computed as unweighted means across pooled episodes.Robot Initial States follows the official no-restore protocol.
- Suite-level results: SFT improves all four LIBERO suites by 3.7–7.5 points, with the largest gain on LIBERO-10.Goal remains the hardest suite at 87.1%, so aggregate gains do not remove goal-conditioned execution difficulty.
- Perturbation analysis: Camera gains are consistent across suites, while Robot Initial States declines in every suite and Layout changes vary by suite.The largest camera gain is on LIBERO-10 (+45.8 points), whereas the largest Robot Initial States decline is on Object (-16.8 points).
- Cross-benchmark reporting: Per-task success rates are reported for RoboTwin 2.0 and RoboCasa365, with RoboTwin evaluated in Clean and Randomized settings and RoboCasa365 across 50 tasks.RoboTwin tasks use 100 episodes per setting; RoboCasa365 tasks use 50 episodes.
- Slow-Fast attention: The Slow DiT uses block-causal attention for parallel teacher-forced transition training, while the Fast DiT fuses current observations, state, action chunks, and a frozen Slow video cache.Fast action queries jointly denoise one action horizon with non-causal attention within each request.