$\tau_0$-WM: A Unified Video-Action World Model for Robotic Manipulation

I. INTRODUCTION

• The paper frames robotic manipulation as acting under uncertain physical consequences, requiring models to connect observations, actions, and future outcomes to improve decisions before execution [17] [40] [41] [3].Predictive capability must remain coupled to an executable interface for a particular robot embodiment, controller, and sensing stack.Source: §I · p. 1
• The framework uses heterogeneous data as complementary supervision, preserving broad visual dynamics while grounding deployment in embodiment-specific continuous actions.Egocentric and human interaction videos provide broad object and contact dynamics, while robot demonstrations provide executable control grounding.Source: §I · p. 1
• τ0-WM unifies a Video Action Model and Action-Conditioned Video Simulator around a shared video diffusion backbone.VAM predicts future visual latents and continuous action chunks, whereas ACVS predicts multi-view rollouts and dense task-progress trajectories for candidate actions.Source: §I · p. 1, Figure 1 · p. 2
• Approximately 27,300 hours of robot, UMI-style, egocentric, rollout, and failure data are jointly trained with modality-specific supervision masks.Each source contributes only to losses supported by its available observations, views, states, actions, and progress labels.Source: §I · p. 2
• At inference, τ0-WM samples action chunks, ranks them by re-denoising consistency, and uses ACVS to simulate, score, and rectify unreliable candidates before execution.The proposal–evaluation–revision procedure conditions a second VAM query on the most promising imagined future.Source: §I · pp. 2–3
• τ0-WM achieves the best average success rate among evaluated baselines on four fine-grained and long-horizon manipulation tasks across multiple embodiments.Ablations show heterogeneous pre-training and test-time computation improve performance through re-denoising selection and simulator-assisted rectification.Source: §I · p. 2

II. RELATED WORK

• Related work is organized around robotic video action models and action-conditioned video simulators, whose intersection motivates τ0-WM’s unified video-action world-modeling framework.The paper surveys both lines and their intersections.Source: §II · p. 3

A. Robotic Video Action Models

• Unlike prior VAMs that use future prediction mainly as auxiliary policy supervision, τ0-WM uses it as a unified foundation for policy learning, action evaluation, and rectification.Its VAM jointly predicts multi-view future visual latents and executable action chunks while sharing representations with the simulator.Source: A. Robotic Video Action Models · p. 3

B. Action-Conditioned Video Simulators for Robotics

• τ0-WM unifies a multi-view video action model and action-conditioned simulator, enabling candidate sampling, re-denoising ranking, and simulator-based rectification before execution.The two interfaces share a predictive representation, action interface, and backbone configuration; the simulator predicts future rollouts and task-progress scores.Source: B. Action-Conditioned Video Simulators… · p. 3, A. Robotic Video Action Models · p. 3

III. DATA SOURCES FOR PREDICTIVE ROBOT LEARNING

• 27.3K hours of training data combine 17.8K hours of real-robot teleoperation, 6.5K hours of UMI-style demonstrations, and 3.0K hours of egocentric human videos.The sources span different embodiments, viewpoints, action fidelity, collection costs, and behavioral diversity.Source: §III · p. 3
• Real-robot teleoperation supplies deployment-aligned action supervision across AGIBOT-G01, ARX, and dual-arm Franka platforms, but remains costly and limited in task and platform coverage.These demonstrations align with robot kinematics, controller interfaces, sensing stacks, and deployment conditions.Source: §III · p. 3
• UMI-style demonstrations provide scalable, diverse visual interaction data and weaker action-like supervision because handheld devices differ from target robots in embodiment and control.They are collected with lower infrastructure cost than full robot teleoperation but are only weakly aligned with deployable actions.Source: §III · p. 3
• Egocentric human videos broaden coverage of objects, contact patterns, state changes, and long-horizon behavior, but supervise only visual prediction because they lack robot-compatible action labels.Their embodiment and viewpoint differ substantially from robotic data, so they are excluded from action losses.Source: §III · p. 3
• Modality-specific supervision masks integrate heterogeneous sources into one end-to-end video-action objective while respecting which inputs, targets, and losses each sample supports.The hierarchy assigns strongest action supervision to real robots, weaker action-like signals to UMI data, and visual-dynamics supervision to human videos.Source: §III · pp. 3–4

A. Model Interface and Problem Formulation

• VAM predicts future visual latents and executable continuous action chunks from multi-view observations, language instructions, and robot state.Future visual prediction learns transferable interaction dynamics from action-free videos, while action prediction grounds representations in executable control.Source: A. Model Interface and Problem… · p. 4

B. Architecture

• VAM couples video prediction and action generation through a shared predictive representation with feature-level cross-attention.The coupling lets future visual dynamics directly support instruction-conditioned action generation.Source: B. Architecture · p. 4
• The model uses a 5B-parameter Wan video DiT backbone and a 0.5B-parameter action decoder [33], forming a 5.5B-parameter VAM.Wan2.2-TI2V-5B [39] encodes synchronized views into a temporally aligned latent canvas; current context remains clean while future slots are denoised.Source: B. Architecture · p. 4
• Action tokens model temporal dependencies before cross-attending to video features conditioned on clean visual context and language.This preserves the video backbone as a shared predictive substrate while providing instruction-aware, dynamics-relevant representations, following recent action-expert designs [29] [20].Source: B. Architecture · p. 4

C. Joint Flow-Matching Objective

• VAM applies flow matching [30] jointly to future video latents and action chunks, predicting their velocity fields from noised inputs and clean visual context.The video and action vector-field heads are coupled through intermediate video features consumed by the action branch.Source: C. Joint Flow-Matching Objective · p. 4
• Supervision masks unify heterogeneous training: robot trajectories train visual and action terms, whereas egocentric human videos train only visual dynamics.Missing modalities are masked so all data sources participate in one objective, with λz = λa = 1 in all experiments.Source: C. Joint Flow-Matching Objective · p. 4

D. Inference and Deployment

• VAM predicts executable action chunks from multi-view observations, language, and robot state, with optional future visual latents for explicit rollout visualization.Action-only deployment executes chunks receding-horizon for real-time control; rollout-enabled deployment additionally decodes predicted latents into multi-view videos.Source: D. Inference and Deployment · p. 4

A. Simulator Interface and Problem Formulation

• ACVS evaluates candidate action chunks by predicting their future visual rollouts and dense reward trajectories instead of physically executing every candidate.It treats each candidate action chunk as a clean condition and estimates the future it induces rather than serving as a policy.Source: A. Simulator Interface and Problem… · pp. 4–5
• Given memory observations, a language instruction, and a candidate action chunk, ACVS predicts imagined future latents together with a reward trajectory.The predicted quantities are denoted by z_hat for future latent rollouts and r_hat for reward trajectories.Source: A. Simulator Interface and Problem… · p. 4
• The architecture uses VAM as the policy interface and ACVS as the evaluation interface, sharing a video-generation backbone while coupling VAM’s action branch through cross-attention.ACVS reuses the backbone to roll out VAM-proposed action chunks and predict dense reward scores for test-time selection.Source: Figure 2 · p. 5

B. Architecture

• ACVS reuses the Wan VAE and video transformer backbone [39] but removes the Action DiT policy branch, denoising noisy future latent slots from encoded observation context.Memory and current observations provide clean latent context, while future slots are initialized with noise.Source: B. Architecture · p. 5
• Temporally aligned candidate actions are grouped into blocks and projected through lightweight MLPs following the action-conditioned design of Cosmos [2].The resulting conditions enter diffusion-time and AdaLN modulation embeddings, then broadcast across spatial tokens and camera views for each future slot.Source: B. Architecture · p. 5
• Unlike VAM, ACVS generates no actions; it estimates how different proposed sequences would evolve the scene under the same observation and instruction.Observation slots remain unconditioned while action conditions affect corresponding future slots.Source: B. Architecture · p. 5

C. Reward and Progress Scoring

• Dense reward trajectories are generated by propagating subtask-level progress labels through each task segment, replacing terminal-only success supervision.Frame-level rewards are estimated with Monte Carlo propagation within subtask segments.Source: C. Reward and Progress Scoring · p. 5
• Negative rewards on failed segments teach ACVS to reject unsuccessful contact, incorrect object motion, and task-regressing actions despite visually plausible motion.Source: C. Reward and Progress Scoring · p. 5
• Failure-heavy and recovery trajectories improve simulator fidelity by exposing ACVS to off-distribution actions and interactions unavailable in successful demonstrations.These trajectories are useful for simulator learning even when they are suboptimal as direct policy supervision.Source: C. Reward and Progress Scoring · p. 5

D. Training Objective

• ACVS jointly supervises future video latents and dense reward trajectories using the same flow-matching formulation as VAM.The construction noises future latent rollouts and target reward trajectories at specified noise levels and learns their velocity targets.Source: D. Training Objective · pp. 5–6
• At test time, the procedure samples N candidate actions, ranks them with re-denoising consistency, and returns the top candidate when its score exceeds threshold γ.Source: D. Training Objective · p. 6
• Candidates that do not meet the re-denoising-consistency threshold are evaluated with ACVS to compute rollout values J(i).Source: D. Training Objective · p. 6
• The reward head consumes action-conditioned video features while separate predictors model video and reward velocities, with λ_z = λ_r = 1 in all experiments.Source: D. Training Objective · p. 6

VI. TEST-TIME COMPUTATION

• The deployment problem arises because heterogeneous pre-training yields multimodal conditional action distributions with feasible solutions differing in precision, robustness, and success likelihood.Test-time computation therefore focuses on selecting a high-quality action among multiple feasible sequences.Source: §VI · p. 6
• The coarse-to-fine procedure samples multiple VAM action candidates, filters them for reliability, and invokes ACVS only when candidates appear unreliable.This preserves real-time performance in most situations while retaining recovery for difficult states.Source: §VI · p. 6
• When selected actions are low quality, Algorithm 2 converts the selected future latent into conditioning, re-queries VAM, and generates a refined action chunk.The rectification sequence uses the current observations, prompt, and state together with the selected future condition.Source: §VI · p. 6

A. Re-denoising Consistency Score

• VAM samples N candidate action chunks from the current observation, language, and robot-state context.The candidates are denoted {ā(i)}N in the described procedure.Source: A. Re-denoising Consistency Score · p. 6
• For each candidate, the method samples K flow timesteps, re-noises the action, and evaluates it with VAM’s action vector field to obtain average error E(i).The re-noising process matches the flow-matching procedure used during training.Source: A. Re-denoising Consistency Score · p. 6
• The Re-denoising Consistency Score uses the average re-denoising error to rank candidate actions.RCS is introduced as the score derived from the candidate-specific error E(i).Source: A. Re-denoising Consistency Score · p. 6
• RCS favors candidates consistent with the learned conditional action manifold while adding negligible overhead compared with rollout-based evaluation.This makes RCS a lightweight distributional filter for deployment-time action selection.Source: A. Re-denoising Consistency Score · p. 6

• LAR is introduced because RCS may select a self-consistent candidate that remains poor in challenging states.Source: B. Low-quality Action Rectification · p. 6
• When the selected candidate is unreliable, ACVS evaluates all candidate action chunks by predicting imagined rollouts and dense reward trajectories.Invocation is controlled by a reliability threshold γ.Source: B. Low-quality Action Rectification · p. 6
• The candidate with the highest rollout value, defined by maximum task progress, is selected as the most promising future.Source: B. Low-quality Action Rectification · p. 6
• Rather than executing that action directly, a second VAM policy query injects the selected rollout latent as an additional future condition to generate a refined action chunk.This conditions the policy explicitly toward the selected high-value future; the procedure is summarized in Algorithm 2.Source: B. Low-quality Action Rectification · p. 7

VII. EXPERIMENTAL EVALUATION

• The experiments test whether heterogeneous robot, UMI-style, and egocentric interaction pre-training improves downstream manipulation performance.Data-composition ablations include zero-shot and supervised fine-tuning evaluation of ego and UMI pre-training recipes.Source: §VII · p. 7, Table I · p. 8
• The study evaluates deployment-time computation by comparing τ0-WM’s action selection and rectification strategy with standard execution, classifier-free guidance (CFG) [18], and Action Coherence Guidance (ACG) [32].Test-time computation is assessed through one-time completion without retries, with success rates reported per task and averaged across tasks.Source: §VII · p. 7, Table II · p. 8
• The evaluation spans AGIBOT-G01, ARX manipulators, and a dual-arm Franka system on language-conditioned, multi-view packing and assembly tasks.Badminton uses ARX, Faucet uses the dual-arm Franka platform, and the remaining tasks use AGIBOT-G01.Source: §VII · p. 7, A. Main Results · p. 8

A. Main Results

• τ0-WM achieves the highest average success rate and performs best on most of four tasks across multiple robot embodiments.Models are evaluated using both task success rate and stepwise task accomplishment progress; Faucet remains challenging for every method, while τ0-WM achieves the highest success rate under strict alignment constraints.Source: A. Main Results · p. 8, Figure 4 · p. 8
• τ0-WM remains consistently strong on longer-horizon, fine-grained tasks, whereas π0.5 is competitive on Toolbox but degrades on tasks requiring extended coordination and precise manipulation.The experiments compare τ0-WM with representative policy and video-action baselines, including π0.5 and Fast-WAM [45].Source: §VII · p. 7, A. Main Results · p. 8
• τ0-WM often performs corrective actions after apparent completion, such as pushing or pressing a tool further into place, unlike baselines that may stop at incomplete insertion.These qualitative differences are not fully captured by the binary success metric and suggest optimization toward final scene quality.Source: A. Main Results · p. 8

B. Ablation Studies

• Adding UMI and egocentric data raises zero-shot average success from 0.14 to 0.55 and retains benefits after supervised fine-tuning, especially in clutter.The results support improved general-purpose manipulation priors, visual understanding, and robustness rather than merely faster adaptation.Source: B. Ablation Studies · pp. 8–9
• The test-time computation ablation uses Tissue→Box and Pen→Box tasks under a single-attempt protocol repeated 20 times, with four proposals per decision step by default.The strategy combines lightweight RCS selection with LAR-based ACVS rectification when the selected action is unreliable.Source: B. Ablation Studies · p. 9
• Average success rises from 0.43 to 0.50 with RCS and to 0.60 with RCS+LAR, with RCS+LAR best across all tested tasks.RCS filters suboptimal samples, while LAR uses action-conditioned rollouts for rectification; improvements are especially large on Pen→Box.Source: B. Ablation Studies · p. 9

VIII. CONCLUSION AND FUTURE WORK

• τ0-WM combines action generation, future prediction, and deployment-time reasoning within one unified video-action world model.Its training incorporates real-robot teleoperation, UMI-style demonstrations, egocentric human videos, and simulator-oriented rollout trajectories.Source: §VIII · p. 9
• Experiments demonstrate strong policy performance across robot embodiments, consistent heterogeneous-pretraining gains, and significant test-time-computation improvements on long-horizon, fine-grained manipulation.These findings summarize the reported scope of the model’s empirical benefits.Source: §VIII · p. 9
• Future work includes tactile and other sensing modalities for contact-rich manipulation, more reliable deployment-time reasoning, and longer-horizon predictive modeling.Suggested reasoning improvements include better uncertainty estimation, longer-horizon evaluation, and more effective search for difficult states; target tasks include insertion, fastening, and deformable-object manipulation.Source: §VIII · p. 9
• τ0-WM is presented as a practical step toward more capable and reliable robot foundation models through predictive robot learning.The conclusion frames predictive learning as a promising path for advancing robot foundation models.Source: §VIII · p. 9

A. Training and Deploymeny Details

• τ0-WM is pretrained on 27.3K hours of heterogeneous interaction data and then adapted to downstream robotic manipulation tasks.The data include real-robot teleoperation, UMI-style interactions, egocentric human videos, and rollout or failure trajectories.Source: 1) Training Configuration · p. 10

1) Training Configuration:

• Training uses global batch sizes of 12,288 and 384 for pre-training and post-training, respectively, with AdamW at a 5 × 10−5 learning rate.The same optimization hyperparameters are used across embodiments and tasks unless otherwise specified.Source: 1) Training Configuration · p. 10

2) Deployment Details:

• Real-robot experiments use language-conditioned multi-view observations and receding-horizon control with fixed-length action chunks of 30.This is the default execution setup unless otherwise stated.Source: 2) Deployment Details · p. 10
• Real-robot inference runs on a single RTX 5090 GPU with approximately 220 ms latency per query, reduced to 180 ms by caching reusable text representations without changing outputs.The reduced-latency configuration preserves model outputs.Source: 2) Deployment Details · p. 10

B. Inference Acceleration

• The inference pipeline applies implementation-level optimizations to improve deployment efficiency.These optimizations are described under inference acceleration.Source: B. Inference Acceleration · p. 10

1) Cross-Attention KV Cache:

• During action-only inference, video-conditioned cross-attention key and value tensors are computed once and reused across all denoising steps.Because the visual context remains unchanged, this caching eliminates redundant projection operations.Source: 1) Cross-Attention KV Cache · p. 10
• The cross-attention KV cache reuses tensors derived from the video feature x(l) at each transformer layer throughout inference.The supplied passage introduces x(l) as the video feature at transformer layer l, while the accompanying description states that the cached tensors are reused during inference.Source: 1) Cross-Attention KV Cache · p. 10

2) Fused QKV Projection:

• The model fuses query, key, and value projections into a single matrix multiplication.This replaces three independent projection layers.Source: 2) Fused QKV Projection · p. 10
• Fused QKV projection reduces kernel-launch overhead and improves memory throughput.The improvement is relative to three independent projection layers.Source: 2) Fused QKV Projection · p. 10

3) Simplified Rotary Position Embedding:

• Precomputed one-dimensional rotary position embeddings are reused throughout deployment for the one-dimensional temporal action-token sequence.This avoids repeated frequency construction.Source: 3) Simplified Rotary Position Embedding · p. 10
• Reusing rotary frequencies reduces positional encoding overhead during deployment.The optimization applies because action tokens form a one-dimensional temporal sequence.Source: 3) Simplified Rotary Position Embedding · p. 10

4) Torch Compile Optimization:

• Deployment latency decreases from approximately 180 ms to 140 ms when torch.compile is combined with the preceding optimizations.Per-block compilation is used to further reduce latency.Source: 4) Torch Compile Optimization · p. 10
• Main-paper results omit torch.compile because compiler graph transformations and kernel fusion can introduce numerical differences that alter diffusion sampling outputs.The omission ensures consistency and reproducibility across experiments.Source: 4) Torch Compile Optimization · p. 10