VideoVLA: Video Generators Can Be Generalizable Robot Manipulators

1 Introduction

• Despite data-efficient, high-performing VLA models built on pre-trained understanding models [2] [3] [4], generalization to unseen tasks, objects, and environments remains limited.The introduction frames such generalization as important for open-world deployment and a step toward artificial general intelligence.Source: §1 · p. 2
• Video generators’ plausible predictions from novel text or images, learned physical dynamics, and ability to forecast instructed future states motivate adapting them for manipulation.The paper relates those abilities to handling unseen instructions and observations, reasoning about action consequences, and planning interactions.Source: §1 · p. 2
• VideoVLA transforms a Video Diffusion Transformer into a multimodal model that jointly predicts executable robot actions and the future videos depicting their consequences.It conditions on language tokens and the current visual observation, represents actions as 7-D vectors, and trains with a DDPM diffusion loss.Source: §1 · p. 2
• When generated future videos closely match environmental outcomes, VideoVLA’s predicted actions consistently achieve higher task success, linking visual-imagination quality with action reliability.The paper argues this supports jointly predicting future visual dynamics alongside actions.Source: §1 · pp. 2–3
• VideoVLA shows promising generalization to novel objects and skills from other embodiments, attributed to pre-trained video generation and its dual-prediction strategy.The paper contrasts this paradigm with prior VLA work that primarily relies on pre-trained understanding models.Source: §1 · p. 3

Vision-Language-Action Models

• VideoVLA departs from understanding-oriented VLAs by leveraging pretrained video generators to jointly predict future actions and visual outcomes.Prior VLAs combine language understanding, environmental perception, and robot control, supported by foundation models and large-scale datasets such as Open X-Embodiment [2].Source: §2 · p. 3
• VideoVLA exploits large-scale video pretraining, generalizes to novel objects and unseen skills, links action predictions with imagined outcomes, and competes with π0 and CogACT [4].The reported comparison spans simulated and real-world settings; UVA [36] and VPP [37] are identified as initial steps toward this paradigm.Source: §2 · p. 3

Video Generation Models

• Modern video generators commonly encode videos into latents with a VAE and train a diffusion transformer to model their spatiotemporal distribution; VideoVLA builds on CogVideoX [24].The authors describe VideoVLA as the first adaptation of a large-scale video generator into a generalizable robotic manipulator for visual planning and action prediction.Source: §2 · p. 3

Video Generation for Robot Manipulation

• Prior robot-manipulation methods use generated video modularly for inverse dynamics, optical-flow actions, goal-frame conditioning, human-to-robot transfer, feature-based action generation, or autoregressive frame-and-action prediction [45] [46] [47] [48] [49] [51] [52] [53] [54] [6].In contrast, VideoVLA jointly predicts video and actions in one end-to-end transformer adapted from a pretrained video generator.Source: §2 · pp. 3–4

3.1 Problem Formulation

• VideoVLA predicts a sequential chunk of 7-D wrist rotation, translation, and gripper actions together with future frames, then repeats prediction from each updated observation until task completion.The action and video frequencies may differ, so one action can correspond to multiple frames.Source: §3.1 · p. 4

3.2 Overview

• Two encoders represent language and video as tokens or latents, and a DiT backbone uses the instruction and current-observation latent to jointly predict actions and future-frame latents.The video decoder is optional and is used only to visualize imagined future frames.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Data Preprocessing

• VideoVLA processes inputs in latent space; its causal CogVideoX video encoder makes the first latent represent the current observation, using that observation alone at inference and the full clip during training.T5 encodes each instruction into 226 tokens; the video encoder temporally downsamples clips into frame latents.Source: §3.3 · p. 4

3.4 Unified Future Modeling

• A DiT-style backbone concatenates conditioning and targets into a multimodal sequence, then uses DDPM diffusion loss to denoise future-video latents and actions jointly.Modalities are projected to a common embedding dimension, timestep embeddings enter through adaptive LayerNorm, and the backbone is initialized from pre-trained CogVideoX.Source: §3.4 · pp. 4–5

4.1 Simulation Experiments

• VideoVLA ranks highest on three reported SIMPLER averages, including the overall average across 12 tasks, and achieves the highest average success rate among the evaluated models in real-world in-domain tasks.In simulation it is highest on VM-WidowX, VA-Google, and the overall average, and second-highest on VM-Google; the real-world tasks include distractor objects.Source: §4.1 · p. 6, Table 1 · p. 6, §4.2 · p. 7
• On eight Google-robot skills present in WidowX training but absent from Google training, VideoVLA exceeds the second-best model, CogACT [4], by 28.2 points in average success rate.VideoVLA also achieves superior performance across all evaluated skills.Source: §4.1 · p. 7, Table 3 · p. 7

4.2 Real-World Experiments

• For novel objects, VideoVLA has the highest average success rate in simulation, outperforming prior models on eight of ten objects, and handles all 12 real-world novel objects with success rates from 16.7% to 83.3%.The real-world task is picking up each novel object and placing it on a specified-color plate.Source: Table 2 · p. 6, §4.1 · p. 7, §4.2 · p. 8
• On five skills observed in WidowX training but not demonstrated by Realman, VideoVLA achieves the highest success rates across all evaluated skills, including “Topple” and “Wipe.”The objects are familiar to Realman; the skills themselves are novel to that embodiment.Source: §4.2 · p. 8, Table 6 · p. 8

4.3 Ablation Study

• The ablations associate higher-quality pretrained video generation with better manipulation success and show that performance improves consistently as the predicted future horizon increases from 13 to 49 frames.Training from scratch without large-scale general-video pretraining yields a dramatically lower success rate; the horizon settings correspond to 4, 7, and 13 video latents.Source: §4.3 · pp. 8–9, Table 7 · p. 8, Table 8 · p. 8
• Removing video denoising loss or eliminating video prediction substantially reduces performance across tasks, supporting the dual-prediction strategy’s importance for robust execution and generalization.The compared variants are joint video-and-action prediction with losses on both modalities, joint prediction with action-only loss, and action-only prediction.Source: §4.3 · p. 9, Table 9 · p. 9

4.4 Imagination-Execution Correlation Analysis

• Higher motion similarity between imagined and executed robot trajectories corresponds to a higher likelihood of task success; imagined trajectories succeed at 84.0% on novel objects and 63.4% on new skills, versus 65.2% and 48.6% for actual execution.Human judges assess imagined outcomes for semantic instruction-following and absence of salient geometric distortions or physical implausibility; the paper attributes the execution gap to physical grounding, actuation noise, and perception errors.Source: §4.4 · pp. 9–10, Table 10 · p. 9, Figure 3 · p. 10

5 Conclusion

• VideoVLA jointly predicts actions to execute and generates visual imaginations of their anticipated outcomes using large pre-trained video generators.The framework leverages video generators rather than relying primarily on large perception or vision-language models.Source: §5 · p. 10
• High-quality visual predictions are consistently associated with reliable action plans and task success.The experiments reveal a strong correlation between imagined futures and actual execution.Source: §5 · p. 10
• VideoVLA performs strongly on seen objects and learned skills, and generalizes to novel objects and cross-embodiment skill transfer in simulated and real-world environments.The paper attributes these capabilities to large-scale pre-trained video generation and the dual-prediction strategy.Source: §5 · p. 10
• The findings highlight the potential of generative video models as a scalable foundation for general-purpose robot manipulation.Source: §5 · p. 10

A Evaluation Details

• Each model, including VideoVLA and the baselines, follows a consistent evaluation protocol.The appendix reports the number of trials for each experiment in both simulation and real-world evaluations.Source: A Evaluation Details · p. 15
• Table 11 lists trial counts for simulation experiments conducted in the SIMPLER environment [55].The listed experiments cover Google robot VM and VA, WidowX robot VM, and Google robot generalization.Source: A Evaluation Details · p. 15, Table 11 · p. 15
• Google robot simulation trials include SIMPLER Visual Matching (VM) and Variant Aggregation (VA) experiments.These are among the simulation experiments listed in Table 11.Source: A Evaluation Details · p. 15
• WidowX robot simulation trials cover SIMPLER Visual Matching (VM).This experiment is included in the simulation trial counts in Table 11.Source: A Evaluation Details · p. 15, Table 11 · p. 15
• The simulation trial counts also include Google robot generalization experiments.These are listed in Table 11.Source: A Evaluation Details · p. 15, Table 11 · p. 15
• Table 12 presents the number of trials conducted for each real-world experiment.The passage does not provide the individual trial counts.Source: A Evaluation Details · p. 15, Table 12 · p. 15

Causal masking vs. bidirectional attention

• Causal masking consistently lowers success rate relative to bidirectional attention in SIMPLER Visual Matching with the Google robot.Allowing video tokens to access action tokens helps capture action-driven visual changes; bidirectional interaction also improves coherence and alignment between action and video prediction.Source: B More Analysis · p. 15, Table 13 · p. 16

Asynchronous noising and inference

• 73.8% and 71.0% average scores for the two asynchronous variants trail the default synchronous strategy’s 80.4% by 6.6 and 9.4 percentage points, respectively.The variants use asynchronous training with either synchronous or asynchronous inference.Source: Table 14 · p. 16, B More Analysis · p. 16
• Joint training and denoising allow temporally aligned action and video modalities to exchange cross-modal information at every step, providing complementary supervision.The authors hypothesize that this explains the asynchronous variants’ performance degradation.Source: B More Analysis · p. 16

• Figures 5 and 6 show VideoVLA’s predicted visual imaginations alongside corresponding task-completion executions in real-world and simulation experiments, respectively.Source: C More Visualizations · p. 16

D Limitations and Broader Impacts

• Inference takes approximately 1.1 seconds on one H100 GPU, yielding around 3 Hz control when predicting 4 future latents and 6 actions with 10 DDIM denoising steps.The 4 latents correspond to 13 video frames, and only the first 3 of the 6 predicted actions are executed.Source: D Limitations and Broader Impacts · p. 16
• The speed limitation primarily stems from using the large pre-trained CogVideoX-5B video generator.Proposed future directions include a smaller robot-oriented video generator, one-step denoising such as ShortCut [62], and model distillation.Source: D Limitations and Broader Impacts · p. 16
• VideoVLA demonstrates the feasibility of using large-scale pre-trained generation models, rather than perception and understanding models, for end-to-end VLA robotic manipulation.The authors leave inference acceleration as an important avenue for future exploration.Source: D Limitations and Broader Impacts · p. 16