Source-linked AI summary
VideoVLA: Video Generators Can Be Generalizable Robot Manipulators
Yichao Shen, Fangyun Wei, Zhiying Du, Yaobo Liang, Yan Lu, Jiaolong Yang, Nanning Zheng, Baining Guo
TL;DR
Robot manipulation still has limited generalization to novel tasks, objects, and settings, motivating alternatives to VLA systems centered on pre-trained understanding models. VideoVLA instead adapts a pre-trained video generator to jointly predict action sequences and their future visual outcomes. Its imagined futures correlate with reliable action plans and task success, while evaluations show generalization to novel objects and cross-embodiment skills.
Problem
Existing VLA models have limited ability to generalize to novel tasks, objects, and settings, despite using large pre-trained understanding models.
Method
VideoVLA uses a multimodal Diffusion Transformer initialized from a pre-trained video generator to jointly forecast future visual latents and 7-D robot action chunks from language and current observations.
Results
High-quality imagined futures correlate with reliable action predictions and task success, while VideoVLA generalizes to novel objects and cross-embodiment skill transfer.
Takeaways & Limitations
Jointly forecasting actions and their visual consequences supports a video-generation-based paradigm for generalizable vision-language-action manipulation.
Takeaways & Limitations
Inference takes approximately 1.1 seconds on a single H100 GPU, yielding an effective control frequency of around 3 Hz.
Abstract
from arXiv · showhide
Generalization in robot manipulation is essential for deploying robots in open-world environments and advancing toward artificial general intelligence. While recent Vision-Language-Action (VLA) models leverage large pre-trained understanding models for perception and instruction following, their ability to generalize to novel tasks, objects, and settings remains limited. In this work, we present VideoVLA, a simple approach that explores the potential of transforming large video generation models into robotic VLA manipulators. Given a language instruction and an image, VideoVLA predicts an action sequence as well as the future visual outcomes. Built on a multi-modal Diffusion Transformer, VideoVLA jointly models video, language, and action modalities, using pre-trained video generative models for joint visual and action forecasting. Our experiments show that high-quality imagined futures correlate with reliable action predictions and task success, highlighting the importance of visual imagination in manipulation. VideoVLA demonstrates strong generalization, including imitating other embodiments' skills and handling novel objects. This dual-prediction strategy - forecasting both actions and their visual consequences - explores a paradigm shift in robot learning and unlocks generalization capabilities in manipulation systems.
1 Introduction
VideoVLA investigates whether large video generation models can provide a foundation for more generalizable robotic manipulation. It jointly predicts robot actions and the visual consequences of those actions, linking visual imagination with manipulation reliability and generalization.
- Generalization is needed for robots to handle unseen tasks, novel objects, and unfamiliar environments in open-world settings.
- Video generators offer relevant capabilities because they produce physically plausible videos under novel text or visual conditions and learn physical dynamics from large-scale video data.
- VideoVLA transforms a Video Diffusion Transformer into a Video-Action Diffusion Transformer by jointly denoising video and action modalities.
- High-quality imagined future visuals correlate with reliable action predictions and successful task completion.
- VideoVLA generalizes to skills from other embodiments and to novel objects absent from training data.
2 Related Works
Related VLA work primarily builds on pre-trained vision, language, and vision-language understanding models, while other approaches connect video prediction or future visual information to action generation.
- Vision-Language-Action Models: VLA models interpret language, perceive environments, and control robots end-to-end using pre-trained understanding-oriented foundation models.
- Vision-Language-Action Models: Large-scale datasets, including Open X-Embodiment’s aggregation of over 60 robot datasets, support pre-training and generalization in VLA systems.
- Prior video-related approaches estimate actions from optical flow and depth, extract goal frames from predicted videos, or use video-model features for diffusion-based action generation.
3 Methodology
VideoVLA encodes language and visual observations into latent tokens, then uses a multimodal Diffusion Transformer to jointly forecast future visual latents and executable action chunks.
- Problem Formulation: Given a task instruction and current observation, VideoVLA predicts an action chunk and future visual content depicting the consequences of executing those actions.
- Problem Formulation: Each action is a 7-D vector encoding wrist rotation, wrist translation, and binary gripper state.
- Problem Formulation: After executing an action chunk, the model receives a new observation and repeatedly predicts the next chunk until task completion.
- Architecture: Text and video encoders convert the instruction and video into tokens or latent representations, while a DiT backbone jointly predicts future frame latents and actions.
- Video Encoder: The causal video encoder makes the first latent represent only the current observation, while later latents represent future frames.
- Unified Future Modeling: The multimodal DiT uses self-attention blocks to model interactions across modalities and time, with the backbone initialized from pre-trained CogVideoX.
- Unified Future Modeling: VideoVLA flattens visual latents, concatenates language, current-observation, future-visual, and action tokens, and denoises the noisy targets with DDPM diffusion loss.
4 Experiment
VideoVLA is evaluated across simulation and real-world settings for in-domain performance, novel-object generalization, cross-embodiment skill transfer, and the role of visual prediction in manipulation.
- Evaluation Setup: VideoVLA is evaluated on in-domain and generalization settings across simulation and real-world experiments, using average success rates over multiple trials.Simulation uses SIMPLER with Google and WidowX robots; real-world experiments use the Realman robot.
- Simulation Experiments: VideoVLA achieves the highest overall average performance across all 12 SIMPLER in-domain tasks and the highest averages on VM-WidowX and VA-Google evaluations.It ranks second-highest on VM-Google, while remaining strongest overall across the evaluated tasks.
- Simulation Experiments: VideoVLA achieves the highest average success rate on 10 simulated novel objects and outperforms prior models on eight of them.The objects are drawn from YCB and GSO and are absent from the Google robot’s training data.
- Simulation Experiments: VideoVLA outperforms CogACT by 28.2 percentage points when transferring eight WidowX skills absent from the Google robot’s training set.The evaluation measures cross-embodiment transfer from WidowX to Google within SIMPLER.
- Real-World Experiments: In real-world evaluation, VideoVLA handles all 12 novel objects with success rates from 16.7% to 83.3%, while OpenVLA and SpatialVLA reach 0% on nearly half.The task is picking up a novel object and placing it on a specified color plate.
- Real-World Experiments: VideoVLA achieves the highest success rates across all evaluated real-world cross-embodiment skills, including the substantially different “Topple” and “Wipe” skills.These skills were learned by the WidowX robot and evaluated on Realman with familiar objects.
- Ablation Study: Higher-quality video backbones, longer predicted horizons, and joint video-action prediction each improve manipulation performance.Training from scratch lowers success substantially; predicting 49 rather than 25 or 13 future frames improves performance consistently; removing video prediction causes declines across tasks.
- Imagination-Execution Correlation Analysis: Visual imaginations achieve 84.0% success on novel objects and 63.4% on new skills, compared with actual-execution success rates of 65.2% and 48.6%.Human judges assessed imagined trajectories for semantic instruction following and physical plausibility; execution additionally faces grounding, actuation, and perception difficulties.
5 Conclusion
VideoVLA uses pre-trained video generators to jointly predict executable actions and imagined visual outcomes, achieving generalization beyond seen objects and skills. Its results associate high-quality imagined futures with reliable action plans and task success.
- 5 Conclusion: VideoVLA jointly predicts future robot actions and visual imaginations of their anticipated outcomes.The framework leverages large pre-trained video generators rather than relying primarily on perception or vision-language models.
- 5 Conclusion: High-quality visual predictions are consistently associated with reliable action plans and task success.
- 5 Conclusion: VideoVLA generalizes to novel objects and cross-embodiment skill transfer in simulated and real-world environments.The paper attributes these capabilities to large-scale pre-trained video generation and the dual-prediction strategy.
A Evaluation Details
The evaluation reports trial counts for simulation and real-world experiments and applies a consistent protocol across VideoVLA and baselines. The experiments cover multiple SIMPLER settings and real-world evaluations.
- A Evaluation Details: Each experiment reports its number of trials for both simulation and real-world evaluations.
- A Evaluation Details: The evaluation protocol is applied consistently to VideoVLA and all baseline models.
- A Evaluation Details: Simulation trials include Google-robot Visual Matching and Variant Aggregation, WidowX Visual Matching, and Google-robot generalization experiments.
B More Analysis
Ablations show that VideoVLA performs better with bidirectional attention and synchronous joint diffusion than with causal masking or decoupled diffusion schedules. The default joint strategy reaches 80.4% average success.
- B More Analysis: Causal masking consistently lowers success rate relative to the default bidirectional model.The ablation evaluates SIMPLER Visual Matching with the Google robot.
- B More Analysis: 80.4% average success exceeds the 73.8% and 71.0% averages of asynchronous training variants.The corresponding drops are 6.6 and 9.4 percentage points.
- B More Analysis: The asynchronous variants decouple action and video noising during training, and one also denoises video before generating actions at test time.
C More Visualizations
Figures 5 and 6 visualize VideoVLA’s predicted visual imaginations alongside corresponding task executions. The visualizations cover both real-world and simulation experiments.
- C More Visualizations: Figures 5 and 6 place predicted visual imaginations alongside corresponding executions during task completion.
- C More Visualizations: Figure 5 presents visualizations from real-world experiments.
- C More Visualizations: Figure 6 presents visualizations from simulation experiments.
D Limitations and Broader Impacts
VideoVLA’s main limitation is inference speed: generating future video latents and actions takes about 1.1 seconds, yielding roughly 3 Hz control. The authors identify the large pre-trained video generator as the primary bottleneck and propose several acceleration directions.
- Inference speed: Inference takes approximately 1.1 seconds on a single H100 GPU, resulting in an effective control frequency of around 3 Hz.This timing uses DDIM denoising with 10 denoising steps.
- Inference speed: VideoVLA predicts 4 future latents corresponding to 13 video frames and 6 future actions, executing the first 3 actions.
- Inference speed: The inference-speed limitation primarily stems from reliance on the large CogVideoX-5B pre-trained video generator.
- Future directions: Proposed acceleration directions include a smaller robot-oriented video generator, one-step denoising, and model distillation.The authors leave inference acceleration as an important avenue for future exploration.