Source-linked AI summary
Video Generators are Robot Policies
Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sudhakar, Paarth Shah, Rares Ambrus, Carl Vondrick
TL;DR
Visuomotor policies struggle with distribution shifts and costly demonstration requirements. Video Policy addresses both by using a large-scale video generator as a policy backbone with an action decoder, achieving broad generalization with limited demonstrations and enabling action-free video learning.
Problem
Visuomotor policies struggle to generalize across perceptual and task distribution shifts, while robot demonstrations are expensive to collect.
Method
Video Policy jointly generates robot-behavior videos and actions, using a video generator as the policy backbone and a learned decoder to produce executable actions.
Results
The method generalizes to unseen objects, scenes, and tasks in simulation and the real world, outperforming behavior cloning with less training data.
Takeaways & Limitations
Casting policy learning as video generation enables action-free training signals and offers a path toward more scalable and generalizable robot policy learning.
Takeaways & Limitations
The study is limited to small-scale simulation benchmarks, one real-world embodiment, one video model family, and costly video-diffusion inference.
Abstract
from arXiv · showhide
Despite tremendous progress in dexterous manipulation, current visuomotor policies remain fundamentally limited by two challenges: they struggle to generalize under perceptual or behavioral distribution shifts, and their performance is constrained by the size of human demonstration data. In this paper, we use video generation as a proxy for robot policy learning to address both limitations simultaneously. We propose Video Policy, a modular framework that combines video and action generation that can be trained end-to-end. Our results demonstrate that learning to generate videos of robot behavior allows for the extraction of policies with minimal demonstration data, significantly improving robustness and sample efficiency. Our method shows strong generalization to unseen objects, backgrounds, and tasks, both in simulation and the real world. We further highlight that task success is closely tied to the generated video, with action-free video data providing critical benefits for generalizing to novel tasks. By leveraging large-scale video generative models, we achieve superior performance compared to traditional behavior cloning, paving the way for more scalable and data-efficient robot policy learning.
1 Introduction
The paper targets poor visuomotor generalization under visual and task distribution shifts while reducing dependence on costly human demonstrations. It proposes Video Policy, using video generation as the policy backbone and showing broad generalization with limited data.
- Motivation: Visuomotor policies often fail to transfer learned behaviors across changes in appearance and entirely new tasks.These shifts range from color variations to more complex behavioral changes.
- Motivation: Robot action and human demonstration data are expensive to collect, making large-scale coverage of real-world variation difficult.
- Motivation: Prior approaches trade expressivity against data efficiency by relying on hand-crafted decoders or learned action decoders trained from limited demonstrations.The paper identifies this as a central challenge in extracting policies from generative video models.
- Contribution: Video Policy uses accurate generated robot-behavior videos so a small decoder can map synthesized frames to executable actions and generalize to unseen tasks.The video generative model therefore serves as the policy, while the decoder primarily acts as an interface.
- Results: The approach generalizes to new objects, scenes, and tasks with less training data than existing methods in both simulation and real-world experiments.The paper also reports improved performance over behavior cloning and analyzes important design choices through ablations.
2 Related Works
Related work spans behavior cloning, visual representation pretraining, and video models for decision-making. These lines of research respectively address action generation, physical-interaction representations, and prediction of future visual consequences for robotics.
- Behavior Cloning: Behavior cloning learns manipulation policies from demonstrations, progressing from end-to-end state-to-action models to energy-based and conditional generative approaches.The paper instead combines video diffusion with an additional action diffusion head that predicts actions jointly with pixels.
- Visual Pretraining for Policy Learning: Video prediction is used as a self-supervised objective to forecast future frames while capturing dynamics and causal relationships relevant to physical interactions.Contrastive learning and masked autoencoding are also described as visual pretraining techniques.
- Video Models for Decision-Making: Video-model decision-making research uses generative models to predict future frames, including action-conditioned world simulation for robotics.Recent text-to-video advances have renewed interest in internet-scale video data for robotic learning.
3 Methods
Video Policy conditions joint video and action diffusion on an initial scene and task description, using a video generator as the policy backbone and an action model as its decoder. Training combines video prediction with behavior cloning, while inference jointly generates frames and directly executable end-effector actions.
- 3.1 Overview: Given an initial scene and task description, Video Policy generates future robot-rollout videos and action sequences through a video generator f and learned action model g.The action dimension k corresponds to the robot end-effector.
- 3.1 Overview: The architecture fine-tunes the video generator and action decoder using passive video pretraining together with active robot demonstrations.At inference, generated actions are directly executed on the robot.
- 3.2 Architecture: Video and action denoising networks jointly predict future frames and action sequences, with the action U-Net conditioned on intermediate video-network features.This tightly integrates video and action prediction so both are generated simultaneously.
- 3.2 Architecture: The video U-Net conditions on the task’s CLIP embedding and combines an encoded initial image with noisy video-frame latents.
- 3.2 Architecture: At each denoising step, selected video-U-Net hidden embeddings are adapted into a global vector that conditions a 1D action U-Net.The adapter uses five evenly spaced decoder-layer embeddings to condition action generation.
- 3.3 Learning: Training proceeds in two stages: video prediction first, followed by action-model behavior cloning on demonstrations containing videos, task descriptions, and robot actions.
- 3.3 Learning: Figure 2 depicts joint diffusion from an initial image and future-frame/action noise, with frame representations conditioning the action denoiser.The modular design also permits separate network training and action-free learning of task dynamics through video generation.
- 3.3 Learning: Stopping action-loss gradients from propagating into the video generator preserves the video network’s policy-driving role and significantly improves performance.
4 Experiments
Experiments evaluate Video Policy across simulation benchmarks, ablations, action-free video pretraining, and real-world distribution shifts. The method achieves strong benchmark performance with limited demonstrations, while video-generation design choices and action-free data support generalization.
- Simulation Benchmarks: Video Policy achieves state-of-the-art average and most per-task RoboCasa performance using 50 human demonstrations.Training with 300 MimicGen demonstrations further improves performance.
- Simulation Benchmarks: Video Policy achieves the highest overall performance on the Libero10 benchmark compared with the reported baselines.The comparison uses average success rates across Libero10 tasks.
- Simulation Benchmarks: Video Policy outperforms baselines by a large margin on average and on most individual RoboCasa tasks, especially Pick and Place under distribution shift.It remains competitive with methods using large-scale visual-language or video pretraining, despite their larger demonstration sets.
- Ablations: Learning policy-execution videos is necessary and sufficient for robust manipulation representations, and two-stage training outperforms joint end-to-end training in average success rate.The two-stage variant first fine-tunes video generation, then learns the action denoising head on the frozen video representation.
- Ablations: Longer video prediction horizons matter more for tasks requiring stronger generalization, indicating that accurate environment dynamics are critical for policy generalization.All variants predict actions 1.6 seconds ahead while varying the video horizon; the distribution-shift task shows the larger horizon effect.
- Ablations: Action-free video pretraining enables strong generalization to unseen tasks, whereas a Diffusion Policy baseline trained on the same 12 supervised tasks shows only minimal generalization.The video model accesses action-free videos for all 24 tasks, while the action head is trained on 12 tasks.
- Real-World Generalization: In real-world experiments, Video Policy maintains strong performance across unseen object shapes or colors and black, red, or blue backgrounds.It also handles varied object locations, although some failures arise from unrealistic video predictions and limited pretrained real-world physics priors.
5 Conclusion
The paper presents pixel generation as an effective proxy for policy learning, improving behavior-cloning robustness and generalization while enabling action-free data as a training signal.
- Generating pixels can serve as an effective proxy for learning policy and substantially improve behavior-cloning robustness and generalization.
- Casting policy learning as video generation unlocks the use of action-free data, broadening the available training signals.
- Scaling generative models with large in-the-wild video datasets is presented as a promising path toward more scalable and generalizable real-world manipulation policies.
6 Limitations
The study is limited by narrow evaluation coverage, reliance on one video-generation model, and the computational cost of video diffusion for deployment.
- The evaluation is restricted to simulation benchmarks of limited scale and a single real-world embodiment.
- The analysis covers only one video-generation model, Stable Video Diffusion, leaving broader validation across tasks, environments, and model families for future work.
- Video diffusion models remain computationally expensive, creating a major practical bottleneck for real-world deployment.
A.1 Video Model Implementation
The implementation adapts Stable Video Diffusion for multi-view robot-video prediction, uses standardized benchmark protocols with documented variants, and evaluates sampled environments from MimicGen.
- Video Model Implementation: The adapted SVD model generates 25-frame sequences representing three camera views and a padded conditioning frame.Frames 1, 2–9, 10–17, and 18–25 correspond to the padded frame and gripper, left, and right camera views, respectively.
- Video Model Implementation: The generated videos represent robot actions over a 32-step prediction horizon, with eight frames allocated to each camera view.Videos are subsampled using a stride of 4.
- Evaluation Protocols: RoboCasa experiments follow its standard evaluation protocol except for the separate video-prediction-horizon study, which uses a modified protocol to isolate distribution shift.
- Experimental Variants: The implementation and evaluation compare variants including Joint, 2-Stages, No Video, Half Tasks, and DP Half Tasks.These variants are defined by whether video and action objectives are jointly trained, separated, omitted, or trained with reduced task coverage.
- Experimental Variants: Video prediction horizons are compared at 32, 16, and 0 steps, where the 0-step output reproduces the input images.
- Evaluation Environments: Evaluation includes per-task success rates on Libero10 and sampled MimicGen environments, including Open Single Door and PnP Counter to Cabinet.
A.2 Unified Video Action Model Baseline
The Unified Video Action Model baseline is adapted to generate three camera views from three conditioning images and is trained and evaluated under the same data and conditions as Video Policy.
- The baseline accepts three conditioning images and generates all three camera views by concatenating frames along the temporal axis.
- It uses a pretrained VAE and MAR image-generation model for initialization and produces videos at 256×256 resolution.
- The baseline is trained on the same dataset and evaluated under identical conditions as the video-conditioned policy model.
A.3 Diffusion Policy Baseline
The baseline comparison uses ResNet- and CLIP-based CNN Diffusion Policy variants, trained and evaluated under the same conditions as the video-conditioned policy.
- ResNet18 and CLIP-Base variants serve as CNN Diffusion Policy baselines initialized with pretrained weights.ResNet uses ImageNet pre-training, while both variants use pretrained initialization.
- Both baseline variants are trained on the same dataset and evaluated under the same conditions as the video-conditioned policy.
- The baselines receive task-name conditioning through the same CLIP text encoder used for the video model.This matches the video model's text-conditioning setup.
A.4 Real-World Experiment Setup
The real-world experiments use multi-camera demonstrations and evaluate robot behavior across novel backgrounds and objects. Qualitative rollout comparisons show alignment between predicted videos and executed actions.
- Experiment setup: Demonstrations use left, right, and gripper-mounted RGB cameras, with gripper pose, jaw opening, and grasping force additionally tracked.All sensors operate at 30 Hz, and the model predicts 32 steps into the future.
- Qualitative results: Pick and Place video predictions align with corresponding real-world rollouts during grasping and placement, showing coherent visual predictions alongside robot actions.
- Generalization settings: The unseen-background evaluation changes the white training table to black, red, and blue cloth-covered tables across four tasks.
- Generalization settings: Object-generalization tests use novel shapes and colors across Pick and Place, M&Ms to Cup, Upright Object, and Stack Cups.The corresponding test sets contain four or five novel objects, depending on the task.
- Experiment setup: During data collection, a human demonstrator performs each task with a modified gripper, and the robot later executes tasks with the same setup.