Source-linked AI summary
DreamGen: Unlocking Generalization in Robot Learning through Video World Models
Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, Linxi Fan
TL;DR
Robot learning needs scalable alternatives to costly teleoperation and simulation engineering. DREAMGEN adapts video world models to robots, generates neural trajectories with pseudo-actions, and uses them to train policies; it improves generalization across behaviors and environments, while remaining limited by task scope and substantial compute requirements.
Problem
Manual teleoperation for every new task and environment is costly, while simulation requires engineering and can suffer from a sim2real gap.
Method
DREAMGEN fine-tunes video world models, generates robot videos from initial frames and language instructions, extracts latent or IDM pseudo-actions, and trains policies on neural trajectories.
Results
43.2% success on new behaviors in seen environments and 28.5% in completely unseen environments are achieved from teleoperation data covering one task in one environment.
Takeaways & Limitations
DREAMGEN provides a synthetic-data route for scaling robot learning beyond extensive manual demonstrations, with DreamGen Bench linking video-model scores to downstream policy performance.
Takeaways & Limitations
The evaluated tasks are relatively simple, cover limited portions of the robot’s kinematic capabilities, and the pipeline requires significant compute.
Abstract
from arXiv · showhide
We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.
1 Introduction
DREAMGEN uses video world models to generate synthetic robot training data, reducing reliance on manually collected demonstrations while supporting behavior and environment generalization. Its experiments show gains across embodiments and a benchmark connection to downstream policy performance.
- Manual teleoperation for each new task and environment is costly, while simulation requires engineering and can suffer from the sim2real gap.
- DREAMGEN fine-tunes video world models to a target robot, generates videos from initial frames and language instructions, extracts pseudo-actions, and trains visuomotor policies on neural trajectories.Pseudo-actions come from either a latent action model or an inverse dynamics model.
- 333× scaling of synthetic data in RoboCasa yields log-linear policy improvements as neural trajectories increase.
- 37% to 46.4% average success on 4 GR1 tasks, 23% to 37% on 3 Franka tasks, and 21% to 45.5% on 2 SO-100 tasks using 10 to 13 real-world trajectories per task.
- 22 novel behaviors and unseen environments reach 43.2% and 28.5% success, respectively, from teleoperation data covering only pick-and-place in one environment.The baseline achieves 0% on most novel behavior and environment experiments.
- DreamGen Bench evaluates video-world-model adaptation to novel embodiments, objects, behaviors, and environments, and higher benchmark scores correlate with stronger downstream policy performance.The benchmark evaluates 8 models and does not require a physical robot in the loop.
2 DREAMGEN
DREAMGEN constructs neural trajectories by adapting video world models, generating robot videos, extracting pseudo-actions, and training policies on the resulting video-action pairs. The pipeline supports both latent and IDM action labels and can combine synthetic with real trajectories.
- 2 DREAMGEN: DREAMGEN is organized as four steps: fine-tune video world models, generate videos, extract pseudo-actions, and train policies on neural trajectories.
- 2.1 Video World Model Fine-tuning: Video world models are fine-tuned on teleoperated robot trajectories to capture embodiment-specific physical constraints and movement capabilities.LoRA is used by default to mitigate forgetting prior internet-video knowledge.
- 2.2 Video Generation: Synthetic videos use randomized initial frames and language instructions, including frames from new environments for environment-generalization experiments.
- 2.3 Pseudo Action Labeling: IDM predicts action chunks between two image frames, while LAPA extracts continuous latent actions representing visual changes between frames.IDM uses no language or proprioception; LAPA conditions on the current and one-second-ahead frames.
- 2.4 Policy Training on Neural Trajectories: Policies condition on language instructions and image observations, use zero state information, and predict either latent or IDM-labeled action sequences.
- 2.4 Policy Training on Neural Trajectories: Neural trajectories can be co-trained with real trajectories at a 1:1 sampling ratio or used alone with IDM actions for behavior and environment generalization.
- 2.4 Policy Training on Neural Trajectories: Figure 4 compares average success across 24 RoboCasa tasks while varying neural-trajectory counts, ground-truth data regimes, and pseudo-action labels.The x = 0 condition is the baseline trained only on ground-truth videos.
3 Experiments
DreamGen is evaluated for data augmentation and generalization across simulation, real-world robots, novel behaviors, and novel environments. The experiments show scalable gains from neural trajectories and policy learning beyond the original teleoperation distribution.
- DreamGen evaluates three applications: augmenting existing-task data, generalizing to novel behaviors, and generalizing to novel environments.
- Simulation experiments: 20.6% average success rate is achieved when policies train solely on IDM-labeled neural trajectories across 24 tasks.
- Real-world Experiments: Neural trajectories consistently improve Diffusion Policy, π0, and GR00T N1 across dexterous real-world tasks and robot embodiments.
- Behavior Generalization: 11.2% →43.2% performance is obtained for novel behaviors using neural trajectories generated from pick-and-place supervision.
- Environment Generalization: Environment generalization uses initial frames from new environments without physical data collection beyond a single training environment, while the single-environment baseline achieves 0% Success Rate.
4 DreamGen Bench: A Video Generation Benchmark for Robotics
DreamGen Bench evaluates whether video world models adapt to robot embodiments while following instructions and physical constraints across novel tasks, objects, and environments. Its scores positively correlate with downstream RoboCasa policy performance.
- DreamGen Bench measures instruction following and physics alignment when video world models adapt to specific robot embodiments and generalize to novel objects, behaviors, and environments.Instruction following assesses task consistency, while physics alignment evaluates physical plausibility for downstream robot learning.
- Eight model variants are evaluated across simulation and real-robot setups, including zero-shot and fine-tuned models on Franka Emika and Fourier GR1.The benchmark compares Hunyuan, CogVideoX, WAN 2.1, and Cosmos under both embodiment-adapted and zero-shot conditions.
- DreamGen Bench scores positively correlate with RoboCasa policy performance when models are compared using 7K neural trajectories each.The DreamGen Bench score averages instruction following and physics alignment from Table 2.
- The benchmark is intended as a diagnostic, low-cost proxy for identifying video world models that may support stronger downstream robot policies.The reported correlation links better language instruction following and physics modeling with higher RoboCasa performance.
5 Related Work
Related work studies synthetic robot data, video world models, and learning robot policies from videos. These approaches address data scarcity through simulation, generated trajectories, or visual information that often lacks action labels.
- Synthetic robot data can reduce the time and human cost of teleoperation, but simulation faces sim-to-real gaps and difficulty modeling complex objects.
- Video world models generate synthetic robot trajectories and support action extraction through inverse dynamics, optical flow, or high-level plans.
- Robot-learning methods use unlabeled videos to improve visual representations and extract human-object interactions, affordances, and related information.
6 Conclusion
DreamGen uses state-of-the-art video generative models to create synthetic videos and pseudo-actions for visuomotor policy training. The pipeline augments existing tasks and supports new behaviors in unseen environments.
- DreamGen generates synthetic robot videos and extracts pseudo-actions, enabling visuomotor policy training without relying solely on manual demonstrations.
- The approach augments existing tasks and supports learning entirely new behaviors in unseen environments.
7 Limitation
DreamGen is limited by task complexity, computational cost, manual initial-frame collection, and imperfections in its automatic evaluator. The authors identify broader behaviors, lower compute, automated frame generation, and improved evaluation as future directions.
- The evaluated tasks cover a limited portion of the robot’s kinematic capabilities and do not directly benchmark against existing video-learning methods.
- DreamGen requires substantial computation, with 240k RoboCasa samples taking 54 hours on 1500 NVIDIA L40 GPUs.
- The method relies on manually providing initial frames, creating operational overhead that future automation may reduce.
- DreamGen Bench uses lightweight open-source evaluators that can occasionally hallucinate, especially when judging physical realism.
- Intermediate simulation checks suggest that neural-trajectory quality is usually the main bottleneck rather than the inverse-dynamics model.
B Environment for Teleoperation and Evaluation
The evaluation uses a single seen teleoperation environment alongside 10 unseen environments, with multiview robot data arranged into a standardized 2×2 grid for video-model fine-tuning.
- The seen environment is where the GR1 pick-and-place teleoperation data was collected.
- Environment generalization experiments evaluate all 10 unseen environments.
- RoboCasa and DROID trajectories provide multiview examples for video world model processing.
- Multiview inputs are arranged in a 2×2 grid with left, right, and wrist cameras plus a black filler image.
- WAN 2.1 fine-tuning uses learning rate 1e-4, LoRA rank 4, and LoRA alpha 4, with training schedules varying by dataset.
E Detailed Experimental Results on RoboCasa
The RoboCasa experiments compare neural-trajectory training with experimental setups spanning real-world GR1, Franka, and SO-100 tasks, while reporting 20.55% success from neural trajectories alone.
- RoboCasa: 20.55% average success rate is achieved across 24 RoboCasa tasks using only neural trajectories.The passage presents this as close to ground-truth trajectory performance.
- RoboCasa: Table 4 reports the complete experimental results on RoboCasa, with NT denoting 240k neural trajectories.
- Experimental protocol: The experimental protocol includes training video world models and an inverse-dynamics model for each setup.
- Real-world tasks: Four real-world GR1 dexterous tasks use pick-and-place data collected in one lab environment plus task-specific low-data fine-tuning.
- Real-world tasks: Three Franka tasks use DROID data for video world model training and additional fine-tuning on task-specific low-data trajectories.
- Real-world tasks: Two SO-100 tasks require manually splitting concatenated videos into individual trajectories before fine-tuning.
G Full Real-world Experimental Results
The full real-world experiments report success-rate results across multiple setups and use automated instruction-following and physics-alignment judgments, averaged into a per-video score.
- Results: Table 5 reports success rates for the real-world data augmentation experiments, including high-data variants.
- Success-rate evaluation: Success-rate evaluation prompts a vision-language model to return binary judgments about whether a robot video follows the task instruction.
- Success-rate evaluation: A separate zeroshot prompt checks robot-arm presence and requires every view in a multiview clip to follow the instruction.
- Physics evaluation: Physics alignment is evaluated with VideoCon-Physics, a 7B video-text model fine-tuned on human annotations for physical-world alignment.
- Combined metric: The final score for each video is the average of the success-rate and physics-alignment scores.
H.3 Human Evaluation
Human evaluation is used to validate automatic benchmark metrics, with GPT-4o instruction-following scores showing near-perfect correlations with human judgments across datasets and model variants.
- Human evaluation: Human evaluation labels success rate and physics alignment to assess whether automatic metrics agree with human judgments.
- Correlation analysis: Table 6 reports Pearson correlations between automatic GPT-4o instruction-following scores and human scores across datasets and model variants.
- Correlation analysis: Table 7 reports corresponding correlations for Qwen2.5-VL automatic instruction-following scores on each dataset.
- Correlation analysis: GPT-4o correlations are 0.94 for RoboCasa, 0.93 for GR1-Object, 0.96 for GR1-Behavior, and essentially 1.00 for GR1-Env.
- Downstream validation: Generating 7k neural trajectories per video world model provides a direct but resource-intensive test of downstream policy performance.
I.1 GR1 Humanoid Experiments
The GR1 experiments evaluate DreamGen across task augmentation and generalization settings, using randomized object locations and repeated rollouts. Results include consistent gains from neural trajectories across multiple robot-policy models and task settings.
- Data augmentation: Hammering, Wiping, Folding, and Stacking each use 100 trajectories with randomized target-object locations.Task scores are assigned incrementally according to task-specific milestones.
- Evaluation protocol: 10 rollouts per checkpoint evaluate behavior and environment generalization with randomized target-object locations for fair comparisons.The randomization region is defined for the GR1 experiments.
- Franka experiments: Neural trajectories consistently boost Diffusion Policy, π0, and GR00T N1 across all evaluated Franka tasks.Evaluations used 10 rollouts per checkpoint, identical initial configurations, and randomized object poses.
- SO-100 experiments: SO-100 experiments include Picking 3 Strawberries with 10 real-world and 30 neural trajectories, and Tic-Tac-Toe with 13 real-world and 40 neural trajectories.The strawberry task evaluates three pick-and-place outcomes across randomized table locations.
- SO-100 experiments: Co-training with neural trajectories reduces proprioceptive overfitting and improves conditioning on the current visual state.The augmented policy is also less likely to remain stuck at the initial home position.