PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamGen: Unlocking Generalization in Robot Learning through Neural Trajectories

Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, Linxi Fan

arXiv:2505.12705v1cs.ROcs.AIcs.LG

TL;DR

Manual teleoperation for every new task and environment is costly, while simulation-based synthetic data can require engineering and face a sim2real gap. DREAMGEN adapts video world models to a robot, extracts pseudo-actions from generated videos, and trains policies on neural trajectories; it reports behavior and environment generalization from single-task, single-environment teleoperation, with DreamGen Bench scores correlating with downstream policy performance.

  • Problem

    Collecting teleoperation data manually for every new task and environment is costly, while simulation-based synthetic data can require substantial engineering and face a sim2real gap.

    Source: §1 · p. 2

  • Method

    DREAMGEN fine-tunes video world models on a target robot, generates task videos from initial frames and language instructions, extracts pseudo-actions, and trains policies on the resulting neural trajectories.

    Source: §1 · p. 2

  • Results

    43.2% success on new behaviors in seen environments and 28.5% in completely unseen environments; DREAMGEN enables 22 novel behaviors from teleoperation on one task in one environment.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DreamGen Bench provides a diagnostic for video models adapting to novel robot embodiments, and higher benchmark scores correspond to stronger downstream policy performance.

    Source: §1 · p. 3

    The evaluated tasks cover a limited portion of the robot's kinematic capabilities; supporting more complex dexterous behaviors requiring richer control remains future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection.

1 Introduction

DREAMGEN addresses the cost of collecting teleoperation data across tasks and environments by using video world models to generate synthetic robot training data. The paper reports gains on existing tasks, generalization to new behaviors and environments, and a benchmark linking video-model scores with downstream policy performance.

Motivation

  • Manual teleoperation for every new task and environment is costly and labor-intensive; simulation-based synthetic data can require substantial engineering and face a sim2real gap.Source: §1 · p. 2

Approach

  • DREAMGEN fine-tunes video world models on a target robot, generates videos from initial frames and language instructions, extracts pseudo-actions, and trains policies on the resulting neural trajectories.Pseudo-actions come from either a latent action model or an inverse-dynamics model.Source: §1 · p. 2

Training-data augmentation

  • 37% to 46.4% on GR1, 23% to 37% on Franka, and 21% to 45.5% on SO-100, using 10 to 13 real-world trajectories per task.In RoboCasa, synthetic data scales to 333× the original demonstrations and yields log-linear policy improvements as neural trajectories increase.Source: §1 · pp. 2–3

Generalization

  • 43.2% success on new behaviors in seen environments and 28.5% in completely unseen environments; DREAMGEN enables 22 novel behaviors from teleoperation on one task in one environment.GR00T N1 trained on pick-and-place alone achieves 0% success on most novel behavior and environment experiments.Source: §1 · p. 3

DreamGen Bench

  • DreamGen Bench evaluates video models' adaptation to novel robot embodiments, and higher benchmark scores correspond to stronger downstream policy performance.It assesses eight models on manipulation of unseen objects, unseen behaviors, and unseen environments while following physics.Source: §1 · p. 3

2 DREAMGEN

DREAMGEN's four-stage pipeline adapts video world models to a target robot, generates task videos, labels them with pseudo-actions, and trains visuomotor policies on the resulting neural trajectories.

2 DREAMGEN

  • DREAMGEN comprises four steps for creating and using neural trajectories to train visuomotor robot policies.Source: §2 · p. 3

2.1 Video World Model Fine-tuning

  • Fine-tuning on teleoperated robot trajectories adapts the video model to the robot's physical constraints and movement capabilities; LoRA is used by default to mitigate forgetting prior video knowledge.The authors assess instruction following and physics following, and note that optimal fine-tuning amounts vary by model and data pair.Source: §2.1 · p. 3

2.2 Video World Model Rollout

  • After fine-tuning, DREAMGEN prompts the model with initial frames and language instructions to generate synthetic videos, varying object locations and environments for simulation and real-world experiments.Environment-generalization experiments use initial frames from new environments while fine-tuning the video model on data from a single environment.Source: §2.2 · p. 3

2.3 Pseudo Action Labeling

  • IDM predicts action chunks from pairs of frames, while LAPA encodes visual changes between current and future frames as latent actions.The IDM uses a sliding window to label generated videos; LAPA's latent actions do not require target-robot ground-truth actions for training.Source: §2.3 · pp. 3–4, Figure 3 · p. 4

2.4 Policy Training on Neural Trajectories

  • Policies use image observations and language instructions to predict latent or IDM-labeled actions, with state set to zero because neural trajectories lack state information.Training can combine neural and real trajectories at a 1:1 sampling ratio or use only IDM-labeled neural trajectories; Figure 4 compares average success across 24 RoboCasa tasks as neural-trajectory volume varies.Source: §2.4 · p. 4, Figure 4 · p. 5

3 Experiments

DreamGen experiments test neural trajectories for training-data augmentation and generalization to new behaviors and environments. Evaluations include RoboCasa simulation and real-world tasks on three robot embodiments.

3.1 Training Data Augmentation

  • On RoboCasa, policy performance rises log-linearly with neural-trajectory count; co-training boosts IDM and LAPA, and IDM-only training reaches 20.6% average success across 24 tasks.The video world model is trained on 1,200 original human demonstrations, while IDM and policy training use different benchmark data regimes.Source: §3.1 · p. 5
  • Zero-shot generalization to novel behaviors and environments with robot embodiments lacking ground-truth data remains an open research question.Source: §3.1 · p. 5
  • Neural trajectories consistently improve Diffusion Policy, π0, and GR00T N1 across all robot embodiments on dexterous tool manipulation, deformable-object manipulation, and pick-and-place tasks that pose significant simulation challenges due to complex physical interactions.Empirically, GR00T N1 shows higher performance gains than Diffusion Policy and π0. The authors hypothesize that separate action and decoder parameters for IDM actions help address neural trajectories having zeros as state.Source: §3.1 · p. 6

3.2 Unlocking Generalization

  • Performance rises from 11.2% to 43.2% when policies train on neural trajectories for 14 novel-behavior tasks.The behaviors use action verbs absent from the pick-and-place demonstrations.Source: §3.2 · p. 7, Table 1 · p. 7
  • 0%: the pick-and-place-only baseline fails on new environments, while neural-trajectory-trained policies achieve non-trivial success on seen and unseen behaviors there.The video model is fine-tuned on one environment; new environments contribute initial frames rather than additional physical data collection.Source: §3.2 · p. 7, Table 1 · p. 7

4 DreamGen Bench: A Video Generation Benchmark for Robotics

DreamGen Bench evaluates whether video world models adapt to a robot embodiment while following instructions and physics across new objects, behaviors, and environments. The authors test zero-shot and fine-tuned models and report a positive correlation between benchmark scores and downstream RoboCasa policy performance.

4 DreamGen Bench: A Video Generation Benchmark for Robotics

  • DreamGen Bench quantifies embodiment adaptation and generalization to new objects, behaviors, and environments through instruction-following and physics-following metrics.The benchmark targets internalizing the robot's rigid-body physics.Source: §4 · p. 7
  • Instruction following measures whether generated videos adhere to task instructions, using Qwen-VL-2.5 binary scores that correlate over 90% with human evaluations.Human evaluation agreement is reported as average Pearson correlation above 90%.Source: §4 · p. 8
  • Physics alignment averages VideoCon-Physics and Qwen-VL-2.5 scores because VideoCon-Physics was not trained on multiview RoboCasa videos or diverse robot environments.The scores range from 0 to 1.Source: §4 · p. 8
  • The benchmark tests four video world models in simulation on Franka and in the real world on Fourier GR1, including zero-shot inference.Table 2 reports benchmark statistics and results, with IF and PA metrics and zero-shot and fine-tuned variants.Source: §4 · p. 8, Table 2 · p. 8
  • DreamGen Bench scores positively correlate with RoboCasa policy performance when policies train on 7K neural trajectories per video world model.The benchmark score averages IF (GPT) and PA; the comparison is shown in Figure 6.Source: Figure 6 · p. 8, §4 · p. 8

5 Related Work

Prior robot-learning work uses simulation, video generation, and human videos to address the cost of collecting robot demonstrations, but each line has limits or different aims. DreamGen uses synthetic videos from a world model and explores latent actions co-trained with real-world actions.

5 Related Work

  • Simulation can make synthetic data collection more efficient [30] [31] [32] [33] [34] [20] [35] [36] [37] [38] [39], but faces sim-to-real gaps, hard-to-simulate objects, and constraints from TAMP or teleoperation interpolation.Earlier neural augmentation methods [40] [41] [42] [43] [44] also have limited motion diversity and mainly improve visual robustness to distribution shifts.Source: §5 · p. 9
  • Robot-video methods extract actions with IDMs [15] [16], optical flow [17], or high-level plans [18] [19], while a concurrent task-generalization approach is bounded to simulation [47].DreamGen separates video generation from policy learning rather than co-running these components with a robot policy.Source: §5 · p. 9
  • Human-video research extracts interactions, affordances, trajectories, or motions [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70], while DreamGen uses world-model videos instead.DreamGen also explores latent actions co-trained with real-world actions [13] [71] [72] [73] [74] [4] [75] [76].Source: §5 · p. 9

6 Conclusion

DreamGen uses state-of-the-art video generative models to produce synthetic robot videos and pseudo-actions for visuomotor policy training. The authors present the approach as a step toward learning new behaviors in unseen environments without relying solely on manual demonstrations.

6 Conclusion

  • DreamGen generates synthetic videos with state-of-the-art video generative models and extracts pseudo-actions to train visuomotor policies.Source: §6 · p. 9
  • The pipeline augments existing tasks and supports learning entirely new behaviors in unseen environments.Source: §6 · p. 9
  • The authors describe DREAMGEN as a stepping stone toward using world models for robot learning.Source: §6 · p. 9

7 Limitation

The paper identifies limits in task complexity, computational cost, and manual setup, alongside an evaluator that can hallucinate. Its pseudo-action discussion reports that neural-trajectory quality is often the bottleneck and gives LAPA training details.

7 Limitation

  • The evaluated tasks cover only a limited portion of the robot’s kinematic capabilities, leaving richer-control dexterous behaviors for future work.The authors also note they do not directly benchmark against complementary video-learning methods.Source: §7 · p. 9
  • Generating the 240k-sample RoboCasa dataset took 54 hours on 1500 NVIDIA L40 GPUs, making compute cost a significant constraint.Source: §7 · p. 9
  • Manually providing initial frames adds operational overhead; automated generation or selection remains future work.Source: §7 · p. 10
  • The lightweight open-source automatic evaluator can hallucinate, particularly when assessing physical realism in videos.The authors identify evaluation improvements as future work.Source: §7 · p. 10

A Extracting Pseudo Actions from Synthetic Videos

  • Digital-twin IDM checks indicate that neural-trajectory quality, more often than IDM quality, is the bottleneck in the authors’ observations.For LAPA training, the authors use a codebook size of 8, sequence length 16, and 100K steps with batch size 1024.Source: A Extracting Pseudo Actions from… · p. 15

B Environment for Teleoperation and Evaluation

This appendix documents the teleoperation and evaluation environments, multiview data layout, and video-model fine-tuning settings.

B Environment for Teleoperation and Evaluation

C Examples of Multiview Robot Data Processing

  • Multiview training videos place left, right, and wrist camera views in the top-left, top-right, and bottom-left grid cells, with a black image in the bottom-right.The layout is a 2×2 grid.Source: C Examples of Multiview Robot Data… · p. 16

D Video World Model Training Hyperparameters

  • WAN 2.1 fine-tuning used a 1e-4 learning rate, LoRA rank and alpha of 4, and dataset-specific epochs and batch sizes.RoboCasa: 100 epochs, batch size 32; GR1 and DROID: 75 and 5 epochs, batch size 64; SO-100: 200 epochs, batch size 8.Source: D Video World Model Training… · p. 16

E Detailed Experimental Results on RoboCasa

The appendix reports RoboCasa results and the fine-tuning data used for real-world experiments across GR1, Franka, and SO-100.

E Detailed Experimental Results on RoboCasa

F Fine-tuning Data for Video World Models and IDMs

  • GR1 models used 2,884 pick-and-place trajectories from one lab environment, with task-specific fine-tuning on 10 trajectories for Hammering, Wiping, and Stacking and 25 for Folding.The IDM was trained only on the 2,884 pick-and-place trajectories and reused across the experiments.Source: F Fine-tuning Data for Video World… · p. 17
  • Franka video models used 49,895 DROID examples and task-specific fine-tuning data; SO-100 videos were split into trajectories, yielding 68 and 44 trajectories from 10 and 13 sampled videos.For Franka, the DROID-only model generalized to new environments but made fine-grained errors such as grasping; its IDM used the DROID data without post-training.Source: F Fine-tuning Data for Video World… · p. 17

G Full Real-world Experimental Results

This appendix identifies the full real-world results table and describes automated success-rate and physics-alignment evaluation procedures.

G Full Real-world Experimental Results

H.1 Success Rate

  • Success-rate evaluation prompts Qwen2.5-VL-7B-Instruct to return a binary judgment on whether a video follows the task instruction.The appendix includes standard and zero-shot prompt templates.Source: §H.1 · p. 18

H.2 Physics Alignment

  • Physics alignment is evaluated with VideoCon-Physics and Qwen2.5-VL, and the two scores are averaged for each video.VideoCon-Physics is evaluated against human judgments using ROC-AUC and is reported to correlate strongly with them.Source: §H.2 · p. 18

H.3 Human Evaluation

The appendix compares automatic evaluation with human judgments and describes an intermediate simulation-based check of downstream performance.

H.3 Human Evaluation

  • GPT-4o automatic IF scores correlate strongly with human IF-human scores: 0.94 for RoboCasa, 0.93 for GR1-Object, 0.96 for GR1-Behavior, and essentially 1.00 for GR1-Env.The authors report these as near-perfect linear relationships and a reliable proxy for resource-intensive manual evaluation.Source: §H.3 · p. 19, Table 6 · p. 19

H.4 Intermediary Step for Checking Downstream Performance

  • Testing downstream performance directly required generating 7k neural trajectories per model; replaying IDM actions in the GR1 digital twin was offered as a cheaper intermediary check.The paper reports that benchmark scores directly correlate with downstream policy performance.Source: §H.4 · p. 19

I.1 GR1 Humanoid Experiments

The GR1, Franka, and SO-100 experiments examine data augmentation and generalization with neural trajectories under varied tasks and evaluation conditions; Figure 12 shows generated trajectory examples.

I.1 GR1 Humanoid Experiments

  • GR1 data augmentation covers Hammering, Wiping, Folding, and Stacking, with 100 trajectories per task and target locations randomized within a box.Evaluations use 10 rollouts per checkpoint.Source: §I.1 · p. 20, Figure 11 · p. 20
  • GR1 generalization evaluations use 10 rollouts per checkpoint and randomize target-object starting locations for direct model comparisons.Table 8 presents the evaluation criteria.Source: §I.1 · p. 20, Table 8 · p. 21

I.2 DROID (Franka) Experiments

  • The Franka study includes pick-and-place, cube stacking, and tool use, with a low-data regime of 10 trajectories except 25 for folding.The study collected 100 teleoperation trajectories for the tasks and trained the video world model and IDM on DROID data.Source: §I.2 · p. 22
  • Neural-trajectory conditioning boosts Diffusion Policy, π0, and GR00T N1 across all Franka tasks.The study uses identical initial states for head-to-head comparisons and randomizes object poses within rollout batches.Source: §I.2 · p. 22

I.3 SO-100 Experiments

  • SO-100 experiments evaluate strawberry picking and Tic-Tac-Toe using randomized object or grid configurations across trials.The strawberry task uses 10 real-world and 30 neural trajectories; Tic-Tac-Toe uses 13 real-world and 40 neural trajectories.Source: §I.3 · p. 22
  • Co-training with neural trajectories reduces proprioceptive-state overfitting and improves conditioning on the current visual state.The augmented policy is also less likely to get stuck at its initial home position, a common baseline failure.Source: §I.3 · p. 22

J Examples of Generated Neural Trajectories

Related papers

Loading 2505.12705v1…