Source-linked AI summary
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, Linxi "Jim" Fan
TL;DR
Dexterous robot world modeling is limited by narrow data coverage and scarce fine-grained action labels. DreamDojo pretrains on 44k hours of diverse egocentric human videos using continuous latent actions, adapts to target robots, and distills the model for real-time control. It demonstrates improved OOD physics understanding and action following, real-time interaction, and alignment with real-world policy evaluation.
Problem
Dexterous robot world models face limited data coverage, weak generalization beyond training distributions, and scarce fine-grained action labels.
Method
DreamDojo pretrains on diverse human videos with continuous latent actions as unified proxy actions, then post-trains on target robots and applies distillation.
Results
DreamDojo demonstrates improved physics understanding and action following in OOD scenarios, real-time interactivity, and strong agreement with real-world policy evaluation, including Pearson r=0.995.
Takeaways & Limitations
DreamDojo supports live teleoperation, policy evaluation, and model-based planning for open-world dexterous robot tasks.
Takeaways & Limitations
DreamDojo struggles with uncommon actions, can overestimate absolute success rates, and does not naturally support multi-view simulation.
Abstract
from arXiv · showhide
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
1. Introduction
DreamDojo addresses limited robot-data coverage and scarce action labels by pretraining on diverse human videos with continuous latent actions, then adapting and distilling the model for controllable real-time simulation.
- Robot world models struggle with high-dimensional contact-rich controls because robot datasets have limited coverage and video world models typically plateau at discrete controls.
- DreamDojo pretrains on 44k hours of egocentric human videos spanning approximately 96× more skills and 2,000× more scenes than the most diverse public robot-learning datasets.
- Continuous latent actions provide unified proxy labels that help transfer interaction knowledge from unlabeled videos to action-conditioned world simulation.
- DreamDojo supports live teleoperation, large-scale policy evaluation without real-world deployment, and online model-based planning.
2. Preliminary
The paper treats an interactive world model as an action-conditioned state-transition model, and situates DreamDojo within a latent video diffusion architecture conditioned on text, frames, and actions.
- An interactive world model predicts future states from actions by sampling the next state from a transition distribution over states and actions.
- DreamDojo is built on Cosmos-Predict2.5, a latent video diffusion model that predicts future frames from text and conditional frame inputs.
- Cosmos-Predict2.5 operates in the continuous latent space produced by the WAN2.2 tokenizer and injects language and timestep conditions into each DiT block.
- The denoiser is conditioned by c, which can include text, conditional frames, and actions, while v_t represents the difference between noise and the clean sample x.
3. Approach
DreamDojo combines large-scale, diverse human videos with continuous latent actions and staged robot adaptation to learn controllable world models for dexterous tasks. Its design targets limited coverage, scarce action labels, and real-time autoregressive prediction.
- Training pipeline: DreamDojo’s three-phase training procedure consists of human-video pretraining, target-robot post-training, and optional distillation.Continuous latent actions condition all pretraining videos, while the action-conditioning layer is reset for each target embodiment.
- Dataset: 15× longer duration, 96× more skills, and 2,000× more scenes distinguish the curated data mixture from the previously largest world-model dataset.The comparison uses dataset-scale and diversity measures reported in Table 1.
- Dataset: 44,711 hours, 9,869 scenes, 6,015 tasks, and 43,237 objects make DreamDojo-HV the largest human interaction dataset for world-model pretraining.The dataset spans diverse daily environments and action distributions.
- Action controllability: Relative robot actions and temporally chunked action conditioning reduce modeling complexity and causality confusion, improving generalization and controllability.Relative actions are rebaselined at each latent frame, while four consecutive actions are sent together to the corresponding latent frame.
- Latent actions: Continuous latent actions provide unified proxy actions extracted from unlabeled videos, addressing scarce fine-grained labels and supporting transfer across embodiments.The latent-action model uses an information bottleneck to produce continuous vectors representing actions between frames.
- Distillation: A distillation stage converts the foundation model into a few-step autoregressive model, addressing bidirectional attention and slow multi-step denoising.The approach targets real-time prediction for applications including live teleoperation and online model-based planning.
4. Experiments
DreamDojo is evaluated through controlled ablations, out-of-distribution benchmarks, and downstream demonstrations spanning action conditioning, data diversity, model design, distillation, and applications.
- Experimental setup: Six evaluation benchmarks reflect diverse human-video scenarios and actions while remaining out-of-distribution for robot training datasets.
- Action conditioning: Latent-action pretraining narrows the gap to ground-truth action conditioning and reaches a higher upper bound than action-free or no pretraining.All compared methods are pretrained for 50k steps and post-trained for 25k steps on GR-1 data.
- Data mixtures: Increasing data diversity improves physics modeling and controllability for out-of-distribution actions across unseen-object and counterfactual evaluations.The ablation evaluates In-lab Eval, EgoDex Eval, DreamDojo-HV Eval, and Counterfactual Eval after GR-1 post-training.
- Unseen scenarios: DreamDojo-2B surpasses Cosmos-Predict2.5 in physics correctness and action following, while DreamDojo-14B performs better than DreamDojo-2B in both axes.The comparison uses human preferences on 50 samples from EgoDex-novel Eval and DreamDojo-HV-novel Eval.
- Design choices: Relative actions, chunked action injection, and temporal consistency loss each improve action-conditioned simulation quality on held-out GR-1 and counterfactual evaluations.Temporal consistency loss further improves action following and object modeling after the architectural modifications are applied.
- Distillation: The distilled student runs nearly 4× faster than its teacher while preserving close performance, enabling real-time streaming and multi-frame context.The student is evaluated on 600-frame, one-minute GR-1 Long Eval tasks and is more robust to occlusions and camera shifts.
5. Conclusion
DreamDojo is a foundation world model for dexterous robot-task simulation, combining human-video pretraining, continuous latent actions, and distillation for real-time interaction. It improves physics understanding and action following in out-of-distribution scenarios, while remaining limited on uncommon actions, nuanced failures, multi-view simulation, and inference speed.
- Conclusion: DreamDojo simulates dexterous robotics tasks and generalizes to unseen scenarios through large-scale human-video pretraining and continuous latent-action conditioning.The model targets general-purpose robot world modeling by transferring interaction knowledge from diverse human experiences.
- Conclusion: DreamDojo demonstrates improved physics understanding and action following in out-of-distribution scenarios, with positive correlation to real-world evaluations.The conclusion also reports real-time interactivity for live teleoperation and test-time policy steering.
- Limitations: Simulation remains imperfect for uncommon actions, nuanced failures, multi-view settings, and further inference-speed optimization.The authors specifically mention slapping and fast waving, overestimated absolute success rates during policy evaluation, and limited multi-view support.
- Conclusion: 10.81 FPS enables real-time autoregressive prediction for applications including live teleoperation and model-based planning.The distillation pipeline also models short temporal context to improve long-horizon consistency.
C. Human Preference Evaluation
The human preference evaluation uses a web interface that presents ground-truth video alongside two model-generated videos for intuitive comparison of physics correctness and action control.
- Human Preference Evaluation: Evaluators assess physics correctness and action following through side-by-side visual comparison.
D.1. Effects of Our Data Mixtures
Visualizations show that pretraining on human interaction data improves simulation of objects absent from the target robot dataset by supporting more precise physics modeling.
- Effects of Our Data Mixtures: Human data pretraining is essential for precise physics modeling of objects not captured by the robot dataset.The qualitative samples provide visual evidence alongside the quantitative evaluations.
D.2. Effects of Our Model Designs
Model-design comparisons show that relative actions, chunked injection, and temporal consistency mechanisms improve simulation quality and action controllability. Human-data pretraining also supports realistic simulation for unseen objects.
- Effects of Our Model Designs: Relative actions and chunked injection significantly enhance simulation quality and precise action controllability.The comparisons are reported across different model designs in Table 5.
- Effects of Our Model Designs: The proposed temporal consistency loss further reinforces modeling quality of objects.
- Effects of Our Data Mixtures: Human-data pretraining produces more realistic simulation for objects unseen in the target robot dataset.
D.3. Benefits of Distillation
Distillation enables DreamDojo to generate stable, long-horizon rollouts in real time while improving consistency under occlusions and camera shifts.
- DreamDojo continuously predicts stable, action-following 1-minute rollouts in real time after distillation.
- The distilled model recovers objects after occlusions by modeling short temporal context, unlike the teacher’s single-frame conditioning.
D.4. DreamDojo-HV Samples
DreamDojo-HV contains diverse interaction data spanning many action types and scenarios, while the full design achieves strong object modeling and action following.
- DreamDojo-HV samples highlight extensive coverage of interaction types and daily scenarios.
- Applying all design techniques produces the best capabilities for object modeling and action following.
D.5. PSNR Curves in Post-Training
Post-training PSNR curves show that latent-action pretraining reaches a higher performance ceiling than action-free or no pretraining, especially on EgoDex Eval.
- Latent-action pretraining reaches a much higher PSNR upper bound than action-free pretraining and without pretraining, especially on EgoDex Eval.
D.6. Value Model
DreamDojo combines real-time rollout generation, context advantages, diverse data, latent-action conditioning, and automated value estimation for selecting promising action proposals.
- The value model estimates the number of steps remaining to complete the current subtask and supports selecting the lowest-value action proposal for execution.
- 2.72 FPS versus 10.81 FPS: the student model generates chunk-wise rollouts four times faster than the teacher.
- Student context improves consistency under occlusions and camera shifts because the teacher lacks sufficient context.
- DreamDojo-HV encompasses extremely diverse actions and tool-using scenarios.
- Latent-action conditioning achieves comparable PSNR to high-quality action labels and a higher upper bound than action-free or no pretraining on EgoDex Eval.