PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Agents Inside of Scalable World Models

Danijar Hafner, Wilson Yan, Timothy Lillicrap

arXiv:2509.24527v1cs.AIcs.LGcs.ROstat.ML

TL;DR

Previous world models struggled to predict complex object interactions accurately enough for training agents in imagination. Dreamer 4 combines scalable world-modeling and imagination training, enabling an agent to obtain diamonds in Minecraft purely from offline data without environment interaction.

  • Problem

    Previous world models struggled to accurately predict complex object interactions and game mechanics, limiting their use for training successful agents in imagination.

  • Method

    Dreamer 4 trains a scalable agent through reinforcement learning inside a fast, accurate world model using shortcut forcing and an efficient transformer architecture.

  • Results

    Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data without environment interaction, while its world model significantly outperforms previous world models.

  • Takeaways & Limitations

    Learning behaviors entirely in imagination from offline datasets supports applications where online interaction is impractical or unsafe.

  • Takeaways & Limitations

    Future directions include adding long-term memory, language understanding, corrective online data, and automatic goal discovery, indicating these capabilities remain outside the current system's demonstrated scope.

Abstract

from arXiv · show

World models learn general knowledge from videos and simulate experience for training behaviors in imagination, offering a path towards intelligent agents. However, previous world models have been unable to accurately predict object interactions in complex environments. We introduce Dreamer 4, a scalable agent that learns to solve control tasks by reinforcement learning inside of a fast and accurate world model. In the complex video game Minecraft, the world model accurately predicts object interactions and game mechanics, outperforming previous world models by a large margin. The world model achieves real-time interactive inference on a single GPU through a shortcut forcing objective and an efficient transformer architecture. Moreover, the world model learns general action conditioning from only a small amount of data, allowing it to extract the majority of its knowledge from diverse unlabeled videos. We propose the challenge of obtaining diamonds in Minecraft from only offline data, aligning with practical applications such as robotics where learning from environment interaction can be unsafe and slow. This task requires choosing sequences of over 20,000 mouse and keyboard actions from raw pixels. By learning behaviors in imagination, Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, without environment interaction. Our work provides a scalable recipe for imagination training, marking a step towards intelligent agents.

1. Introduction

World models can support imagination-based agent training, but prior systems struggled with complex interactions and efficient real-time simulation. Dreamer 4 addresses these gaps and demonstrates offline Minecraft diamond collection.

  • Motivation: World models predict action-conditioned futures, enabling agents to plan or learn through reinforcement learning in imagination.They can in principle train agents from fixed datasets without online environment interaction.
  • Limitations: Previous world models struggled to represent complex real-world distributions, precise object interactions, and game mechanics efficiently.Some controllable video models also required many GPUs for real-time simulation.
  • Dreamer 4: Dreamer 4 learns control policies through imagination inside a fast, accurate world model and predicts diverse Minecraft interactions and mechanics.The paper reports substantial improvement over previous world models.
  • Results: Dreamer 4 is the first agent to collect diamonds in Minecraft using only offline data, substantially improving over OpenAI’s VPT offline agent with 100× less data.The task uses no environment interaction.
  • Efficiency: The world model achieves real-time inference on a single GPU through shortcut forcing and an efficient transformer architecture.These design choices target accurate complex interactions and practical imagination training.
  • Data: Dreamer 4 learns action conditioning from unlabeled videos while requiring only a small amount of aligned action data.The paper reports strong generalization from this limited action supervision.

2. Background

The background develops diffusion and flow-matching foundations for shortcut models and diffusion forcing. These methods condition generation on noise and step size, enabling few-step sequential prediction, while Dreamer 4 applies the design to interactive dynamics modeling.

  • Flow matching: Flow matching trains a network to restore clean data from corrupted inputs by predicting a velocity toward the clean data.The signal level τ controls the mixture of noise and data, ranging from pure noise to clean data.
  • Flow matching: Inference begins from pure noise and iteratively transforms it into clean data over K sampling steps with step size d = 1/K.The step size determines the discretization used during generation.
  • Shortcut models: Shortcut models condition on both signal level and requested step size, allowing inference to select the number of sampling steps.Larger steps are trained through bootstrap distillation of two smaller steps.
  • Shortcut models: Shortcut training samples step sizes as powers of two and samples signal levels on the grid reached by the current step size.This couples the corruption level to the chosen generation step.
  • Shortcut models: Shortcut models can generate high-quality samples in 2 or 4 steps, compared with 64 or more for typical diffusion models.They avoid discretization error by learning each step’s endpoint.
  • Diffusion forcing: Diffusion forcing assigns different signal levels to sequence time steps, making each step both a denoising target and context for later steps.It supports generating the next frame from clean or lightly noised history.
  • Dreamer 4 design: Dreamer 4 combines causal tokenization with an interactive dynamics model that denoises action-conditioned representations using shortcut forcing.Both components use a block-causal transformer architecture.

3. World Model Agent

Dreamer 4 combines a causal tokenizer, action-conditioned dynamics model, and imagination-based reinforcement learning into a scalable world-model agent. Its design targets accurate, efficient interactive rollouts while improving policies beyond behaviors present in offline data.

  • Architecture: The agent uses a tokenizer and dynamics model built with the same efficient transformer architecture.The tokenizer compresses video frames into continuous representations, while the dynamics model predicts those representations from interleaved actions.
  • Causal Tokenizer: The tokenizer is trained with masked autoencoding and a reconstruction objective combining mean squared error and LPIPS loss.Randomized patch dropout improves representations and was found to improve the spatial consistency of generated videos.
  • Interactive Dynamics: The dynamics model uses shortcut forcing to generate each frame with K=4 forward passes, supporting fast interactive inference.It operates on interleaved observation and action blocks and conditions on signal levels and requested step sizes.
  • Interactive Dynamics: X-prediction of clean representations enables high-quality rollouts of arbitrary length by avoiding accumulated errors from high-frequency outputs.The model predicts clean representations instead of velocities during iterative frame-by-frame generation.
  • Interactive Dynamics: A ramp loss weight emphasizes cleaner signal levels because low-signal training provides less learning signal.The weight increases linearly with signal level, where τ=0 is full noise and τ=1 is clean data.
  • Imagination Training: Dreamer 4 first learns task-conditioned policy and reward heads from offline data, then improves the policy with reinforcement learning on imagined rollouts.A value head estimates future rewards, while a frozen policy copy serves as a behavioral prior during imagination training.

4. Experiments

The experiments test Dreamer 4 on offline Minecraft control, world-model interaction accuracy, action generalization, and architectural choices. Dreamer 4 combines imagination training with a fast world model to achieve strong long-horizon performance and broad action-conditioned generation.

  • Experimental scope: The experiments evaluate offline diamond collection, Minecraft interaction prediction, action-data requirements, generalization, and contributions from objective and architecture components.The Minecraft experiments use raw pixels and low-level mouse and keyboard actions, with models trained at large scale.
  • Offline Diamond Challenge: Human players take 20 minutes and approximately 24,000 mouse and keyboard actions to collect a diamond, making the challenge long-horizon and procedurally complex.The task requires gathering materials and crafting tools from raw pixels in a procedurally generated 3D world.
  • Offline Diamond Challenge: Dreamer 4 obtains diamonds in 0.7% of episodes, reaches over 90% success up to the stone pickaxe, and achieves 29% success for the iron pickaxe.It improves over behavioral-cloning agents more strongly at harder milestones.
  • Human Interaction: Dreamer 4 exceeds 20 FPS on one H100 GPU and uses a 9.6-second context, supporting real-time interaction with a substantially longer context than prior models.MineWorld achieves 2 FPS and does not support the real-time interactions required by these tasks.
  • Human Interaction: Dreamer 4 completes 14 of 16 Minecraft interaction tasks, while the large Oasis model completes 5 of 16 and Lucid-v1 does not allow task completion.Dreamer 4 predicts interactions including item switching, block placement and breaking, combat, boats, and portals, though temporal consistency is limited to 9.6 seconds.
  • Action Generalization: With 100 hours of actions among 2541 video hours, Dreamer 4 reaches 85% PSNR and 100% SSIM relative to training with all actions.With actions only from the Overworld, it reaches 76% of PSNR and 80% of SSIM on the Nether and End, which have only unlabeled videos.
  • Architecture and objective: Alternating batch lengths, sparse temporal attention, and grouped-query attention accelerate training or inference while maintaining or improving generation quality.The design study describes these choices as part of an efficient transformer architecture supporting long-video generation.

5. Related Work

Prior work established world models for control and scalable video generation, but existing systems remained limited in capacity or interaction fidelity. Dreamer 4 addresses this gap by targeting accurate complex object interactions suitable for imagination training.

  • Minecraft agents: Minecraft diamond collection is a long-horizon benchmark requiring resource gathering and tool crafting through thousands of low-level actions in procedurally generated 3D worlds.Prior work includes VPT, which used contractor gameplay to annotate large-scale web videos.
  • World model agents: Earlier world models achieved accurate pixel-based planning and strong control performance, but transformer and diffusion-based models remained limited in capacity for complex environments.Dreamer-based systems were reported as robust and efficient for high-dimensional control.
  • Scalable world models: Scalable models such as Genie 3, PlayerOne, and PEVA generate diverse scenes or detailed human movement, while Oasis, Lucid, and MineWorld simulate Minecraft from mouse and keyboard inputs.These systems generally support simple interactions or specialized forms of control.
  • Open limitation: Existing scalable world models still struggle to predict complex object interactions precisely enough for imagination training.This limitation motivates improving interaction fidelity rather than scene generation alone.
  • Fast generation: Fast-generation research includes parallel discrete-token sampling, diffusion distillation, and consistency models, each targeting fewer or more efficient generation steps.The cited approaches involve different speed-quality or training-schedule trade-offs.

6. Discussion

Dreamer 4 demonstrates offline imagination training for complex control while retaining important world-model limitations and open research directions.

  • Dreamer 4 obtains diamonds in Minecraft purely from offline data, without online interaction.The result demonstrates learning successful long-horizon strategies in complex environments.
  • Its world model predicts complex object interactions and supports real-time interactive inference on a single GPU.The model uses shortcut forcing and an efficient transformer architecture, but remains far from a full Minecraft clone.
  • Short memory and imprecise inventory predictions remain limitations of the Minecraft world model.
  • Future directions include general internet-video pretraining, long-term memory, language understanding, corrective online data, and automatic goal discovery.

A. Datasets

The study evaluates Dreamer 4 across Minecraft gameplay, separated Minecraft dimensions, robotics demonstrations, and first-person kitchen videos.

  • The Minecraft VPT dataset contains 2541 hours of contractor gameplay split into 90% training and 10% evaluation data.Keyboard actions are binary variables, while mouse actions use μ-law encoding with 11 bins per coordinate.
  • The Minecraft dataset is split into Overworld and Nether/End videos to study out-of-distribution generalization of action conditioning.The split uses item events to prevent Nether/End episodes from entering the Overworld portion.
  • SOAR Robotics contains 180 hours of teleoperated demonstrations and reinforcement-learning trajectories with 7D relative end-effector actions.The data include both successes and failures and use 90% training and 10% evaluation splits.
  • Epic Kitchens 100 contains 100 hours of first-person video recorded across 45 kitchens.The test set contains different tasks performed in the same kitchens.

B. Kitchen Generations

Figure 9 presents kitchen video generations initialized from holdout context.

  • Figure 9 shows kitchen video generations starting from holdout context.

C. Minecraft Tasks

The evaluation materials define the multi-task training set, milestone-based progress measures, and prompt sequences used for assessment.

  • Table 4 lists the taskset used to train the multi-task agent.
  • Table 5 lists milestone items used to measure progress during evaluation.
  • Table 6 lists the prompt sequence used for evaluation.

D. Offline Diamond Challenge

The offline diamond challenge reports milestone success rates and the time required to reach milestones, using separate evaluation summaries.

  • Table 7 reports each milestone item's success rate averaged over 1,000 evaluation episodes.Scores within 5% of the highest recorded score are highlighted in bold.
  • Table 8 reports the minutes needed to reach each milestone, averaged over successful episodes.Timings are omitted for items below a 0.5% success rate; scores within 5% of the fastest are bold.

E. Minecraft Inputs

Dreamer 4 learns directly from high-resolution images that reflect human players' experience.

  • Dreamer 4 uses high-resolution input images reflecting the experience of human players.

F. Previous Dreamer Generations

This section compares video-generation examples from Dreamer 3, Lucid-v1, Oasis, and Dreamer 4, while recalling Dreamer 3's Minecraft setup.

  • Dreamer 3 used low-resolution images, inventory states, and mouse, keyboard, and abstract crafting actions to obtain diamonds through online interaction.Its world model was a recurrent state-space model based on a recurrent neural network and a variational objective.
  • Figure 11 compares multi-step video generations between Dreamer 3 and Dreamer 4.
  • Figure 12 presents Lucid-v1.
  • Figure 13 presents Oasis (large).
  • Figure 14 presents Dreamer 4.
Loading 2509.24527v1…