Source-linked AI summary
Mastering Diverse Domains through World Models
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap
TL;DR
Reinforcement learning algorithms often need substantial expertise and experimentation to transfer across application domains. Dreamer learns a world model and improves behavior through imagined futures, achieving strong results across diverse tasks and collecting Minecraft diamonds from scratch without human data or curricula.
Problem
Reinforcement learning methods often require substantial human expertise and experimentation to configure for sufficiently new application domains.
Method
Dreamer learns a world model that predicts action outcomes and trains actor and critic networks from imagined trajectories, using robustness techniques based on normalization, balancing, and transformations.
Results
Dreamer outperforms specialized methods across over 150 diverse tasks with fixed hyperparameters and is the first algorithm reported to collect Minecraft diamonds from scratch without human data or curricula.
Takeaways & Limitations
Dreamer moves reinforcement learning toward broad practical applicability without extensive experimentation.
Abstract
from arXiv · showhide
Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.
Introduction
Reinforcement learning methods often require domain-specific expertise and reconfiguration for new tasks. Dreamer addresses this challenge with a world model and robustness techniques, and applies them to the difficult Minecraft diamond task without human data.
- Motivation: Current reinforcement learning algorithms often use specialized methods for challenges such as sparse rewards, image inputs, spatial environments, and continuous control.Applying reinforcement learning to sufficiently new tasks requires substantial effort.
- Approach: Dreamer uses a learned world model to predict potential action outcomes, while actor and critic networks choose and evaluate behaviors.The algorithm improves behavior by imagining future scenarios.
- Contribution: Dreamer targets robust performance across diverse domains with fixed hyperparameters through normalization, balancing, and transformation techniques.The paper reports robust learning across more than 150 tasks, model sizes, and training budgets.
- Minecraft challenge: Minecraft diamond collection is challenging because of sparse rewards, difficult exploration, long time horizons, and procedural diversity.Previous approaches used human expert data and domain-specific curricula.
- Minecraft result: Applied out of the box, Dreamer is the first algorithm reported to collect diamonds in Minecraft from scratch without human data.This result addresses a recognized artificial-intelligence challenge involving long-horizon exploration from pixels.
Learning algorithm
Dreamer combines a recurrent world model with actor–critic learning from imagined trajectories. Its normalization, balancing, and transformed distributional objectives are designed to stabilize learning across domains with fixed hyperparameters.
- Architecture: Dreamer comprises a world model, critic, and actor trained concurrently from replayed experience while the agent interacts with the environment.The world model predicts outcomes, the critic evaluates them, and the actor selects valuable actions.
- World model: The world model encodes sensory inputs into stochastic representations, predicts their dynamics, rewards, and continuation, and reconstructs inputs to keep representations informative.It is implemented as a recurrent state-space model with encoder, sequence, predictor, and decoder components.
- World-model robustness: Free bits and a small representation loss support fixed hyperparameters across environments with different visual complexity.These techniques address the differing regularization needs of complex 3D environments and tasks where individual pixels matter.
- Imagination learning: Imagined trajectories begin from replayed representations and contain predicted states, actions, rewards, and continuation flags for actor and critic learning.The critic uses value distributions and bootstrapped λ-returns to estimate rewards beyond the imagination horizon.
- Return robustness: Return normalization with a denominator limit supports rapid exploration under sparse rewards and high performance under dense rewards.The cited comparison describes this as overcoming the lack of stable hyperparameters across domains.
- Robust predictions: Twohot targets represent continuous values with neighboring bins, while categorical cross-entropy makes gradient size independent of target magnitude.The method is applied to stochastic reward and return targets using exponentially spaced bins.
- Robust predictions: Dreamer applies symlog transformations to vector observations and decoder targets and uses a symexp twohot loss for reward prediction and critic learning.The transformations compress large positive and negative values while preserving sign.
Results
Dreamer is evaluated across eight domains and more than 150 tasks, where it generally outperforms specialized and standard reinforcement-learning baselines. Ablations and scaling experiments further show that its robustness techniques, world-model learning signals, model size, and replay ratio contribute to performance and data efficiency.
- Across 8 domains and over 150 tasks, Dreamer outperforms a wide range of previous expert algorithms and substantially outperforms PPO across all domains.
- Atari: Dreamer outperforms MuZero, Rainbow, and IQN on the 57-game Atari benchmark while using only a fraction of MuZero’s computational resources.
- DMLab: Dreamer exceeds IMPALA and R2D2+ performance on DMLab at 100M frames versus 1B environment steps, amounting to a data-efficiency gain of over 1000%.
- Data efficiency and Minecraft: Dreamer outperforms the best remaining Atari100k methods without EfficientZero’s additional complexity, and collects Minecraft diamonds from scratch without human data or adaptive curricula.
- Control benchmarks: Dreamer sets new state-of-the-art results on Visual Control and Proprio Control, outperforming specialized methods including DrQ-v2, CURL, D4PG, DMPO, and MPO.
- BSuite: Dreamer establishes a new state-of-the-art on BSuite, especially improving over previous algorithms in scale robustness.
- Ablations and scaling: All robustness techniques contribute on average, while Dreamer relies predominantly on the world model’s unsupervised objective; larger models and higher replay ratios improve performance and data efficiency.
Previous work
Prior general-purpose reinforcement-learning approaches each have important trade-offs involving robustness, tuning, computational complexity, input dimensionality, or expert-data requirements. Dreamer is presented as mastering diverse environments with fixed hyperparameters, without expert data, and with an open-source implementation.
- PPO is relatively robust but requires large amounts of experience and often underperforms specialized alternatives, while SAC requires tuning and struggles with high-dimensional inputs.
- MuZero uses complex components and lacks a released implementation, making reproduction challenging, whereas Gato requires expert demonstrations and is limited to settings where such data exists.
- Dreamer is positioned as handling diverse environments with fixed hyperparameters, without expert data, and through an open-source implementation.
- Minecraft research: Minecraft research includes competition environments based on MineRL, which supports learning meaningful skills through a diverse human dataset; Voyager uses language-model API calls to obtain items.
Conclusion
DreamerV3 is presented as a general reinforcement learning algorithm that masters diverse domains with fixed hyperparameters. The authors report strong performance across over 150 tasks and out-of-the-box diamond collection in Minecraft.
- DreamerV3 masters a wide range of domains with fixed hyperparameters.
- Over 150 tasks are covered, with robustness across varying data and compute budgets.
- Dreamer is the first algorithm reported to collect diamonds in Minecraft from scratch out of the box.
Methods
The methods evaluate Dreamer against PPO and other benchmark-specific algorithms under standardized computational and training protocols. DreamerV3 combines robustness techniques, architectural changes, optimizer choices, and replay-buffer modifications.
- Baselines: PPO is evaluated with fixed hyperparameters across all benchmarks using a high-quality Acme implementation.The implementation is checked against a published ProcGen performance reference.
- Protocols: Experiments use benchmark-specific protocols, including 57 Atari tasks, 30 DMLab tasks, and evaluation at 100M steps for DMLab.
- Seeds and error bars: Dreamer and PPO are generally run with 5 seeds, while ProcGen uses 1 seed, BSuite uses 10, and Minecraft uses 10.Curves report seed means with one standard deviation shaded.
- Computational choices: Each agent is trained on a single Nvidia A100 GPU, with Dreamer using a 200M model by default and a 12M model on two control suites.Replay ratio is selected to fit each benchmark's step budget.
- DreamerV3 changes: DreamerV3 introduces robustness techniques including observation symlog, KL balancing, free bits, unimix, return normalization, and symexp twohot loss.
- DreamerV3 changes: The architecture uses a block GRU, RMSNorm, SiLU activations, AGC, LaProp, and a larger replay buffer with an online queue.
Supplementary material
The supplementary material includes Minecraft prediction and learning-curve figures, diamond-score tables, and baseline labels. These materials distinguish episode-level item success from cumulative discovery over training.
- Minecraft predictions: The Minecraft world model receives 5 context frames and predicts 45 future steps from the action sequence without intermediate images.
- Minecraft results: At 100M environment steps, the Minecraft diamond-score table reports item success rates for Dreamer and baselines.
- Minecraft results: Dreamer obtains diamonds in 0.4% of episodes at the budget, while 100% of Dreamer agents discover one or more diamonds over training regardless of episode boundaries.The latter metric is distinct from episode-level success rates.
Atari learning curves
The supplementary figures include Atari and ProcGen learning curves and an Atari score table. These materials provide benchmark-specific performance visualizations and tabulated Atari results.
- Figure 10 presents Atari learning curves.
- Table 6 presents Atari scores.
- Figure 11 presents ProcGen learning curves.
ProcGen scores
The ProcGen evaluation compares Dreamer with PPO under fixed hyperparameters, while related learning curves and DMLab scores provide broader benchmark context.
- The ProcGen benchmark uses a PPO implementation with fixed hyperparameters that performs on par with or better than the original PPO.
- Table 7 reports ProcGen scores, while Figures 12 and 13 show learning curves for DMLab and Atari100k.
- The benchmark captions identify DMLab and Atari100k as separate evaluation domains alongside ProcGen.
Atari100k scores
The Atari100k section reports scores at a specified interaction budget and places them alongside evaluation protocols and DeepMind Control Suite results.
- Atari100k scores are reported at 400K environment steps, corresponding to 100k agent steps.
- The Atari100k evaluation protocols convert computational resources into A100 GPU days.
- EfficientMuZero achieves the highest Atari100k scores in the cited protocol, but changed the standard environment configuration.
- The surrounding benchmark materials include learning curves and tables for DeepMind Control Suite under proprioceptive inputs.
Visual control learning curves
Visual-control evaluation includes DeepMind Control Suite learning curves under visual inputs and BSuite category-level scores, including scale and memory.
- Figure 15 shows DeepMind Control Suite learning curves under visual inputs.
- Table 12 reports DeepMind Control Suite scores under visual inputs.
- BSuite scores are visualized by category, with Dreamer exceeding previous methods in the scale and memory categories.
- The BSuite scale category measures robustness to reward scales.
BSuite scores
The BSuite evaluation reports task-level scores averaged over environment configurations and aggregates performance by category and across all tasks, with robustness and learning-signal ablations.
- BSuite scores: Table 13 reports BSuite scores averaged over environment configurations, with aggregates by category and across all tasks.
- Robustness ablation: All robustness techniques contribute to Dreamer’s overall performance, although each may improve only a subset of tasks.
- Learning signal ablation: Dreamer relies predominantly on the undersupervised reconstruction objective, while reward and value gradients further improve performance on a subset of tasks.