Source-linked AI summary
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, Susan Zhang
TL;DR
Dota 2 challenges reinforcement learning with long horizons, partial observability, and high-dimensional state-action spaces. The paper scales self-play reinforcement learning through distributed training and continual-transfer surgery, and reports that OpenAI Five defeated Team OG after ten months of training. The system reached superhuman performance in the reported evaluations, though surgery-based training ultimately plateaued below a from-scratch model.
Problem
Dota 2 combines long time horizons, partial observability, and complex high-dimensional observation and action spaces that challenge reinforcement-learning systems.
Method
The paper trains a recurrent policy with self-play reinforcement learning, distributed optimization, and surgery for continuing training across environment and model changes.
Results
OpenAI Five trained for ten months using 770±50 PFlops/s·days of compute, defeated the Dota 2 world champions, and won 99.4% of human-player games during an online showcase.
Takeaways & Limitations
The results demonstrate that self-play reinforcement learning can achieve superhuman performance in a difficult competitive esports environment.
Takeaways & Limitations
Surgery enabled weekly environment changes but ultimately plateaued at a weaker skill level than training from scratch.
Abstract
from arXiv · showhide
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as long time horizons, imperfect information, and complex, continuous state-action spaces, all challenges which will become increasingly central to more capable AI systems. OpenAI Five leveraged existing reinforcement learning techniques, scaled to learn from batches of approximately 2 million frames every 2 seconds. We developed a distributed training system and tools for continual training which allowed us to train OpenAI Five for 10 months. By defeating the Dota 2 world champion (Team OG), OpenAI Five demonstrates that self-play reinforcement learning can achieve superhuman performance on a difficult task.
1 Introduction
Dota 2 combines real-time multiplayer complexity with long horizons, partial observability, and high-dimensional state-action spaces. OpenAI Five addressed this by scaling reinforcement learning and continual-training infrastructure, ultimately defeating the world champions.
- Approach: OpenAI Five scaled existing reinforcement learning with thousands of GPUs over multiple months.The distributed system supported training at unprecedented computational scale.
- Results: OpenAI Five defeated Dota 2 world champions Team OG in April 2019, the first AI system to beat an esport world champion.The system was also opened to community play, where it won 99.4% of over 7,000 games.
- Continual training: Continual-training tools called surgery preserved training progress while the environment and code changed.The team performed approximately one surgery every two weeks during the 10-month training process.
- Scope: The paper presents lessons from the experiment that may generalize to other complex reinforcement-learning tasks.The authors describe these lessons alongside the technical system and the path to defeating the world champions.
2 Dota 2
Dota 2 is a complex, partially observed real-time strategy game with long episodes and large state-action spaces. OpenAI Five simplified the regular game by supporting only a subset of heroes and excluding mechanics requiring simultaneous control of multiple units.
- Game setting: Dota 2 pits five-player teams against each other on a map containing heroes, buildings, creeps, items, and other game features.The game ends when one team destroys the opposing team’s ancient.
- Challenges: Each episode lasts approximately 45 minutes at 30 frames per second, yielding about 20,000 agent timesteps.OpenAI Five acts every fourth frame.
- Challenges: Partial observability requires players to infer unseen game states and model opponent behavior.Teams see only areas near their units and buildings.
- Challenges: Each timestep exposes approximately 16,000 values while the discretized action space contains 8,000–80,000 actions, depending on the hero.The spaces include categorical values with hundreds of possibilities.
- Limitations: The system supported only 17 of the regular game’s 117 heroes.This was one of two stated limitations relative to regular Dota 2.
- Limitations: OpenAI Five excluded items that temporarily let players control multiple units simultaneously.The items were removed to avoid the technical complexity of multi-unit control.
3 Training System
OpenAI Five used a recurrent policy, self-play rollouts, distributed PPO optimization, and infrastructure for continual transfer across environment changes. Surgery preserved performance during ongoing modifications, although a rerun later plateaued below a from-scratch model.
- Action design: Some mechanics remained hand-scripted, including item and ability purchasing, courier control, and item reserves.The authors state the agent achieved superhuman performance before these actions were learned by policy.
- Training design: Training randomized heroes and item purchases to expose the agent to diverse situations and strategies.The purpose was to support robustness against the variety of human-opponent games.
- Playing Dota using AI: The agent converts game observations into discrete actions at every fourth frame, operating on a recurrent policy with approximately 159 million parameters.The policy is parameterized as a recurrent neural network centered on a 4096-unit LSTM.
- Playing Dota using AI: The policy uses a shared LSTM whose outputs feed separate policy and value-function layers, with one replica controlling each hero.The five replicas share parameters but maintain separate hidden states and receive hero-specific information.
- Optimizing the Policy: Training uses PPO with Generalized Advantage Estimation, Adam optimization, truncated backpropagation through time, and gradient clipping.The optimizer applies updates using 16-timestep samples and publishes parameter versions to a central controller.
- Optimizing the Policy: Self-play rollouts generate experience while optimizer GPUs asynchronously sample buffers and synchronously average gradients across the optimizer pool.Rollouts play the latest policy against itself in 80% of games and older policies in 20%.
- Continual Transfer via Surgery: Surgery transfers learned parameters across changing environments and models while preserving the old policy function when possible.The method implements Equation 1 exactly when possible and approximately when changes prevent exact preservation.
4 Experiments and Evaluation
OpenAI Five reached superhuman Dota 2 performance through large-scale training, continual model updates, and experiments on batch size, data quality, and long-horizon credit assignment. It defeated world champions and showed measurable gains from scaling, high-quality data, longer horizons, and surgery-based continuation.
- 4 Experiments and Evaluation: 770±50 PFlops/s·days of compute supported a single ten-month training run that reached superhuman performance.The training used batches of 1 to 3 million timesteps, a model with over 150 million parameters, and 180 training days spread over ten months.
- 4.1 Human Evaluation: OpenAI Five defeated the Dota 2 world champions 2-0 in a best-of-three match.The system later won 99.4% of human-player games during a multi-day online showcase.
- 4.1 Human Evaluation: OpenAI Five’s playstyle evolved toward human-like resource concentration and selective fighting as the agents improved.Earlier training favored risky group fights, producing quick wins after early advantages but prolonged losses when the agent fell behind.
- 4.3 Batch Size: 2.5x speedup came from Rerun’s 983k-timestep batch size versus the 123k baseline at TrueSkill 175.Increasing batch size sped training through the tested regime, reaching batches of millions of observations; small degradations in data quality severely harmed learning.
- 4.5 Long term credit assignment: Longer horizons improved resumed-agent win rate through the explored 6-12-minute range, with diminishing returns at high horizons.The result indicates accurate credit assignment for policies and actions affecting rewards several minutes into the future.
5 Related Work
OpenAI Five builds on deep reinforcement learning, large-scale optimization, distributed training, and self-play research. Related systems established these components across board games, Atari, robotics, and complex real-time environments.
- OpenAI Five combines deep reinforcement learning, large-scale optimization, and self-play to address complex game environments.
- Self-play has supported skill learning in high-dimensional continuous environments and automatic curriculum generation.Related work also connects this approach to fictitious play and opponent-distribution learning.
- AlphaGo, AlphaZero, and related systems used self-play or search-based training across Go, Chess, Shogi, and Hex.
- AlphaStar provides a close comparison through its use of a hand-designed architecture, autoregressive actions, an LSTM, and actor-critic reinforcement learning in StarCraft 2.
- Deep reinforcement learning previously achieved superhuman performance on Atari games using convolutional networks, Q-learning, and experience replay.
- Distributed learning research showed that larger batches and asynchronous rollout collection can accelerate training across games and robotics.
6 Conclusion
The paper concludes that scaling modern reinforcement learning can produce superhuman performance in competitive esports. It attributes the result to larger computation and surgery techniques that maintained training across changes.
- Modern reinforcement learning can achieve superhuman performance in competitive esports when successfully scaled up.
- Increasing batch size and total training time were identified as key ingredients in scaling compute.
- Surgery techniques extended a single training run to ten months across changes to the model and environment.
- The authors hypothesize that these results may apply beyond Dota 2 to other complex tasks.
A Compute Usage
The compute analysis estimates GPU optimization usage for OpenAI Five and Rerun, while emphasizing that the accounting excludes substantial non-optimization costs and carries uncertainty.
- The estimate has a crude net ambiguity of 5-10% from metric definitions, omitted computation, and a simplified experiment history.
- 770±50 PFlops/s·days was the estimated GPU optimization compute used by OpenAI Five when it played the world champions.
- 820±50 PFlops/s·days was the estimated total optimization compute for OpenAI Five when training ended.
- 150 ± 5 PFlops/s·days was the estimated optimization compute used by Rerun between May 18th and July 12th, 2019.
- The calculation covers optimization compute only, while rollout GPUs, self-play CPUs, and overhead account for additional costs.The passage estimates these components at roughly 30%, 30%, 30%, and 10% of dollar costs, respectively.
- OpenAI Five’s compute number excludes earlier studies, ablations, false starts, and resources spent before restarts, whereas this caveat does not apply to Rerun.
B Surgery
Surgery is a set of techniques for continuing to train a high-skill agent across changes to architecture, observations, actions, and environment. The procedures preserve existing behavior where possible while enabling new capabilities to be learned.
- Surgery: Surgery resumes training a high-skill model in a newly shaped model or environment while targeting matched TrueSkill.
- Changing the architecture: For unchanged observation, action, and environment spaces, surgery can preserve the old policy’s exact mathematical function.
- Changing the architecture: Expanding an internal layer initializes parameters so old activations and downstream outputs remain unchanged while new dimensions can learn.
- Changing the architecture: Zero-initializing only the minimal set of weights avoids undesired output symmetries while randomized weights break symmetry among new dimensions.
- Changing the architecture: Increasing the LSTM hidden dimension from 2048 to 4096 required small random new weights selected by their effect on TrueSkill.
- Changing the observation space: Adding observations enriches the encoder’s output for the same game state, allowing unchanged downstream computation when the change is additive.
- Changing the environment or action space: New environment features and actions were annealed from 0% to 100% of rollout games to avoid sudden strategy disruption.When TrueSkill fell, the authors reverted and retried the annealing more slowly.
- Removing model parts: Exact policy equivalence prevents many parameter-removal surgeries, so deprecated observations remained as constant inputs.
C Hyperparameters
OpenAI Five’s training used evolving hyperparameters and auxiliary predictors to improve learning in a changing, distributed experiment. The authors report that these settings were not fully optimized or exhaustively evaluated.
- C Hyperparameters: Hyperparameters were modified during training when performance reached improvement plateaus, with some changes retained and others reverted.Compute limitations prevented testing changes in separate experiments.
- C Hyperparameters: The authors state that the hyperparameters used cannot be claimed optimal, and the Rerun schedule could likely be improved substantially.The large experiment made systematic hyperparameter scans impractical.
- C Hyperparameters: Rerun changed four key hyperparameters: entropy penalty coefficient, Team Spirit, GAE time horizon, and learning rate.The schedule was simplified using lessons from the original training process.
- C Hyperparameters: Rerun reached OpenAI Five’s skill level before the planned settings of 840-second horizon, Team Spirit 1.0, and learning rate 1e-6 were reached.Each scheduled change was applied gradually over one to two days and several thousand iterations.
- Auxiliary predictions: Auxiliary prediction heads estimate win probability, net worth rank, and whether heroes will help destroy enemy buildings.The predictors transform LSTM outputs into future-state estimates, with win probability trained using a small gradient weight and other predictions using stop_gradient.
- Auxiliary predictions: Enemy-building predictions use a two-minute discount horizon, so they represent time-discounted event likelihoods rather than calibrated probabilities.Win probability labels are defined at the end of the game, while some predictors receive ground truth only on the final segment.
D.1 Understanding OpenAI Five Finals
The authors use prediction heads and drafting analysis to inspect what OpenAI Five understood during Finals games. These analyses connect predictions to game events, hero participation, matchup strength, and observation-space design.
- D.1 Understanding OpenAI Five Finals: Win-probability predictions improved throughout Rerun, with even an early low-performing agent recognizing situations associated with eventual victory.The progression is presented as evidence of increasingly refined game understanding.
- D.1 Understanding OpenAI Five Finals: At roughly 5 minutes, losing three heroes caused a win-probability drop; at roughly 18 minutes, killing a key enemy hero improved the game outlook.Figure 9 marks both events on the predicted win-probability trajectory.
- D.1 Understanding OpenAI Five Finals: Hero-specific building predictions often matched later participation, including cases where heroes absent from the attack were occupied elsewhere on the map.The predictions are discounted by expected time to the event and should not be read as calibrated probabilities.
- D.2 Hero selection: The drafting program uses precomputed lineup win probabilities and dynamic programming to choose the best available hero on each turn.It applies a minimax strategy to account for the opponent’s possible selections.
- D.2 Hero selection: At the start of the draft, OpenAI Five estimated Radiant’s win chance at 54% when picking first and 53% when picking second.The authors note that this estimate likely depends on the available hero set.
- Observation space: OpenAI Five observed approximately 16,000 structured inputs per timestep rather than screen pixels, an imperfect approximation that also exposed all encoded information simultaneously.The observation space was designed to study strategic planning while keeping rendering costs manageable.
F Action Space
OpenAI Five represents Dota 2 actions as factorized primary commands with parameters for timing, unit selection, and spatial offsets. The resulting space is large, variable across heroes, and partly supplemented by scripted systems.
- F Action Space: Each action combines a primary command with parameter outputs for delay, unit selection, and spatial offset.The model masks parameters that are irrelevant to the selected primary action.
- Action parameters: Spatial offsets use discrete X and Y values from -4 to +4, producing 81 coordinate pairs relative to a caster or selected unit.This parameter applies to abilities targeting locations on the map.
- Primary actions: Primary actions average 8.1 per timestep in games against OG and include universal, spell, item, and situational commands.Action filters restrict many commands according to the current game state.
- F Action Space: The factorized action space reaches 30 × 4 × 189 × 81 = 1,837,080 dimensions before accounting for masking and invalid combinations.The 30 primary actions represent the maximum supported at a timestep.
- F Action Space: Across games against Team OG, the average number of available actions varied from 8,000 to 80,000 across heroes.Different heroes have different numbers of spells, items, and parameter combinations.
- Scripted actions: Some decisions remained under scripted control, including ability builds, item purchasing and swapping, and courier control.The agent reached superhuman performance before these systems were transferred to the policy.
G Reward Weights
The reward design combines partial game signals with zero-sum adjustment, time weighting, and team-spirit sharing to address credit assignment in a five-agent team. A small ablation found that sparse rewards could work but reduced sample efficiency.
- G Reward Weights: The shaped reward adds intermediate signals to simplify credit assignment for actions contributing to eventual victory.Signals include events such as deaths, resource collection, and building damage.
- Reward components: Zero-sum adjustment subtracts the average enemy reward from each hero’s reward so benefits to one team correspond to losses for the other.This exploits the competitive structure of Dota 2 during reward computation.
- Team Spirit: Team Spirit τ interpolates between individual rewards at τ = 0 and equal sharing of team rewards at τ = 1.For intermediate values, each hero’s reward combines their own raw reward with the team mean.
- Team Spirit: Lower Team Spirit reduces gradient variance early in training, while the final objective is to optimize success for the entire team.The authors describe this as helping agents first learn individual mechanical and tactical abilities.
- Ablation: A small ablation with partial reward weights disabled still beat a hand-coded scripted agent consistently but incurred a large sample-efficiency penalty.The resulting policy appeared weaker early in games but coordinated fights nearer the end.
H Neural Network Architecture
OpenAI Five uses a shared recurrent policy-and-value architecture that processes structured observations for five heroes and projects the LSTM state into action choices. The architecture includes hero-specific processing, cross-hero pooling, unit embeddings, and action selection over available actions.
- The final combined policy-and-value network contains 158,502,815 parameters and uses shared gradients between policy and value outputs.
- Five replica networks process observations and actions for the team’s five heroes using largely shared structure with small hero-specific differences.
- The primary action is selected by a linear projection over available actions, followed by sampling or argmax selection.
- Observations are processed and pooled into a state vector, passed through a single-layer large LSTM, then projected into action outputs.
- Unit embeddings accompany the LSTM state and help action heads choose target units.
- Agents are evaluated with TrueSkill against reference agents, but changing game versions can inflate comparisons for newer agents.
K.1 Data flow between the training environment and Dota 2
The training environment connects Dota 2 to distributed learning through a gRPC-based, containerized interface, asynchronous execution, and frameskipped observations. This design supports scalable experimentation while imposing a measured reaction-time range and experimental controls.
- 167–267ms is the agent’s reaction-time range because observations arrive in four-frame bundles and processing overlaps with game execution.
- A Go helper process exposes Dota 2 through a gRPC server, enabling configuration, stepping, and isolated Docker-based game instances.
- Frameskip downsamples the 30-step-per-second game to an effective 7.5-frame-per-second observation and action rate.
- Rollout machines send data asynchronously from games in progress, keeping the rollout–optimization loop tight.
- Experiments monitor TrueSkill and speedup, use specified baseline hyperparameters, and account for uncertainty when estimating threshold-reaching speed.
- Large-batch reinforcement learning was presented as novel relative to prior large-scale supervised-learning work.
M.3 Sample Quality — Sampling and Sample Reuse
OpenAI Five’s asynchronous training depends on managing sample quality: stale or excessively reused data harms training, while lower sample reuse can modestly help. Self-play and exploration mechanisms provide the surrounding data-generation process.
- Sample reuse is computed from optimizer consumption and rollout production rates, with target values varied by changing rollout-worker capacity.
- Sample reuse measurements ranged from 0.57 to 6.3 across experiments that changed the number of rollout workers.
- Increasing sample reuse causes a significant performance decrease, while adding rollout workers can accelerate training when optimizers reuse data.
- The methods remain sample inefficient because rollout data cannot be usefully optimized for more than a few gradient steps.
- Sample reuse below one can be beneficial, although its improvement over reuse one is minor compared with penalties from more severe reuse levels.
- Random buffer sampling means reuse one is only an average: some samples are selected repeatedly while others are never selected.
- The training process uses self-play against current and past opponents, with 80% of games against the latest parameters and 20% against past versions.
P Hero Pool Size
The experiments examine how training choices and evolving model components affected OpenAI Five’s learning path and performance. Larger hero pools caused only modest early slowdowns, while several training-path effects revealed important optimization sensitivities.
- Hero Pool Size: Training with 80 heroes had a speedup factor of approximately 0.8, making early training 20% slower than with the base 17 heroes.The authors hypothesized that comparable skill would require approximately 20% more training time, although the experiment covered only early training.
- Hero Pool Size: Additional heroes slowed early training only slightly, while the 5-hero run severely underperformed during its first 4k versions.The authors did not investigate the underperformance in detail and considered instability in that run more likely than hero count as its cause.
- Manually Tuned Hyperparameters: Human intuition under time pressure was not considered the best way to set hyperparameters after the team lost both games at The International 2018.The team explored high-variance options in the final days before the event, but the agent was not yet believed to match the best humans.
- Zero Team Spirit Embedding: Zeroing a particular set of 128 learned parameters increased performance to about a 55% winrate, despite approximately unchanged shaped reward.The authors believed optimization missed this improvement because the shaped reward did not distinguish the higher winrate.
- Zero Team Spirit Embedding: The 128 parameters came from a fixed team-spirit embedding and did not expand the neural network’s representable function space.Because the embedding was consumed by a fully connected layer, it was exactly equivalent to modifying the bias vector.
- Learning Path Dependency: Adding Divine Rapier caused a negative feedback loop in reproduction, whereas banning it allowed TrueSkill to continue improving.The authors hypothesized that Rapier increased reward variance because it could be transferred after dropping on death, impairing reliable value-function learning.