Dota 2 with Large Scale Deep Reinforcement Learning

• Dota 2 challenges reinforcement learning with long time horizons, partial observability, high-dimensional observation and action spaces, and complex evolving rules.Its game logic spans hundreds of thousands of lines of code after more than a decade of active development.Source: §1 · pp. 1–2
• OpenAI Five defeated Dota 2 world champions Team OG1 in April 2019, becoming the first AI system to beat an esport world champion.The agent also won 99.4% of over 7000 community games.Source: §1 · p. 2
• The key approach was scaling existing reinforcement learning with thousands of GPUs over multiple months through a distributed training system.This system trained the Dota 2-playing agent OpenAI Five.Source: §1 · p. 2
• Surgery tools enabled training to resume after environment and code changes with minimal performance loss instead of restarting from scratch.During the 10-month training process, the team performed approximately one surgery every two weeks, enabling frequent improvements to the strongest agent.Source: §1 · p. 2

• Teams of five control heroes, gather resources from creeps, and improve abilities and items while defending their ancient and attacking the opponent’s.The game ends when one team destroys the opposing team’s ancient.Source: §2 · p. 2
• Approximately 20,000 decision steps span a 45-minute game, while partial observability requires inference from hidden map state and opponent behavior.OpenAI Five selects an action every fourth frame during gameplay running at 30 frames per second.Source: §2 · p. 2
• OpenAI Five observes ∼16,000 values and chooses among 8,000 to 80,000 actions per timestep, reflecting Dota 2’s high-dimensional state and action spaces.The environment includes ten heroes, many buildings and non-player units, and numerous game features; the action space is discretized.Source: §2 · p. 3
• The system supported a subset of 17 heroes rather than the normal pool of 117 and excluded items that temporarily control multiple units.These items were removed to avoid the technical complexity of enabling simultaneous control of multiple units.Source: §2 · p. 3

3.1 Playing Dota using AI

• At each timestep, OpenAI Five converted semantic observations into a discrete action, using a roughly 159-million-parameter recurrent network centered on a 4096-unit LSTM.Five replicas with shared parameters controlled the team’s heroes, each maintaining its own hidden state and receiving an indicator of the controlled hero.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 Optimizing the Policy

• The policy optimized win probability using auxiliary rewards for events such as deaths and resource collection, with reward symmetrization exploiting Dota 2’s zero-sum structure.Training used PPO with Generalized Advantage Estimation, Adam, truncated backpropagation through time over 16-timestep samples, and per-parameter gradient clipping.Source: §3.2 · p. 5
• The distributed self-play system used up to 1536 optimizer GPUs to synchronously train PPO, reaching an effective batch size of 2,949,120 time steps.Rollout machines played the latest policy against itself in 80% of games and older policies in 20%, while separate forward-pass GPUs served batched policy evaluations.Source: §3.2 · p. 5

3.3 Continual Transfer via Surgery

• Surgery transformed an old policy into a compatible new model while preserving its function when possible, or approximating it when environment, observation, or action changes prevented exact preservation.The method addressed changes from experimentation, incremental mechanic support, and new Dota 2 versions, where relevant old model components were otherwise difficult to carry over reliably.Source: §3.3 · pp. 7–8
• Over ten months, more than twenty surgeries enabled continuous training without loss in performance as OpenAI Five adapted to changing environments and model structures.Training from scratch after changes occurring approximately every two weeks was prohibitively costly; surgery was necessary before the team’s match against OG after switching to Dota 2 version 7.21d.Source: §3.3 · pp. 7–8

4.1 Human Evaluation

• 2-0: OpenAI Five defeated reigning Dota 2 world champions OG in a best-of-three, and won 99.4% of human-player games in its public showcase.Across 3,193 teams and 7,257 games, 29 teams defeated OpenAI Five in 42 games; abandoned games were counted as wins for OpenAI Five.Source: §4.1 · p. 8, §4 · p. 8
• OpenAI Five reacted in 217ms on average, developed a human-aligned but distinct playstyle, and adapted its strategy by fighting when ahead and gathering resources when behind.The final agent moved heroes more frequently, accepted calculated low-health aggression, and consumed resources and long-cooldown abilities more readily than humans.Source: §4.1 · pp. 9–10

4.2 Validating Surgery with Rerun

• Rerun reached over 98% winrate against the final OpenAI Five while taking 2 months and 150 ± 5 PFlops/s·days, validating surgery’s resource and iteration benefits.Training from scratch after each of twenty major surgeries would have taken 40 months instead of 10, although Rerun plateaued below the from-scratch model and surgery is less useful for a pre-built environment.Source: §4.2 · pp. 10, 12

4.3 Batch Size

• 2.5x: A 983k-time-step batch reached superhuman performance in two months versus a 123k baseline, though batch-size speedup remained below linear.Increasing batch size accelerated training through the tested range, up to batches of millions of observations; the smaller batch might have required 5 months for the same result.Source: §4.3 · p. 12

4.4 Data Quality

• A staleness increase of ∼8 versions caused significant slowdowns, while reusing data 2-3 times caused a factor of two slowdown and 8 times could prevent competent learning.The final system targeted staleness between 0 and 1 and sample reuse around 1, indicating that data quality had severe effects on learning, even beyond compute consumed.Source: §4.4 · p. 14

4.5 Long term credit assignment

• Longer reward-discount horizons improved a skilled agent’s performance through the longest tested horizons of 6-12 minutes, demonstrating long-term credit assignment.The base agent used a 180-second horizon with γ = 0.9993, and longer horizons produced diminishing returns at high values.Source: §4.5 · pp. 14–15, Figure 6 · p. 15

• OpenAI Five builds on deep reinforcement learning, large-scale optimization of deep models, and self-play for exploring environments and strategies.Source: §5 · p. 15
• Self-play has supported learning in high-dimensional environments and automated curricula, while OpenAI Five learns an opponent distribution using the latest policy rather than an average policy, unlike fictitious play.Source: §5 · p. 15
• Prior systems progressed from game mastery in Backgammon, Checkers, and Chess to Go, Chess, Shogi, Hex, multiplayer environments, and StarCraft 2.AlphaGo used imitation learning and self-play, whereas AlphaGoZero, AlphaZero, and ExIt used Monte-Carlo Tree Search during training.Source: §5 · p. 15
• AlphaStar and OpenAI Five both used actor-critic methods, recurrent architectures, and autoregressive action decoding, but differed in self-play versus league training and in handling hidden state information.OpenAI Five hard-coded some subsystems, while AlphaStar conditioned training on human-replay statistics; AlphaStar’s full-information value network is proposed for future Dota 2 exploration.Source: §5 · p. 16
• Deep reinforcement learning combined convolutional networks, Q-learning, and experience replay to reach superhuman Atari performance, motivating later work on sample efficiency and intrinsic rewards [3, 40, 16, 41–43].Source: §5 · p. 16
• Distributed training research found benefits from larger batches, asynchronous rollout collection through PPO [14] and A3C, and distributed learning in games and robotics [48, 5].Network surgery relates to function-preserving expansion [23], incremental growth,, [51], transfer methods, [53], distillation [54], imitation learning, and behavioral cloning [24].Source: §5 · p. 16

6 Conclusion

• Scaling compute through larger batch sizes and longer training can enable modern reinforcement learning techniques to achieve superhuman performance in competitive esports games.The authors identify increased compute scale, batch size, and total training time as key ingredients.Source: §6 · p. 16
• Training runs can be extended to ten months using surgery techniques that continue training across changes to the model and environment.These techniques supported the paper’s long-duration training process.Source: §6 · p. 16
• Although the work focused on Dota 2, the authors hypothesize that the methods generalize to zero-sum two-team continuous environments simulatable in parallel across hundreds of thousands of instances.This is presented as a hypothesis about broader applicability rather than an established result.Source: §6 · p. 16
• The conclusion anticipates that scaling will become more important as environments and tasks grow more complex.The claim is framed as a future direction for current methods.Source: §6 · p. 16
• The conclusion frames parallel simulation across hundreds of thousands of instances as a condition for the hypothesized broader applicability of the methods.This condition qualifies the authors’ proposed generalization beyond Dota 2.Source: §6 · p. 16

Appendix

• The appendix includes a section titled “Environment Randomization,” listed on page 62.The supplied appendix passage is a table-of-contents entry and provides no further findings.Source: Appendix · p. 24

Estimation method

• The methodology estimates compute by segmenting training around surgeries or batch-size changes, counting gradient-descent steps, and profiling floating-point operations per GPU step.For each segment, gradient-descent steps equal iterations × 32 before multiplication by profiled per-step GPU operations.Source: A Compute Usage · p. 25

Uncertainty

• The authors assign a crude 5-10% net ambiguity to the estimate because counted operations include unnecessary metrics and auxiliary heads, omit some non-GPU work, and simplify OpenAI Five’s crash-and-restart history.These uncertainties concern what computation counts and how the experiment history is reconstructed.Source: A Compute Usage · p. 25

Compute estimates

• 770±50 PFlops/s·days of GPU optimization compute were used by OpenAI Five when playing the world champions, rising to 820±50 PFlops/s·days before shutdown; Rerun used 150 ± 5 PFlops/s·days.The OpenAI Five estimate covers April 13–22, 2019, while Rerun’s covers May 18–July 12, 2019.Source: A Compute Usage · p. 25

Accounting scope

• The estimate counts optimization compute on GPUs but excludes rollout-worker forward passes, self-play CPUs, and other overhead that together comprise most of the broader training budget.The authors estimate optimization at roughly 30% of dollar cost, with rollout forward passes, self-play, and overhead accounting for the remainder.Source: A Compute Usage · p. 25

Accounting caveats

• OpenAI Five’s compute total omits resources spent before the last restart, including earlier training, studies, ablations, false starts, and imperfect utilization, whereas this caveat does not apply to Rerun.The omission is particularly relevant because surgery blurred the boundary between retained training and restarted runs.Source: A Compute Usage · p. 25

Surgery objective

• The central objective was to resume training a newly shaped policy in a changed environment while matching the original policy’s TrueSkill.The surgery tools covered changes to the environment, model architecture, observation space, and action space.Source: B Surgery · pp. 25, 27

Changing the architecture

• For architecture changes, parameters were initialized so the expanded model initially implemented the same function, while randomized new dimensions could later learn useful behavior.Increasing an LSTM from 2048 to 4096 units required small random weights chosen at the highest scale that did not noticeably reduce TrueSkill.Source: B Surgery · pp. 27–28

Changing the Observation Space

• For additive observation changes, surgery preserved behavior at the encoder or downstream-model interface while allowing zero-initialized weights to learn whether new observations were useful.One example added three float observations encoding spawn times for neutral creeps, bounties, and runes.Source: B Surgery · p. 28

Changing the Environment or Action Space

• For environment or action changes, features were preferably annealed from 0% to 100% of rollout games, with slower retries when TrueSkill declined.Without annealing, model-controlled Buyback initially degraded strategy and overall skill; direct environment updates were sometimes stable enough to adapt smoothly.Source: B Surgery · pp. 28–29

Removing Model Parts and Smooth Training Restart

• Exact policy preservation limited surgery by making most parameter-removal operations impossible, while restarts used a zero learning rate for several hours to accommodate optimizer moments and rollout stabilization.Past opponent versions also had to undergo compatible surgery so the opponent pool would not permanently contain degraded agents.Source: B Surgery · p. 29

Benefits of Surgery

• Surgery enabled feasible testing of many minor game features and improvements that would otherwise have required impractical retraining from scratch.Examples included long-tail items such as Bottle and Rapier, plus observation improvements for stock counts and nonhero modifiers.Source: B Surgery · p. 29

C Hyperparameters

• OpenAI Five’s hyperparameters were adjusted during 10 months based on improvement plateaus, without claiming the resulting settings were optimal.Compute limitations prevented separate experiments for hyperparameter changes, so successful changes were retained and others reverted.Source: C Hyperparameters · p. 29
• Rerun simplified the schedule to four key hyperparameters, including entropy penalty, Team Spirit, and GAE time horizon.The listed changes were applied gradually during training rather than scanned exhaustively.Source: C Hyperparameters · pp. 29–30
• Rerun reached OpenAI Five’s skill level before its planned schedule reached final settings of 840 seconds horizon, 1.0 Team Spirit, and 1e-6 learning rate.The schedule remained far from optimized and was used only in the second iteration of the large experiment.Source: Figure 7 · p. 30, C Hyperparameters · p. 30

D Evaluating agents’ understanding

• The evaluation predicts future game features from the agent’s LSTM state to probe otherwise difficult-to-infer intentions and decisions.The motivation includes distinguishing deliberate objectives from opportunistic short-term actions.Source: D Evaluating agents’ understanding · p. 30
• The auxiliary targets are win probability, end-game net worth rank, and whether the hero will help destroy a specified enemy building soon.Net worth predictions support scripted item-buying decisions for team-shared items such as wards.Source: D Evaluating agents’ understanding · pp. 30, 33
• Despite sparse labels in 30-second segments, backward-propagated segment targets train the predictors to a degree of calibration and accuracy.Ground-truth labels are used when available; otherwise, the model’s end-of-segment prediction supplies the label, while team-objective predictions use a 2-minute discount horizon and are not calibrated probabilities.Source: D Evaluating agents’ understanding · p. 33

D.1 Understanding OpenAI Five Finals

• Win-probability predictions improved throughout training, while correctly reflecting major Finals events such as losing three heroes and later killing key enemies.Version 5,000 already recognized situations associated with eventual victory, and predictions improved as training proceeded.Source: §D.1 · p. 33, Figure 9 · p. 34
• Hero-specific objective predictions matched participation: heroes predicted to attack buildings generally did so, while predictions fell when heroes were diverted elsewhere.The Finals examples include heroes being chased away from towers and others pursuing defenders instead of participating in tower kills.Source: §D.1 · p. 35, Figure 10 · p. 35, Figure 11 · p. 36

D.2 Hero selection

• OpenAI Five’s drafting program precomputed win probabilities for 4,900,896 lineups and used dynamic programming with minimax selection to choose heroes.The method estimates matchup strength from the opening frames, then selects the best available hero on each drafting turn.Source: §D.2 · pp. 35, 37, Figure 12 · p. 37
• The drafting exercise revealed preferences that partly matched humans, including valuing Sniper, but undervalued Earthshaker because its Fissure geometry was difficult to handle.These preferences emerged despite OpenAI Five being trained with randomly selected heroes rather than learning a drafting policy.Source: §D.2 · pp. 35, 37
• At the start of drafting, OpenAI Five estimated Radiant’s win chance at 54% when picking first and 53% when picking second.The higher Radiant estimate agreed with conventional Dota 2 community wisdom but may depend on the available hero pool.Source: §D.2 · p. 37

E Observation Space

• Each hero observed approximately 16,000 inputs per timestep, including per-unit data for up to 189 units, with zero-padding and last-visible values for fogged enemy heroes.Table 4 specifies the full observation space and its handling of visibility and variable unit counts.Source: E Observation Space · p. 37, Table 4 · p. 38
• OpenAI Five used semantic arrays instead of pixels because the study targeted strategic planning and pixel rendering across all training games was computationally infeasible.The representation approximated human-accessible information, while allowing the model to access all encoded information simultaneously each timestep.Source: E Observation Space · pp. 37, 39
• Float observations were normalized using running means and standard deviations, then clipped to (-5, 5) before entering the neural network.Boolean observations were treated as floats during normalization.Source: E Observation Space · p. 39

F Action Space

• The action space factorized each timestep into one primary action and three parameter outputs: Delay, Unit Selection, and Offset.Ignored parameters were masked during optimization, and the parameter dimensions were 4, 189, and 81 respectively.Source: F Action Space · pp. 39, 41
• The model restricted primary actions with filters and averaged 8.1 available choices per timestep in games against OG.Available choices included universal commands, spells, items, and situational actions such as Buyback, Shrine, and Purchase.Source: F Action Space · p. 41
• The factorized action space includes a primary action plus three parameter outputs—Delay (4 dimensions), unit selection (189), and offset (81)—with action-dependent ignored parameters masked during optimization.Source: F Action Space · p. 41
• Across two games, the average number of available actions per hero varied from 8,000 to 80,000 because heroes differed in spells, items, and parameter counts.This measurement came from directly counting available actions at each timestep in the games against Team OG.Source: F Action Space · pp. 41–42
• Distinct mutually exclusive parameter heads handled rare or specialized actions, including Teleport Selection, Caster Offset, Regular Offset, and Ward Placement Offset.Separate heads prevented rare teleport-targeting signals from being drowned out by more common tactical abilities.Source: F Action Space · p. 42

F.1 Scripted Actions

• Some decisions remained scripted because transferring actions to reinforcement learning increased potential skill but required engineering effort and could destabilize exploration and learning.The agent could learn never to take a newly introduced action, preventing it from discovering circumstances where that action helped and disrupting dependent gameplay.Source: §F.1 · p. 42
• By superhuman performance, scripted systems still controlled ability builds, item purchasing, item swapping, and courier control, since removing them offered no necessary benefit.Training randomized ability and non-consumable item schedules for robustness, while heuristics managed consumables, inventory priorities, and courier behavior.Source: §F.1 · pp. 42–43

• Shaped rewards for human-recognized beneficial actions, such as gaining resources and killing enemies, simplify credit assignment toward the ultimate goal of winning.The shaping is loosely modeled on potential-based functions, but their guarantees do not apply here.Source: G Reward Weights · p. 43
• The reward design enforces zero-sum outcomes and exponentially renormalizes non-win/loss rewards over game time to prevent later stages from dominating learning.Each hero’s reward is adjusted by subtracting the average reward of enemy heroes.Source: G Reward Weights · pp. 43, 45
• The shaped reward weights are specified in Table 6, with rewards assigned either to every team hero or only to the acting hero.At team spirit 1.0, team-wide rewards produce five times the total reward of solo rewards.Source: G Reward Weights · p. 43, Table 6 · p. 44
• Team spirit interpolates between individual rewards at τ = 0 and equal sharing across five heroes at τ = 1, addressing multi-agent credit assignment.Lower team spirit reduces early-training gradient variance, while the optimization target is τ = 1 for team success.Source: G Reward Weights · p. 45
• The partial-reward ablation consistently beat a hand-coded scripted agent but incurred a large sample-efficiency penalty versus the shaped-reward baseline.The resulting policy appeared weaker early in games but learned to coordinate fights near the end.Source: G Reward Weights · p. 45

H Neural Network Architecture

• The final joint policy-and-value network used 158,502,815 parameters, with the policy and value function sharing the network and gradients.The value function was computed as a linear projection of the LSTM state.Source: H Neural Network Architecture · pp. 45, 48
• A cross-hero max-pool operation exposed non-identical hero observations to other replicas, while unit embeddings were retained for action heads that select target units.The pool operated over the first 25% of the vector across the five replica networks.Source: H Neural Network Architecture · pp. 47–48
• The architecture cloned observation processing across five hero replicas, pooled structured observations, processed them with a single-layer large LSTM, and projected its outputs into actions.Replica weights were identical, while derived observations could differ across heroes.Source: H Neural Network Architecture · p. 48

I Human Games

• OpenAI Five played games against high-profile human teams, with the games listed in Table 7.The supplied passage identifies the table but does not provide its game results.Source: I Human Games · p. 49

J TrueSkill: Evaluating a Dota 2 Agent Automatically

• TrueSkill evaluated agents using a calibrated pool of reference agents, including a scripted agent near 105 and 83 agents ranging from 0 for random play to 254 for the world-champion version.Reference-agent means were aligned so random play had µ = 0.Source: J TrueSkill: Evaluating a Dota 2 Agent… · p. 49
• During training, dedicated computers evaluated the latest agent against nearby-skill references, updating TrueSkill after 750 games and producing updates about every two hours.Games against agents more than 10 TrueSkill points away were avoided because they corresponded to winrates below 15% or above 85%.Source: J TrueSkill: Evaluating a Dota 2 Agent… · p. 49
• Changing game versions limited TrueSkill consistency because newer agents trained on the evaluation version and could gain metric inflation from hero, client, balance, and action updates.Older agents were upgraded for compatibility, but this did not remove the resulting advantage for newer agents.Source: J TrueSkill: Evaluating a Dota 2 Agent… · p. 49

K.1 Data flow between the training environment and Dota 2

• The Dota 2 environment follows the OpenAI Gym step contract and uses a Go helper process with an attached-debugger gRPC server to configure games and perform steps [59].The server exposes the game over the network, allowing remote processes to interact with it.Source: §K.1 · p. 51
• Lua and gRPC run in parallel, exchanging observations and actions while the game blocks until an action is available.Go channels coordinate the two processes, with Lua sending current state to gRPC and receiving the selected action.Source: §K.1 · p. 51
• Packaging the gRPC-backed game environment in Docker enabled many isolated instances per machine and simplified setup, resetting, and remote use.The design significantly improved researcher productivity during iteration and debugging.Source: §K.1 · p. 51

L Reaction time

• Frameskipping reduced interaction to 7.5 frames per second, while asynchronous action offsets let computation overlap with game execution at the cost of reaction time.The action space included a delay for precise timing, but the model performed better by acting at the start of each frameskip.Source: L Reaction time · p. 52
• 167–267ms was the resulting reaction-time range, randomly determined by when new information arrived within the frameskip.This was compared with a 250ms human reaction time measured in controlled experiments, likely an underestimate for Dota gameplay [26].Source: Figure 19 · p. 51, L Reaction time · p. 52

M Scale and Data Quality Ablation Details

• The scale and data-quality experiments monitored TrueSkill and speedup using smaller-batch Dota 2 runs because full OpenAI Five-sized scans were prohibitively expensive.Baseline variation was estimated at approximately 2 TrueSkill points across four runs, and speedup uncertainty combined numerator and denominator uncertainty in quadrature.Source: M Scale and Data Quality Ablation… · p. 52, Figure 20 · p. 53

M.1 Batch Size

• Larger batch sizes significantly speed training, but scaling is sublinear in resources and benefits later training more than earlier training.The comparison remains inconclusive for most of OpenAI Five’s training because TrueSkill 175 is still early relative to its eventual score above 250.Source: Figure 21 · p. 54

M.2 Sample Quality — Staleness

• Staleness negatively affects training speed, with severe losses when data is delayed by more than a few parameter versions.The system therefore attempted to keep staleness as low as possible.Source: §M.2 · p. 55, Figure 22 · p. 55

M.3 Sample Quality — Sampling and Sample Reuse

• Sample reuse depends on asynchronous data flow and rollout-to-optimizer capacity, with random buffer sampling causing reuse 1 to repeat some samples while omitting others.The experience buffer holds 4096 samples, and reuse is measured as batch consumption divided by buffer intake; increasing rollout workers can accelerate training when optimizers reuse data.Source: §M.3 · pp. 56, 58
• Increasing sample reuse significantly decreased performance; sample reuse below one produced a minor but significant improvement over sample reuse 1, and reuse around 6.3 converged below 75 TrueSkill.Source: §M.3 · pp. 56, 58, Figure 24 · p. 57

N Self-play

• OpenAI Five used self-play without human gameplay data, playing 80% of games against the latest parameters and 20% against past versions to promote robust strategies and avoid strategy collapse.Past opponents were sampled dynamically by quality score, with new versions added every 10 iterations and scores updated when the current agent defeated them.Source: N Self-play · p. 59
• Self-play opponent diversity reflected learning progress: rapidly improving agents assigned low scores to older opponents, whereas slower progress maintained a broader opponent distribution.Later versions included more past opponents, suggesting that skill progression slowed after a warmup period.Source: N Self-play · p. 59, Figure 27 · p. 60

O Exploration

• Exploration was encouraged through both loss shaping and randomized training environments.The paper motivates randomization by improving discovery of long-horizon rewards, avoiding repetitive-environment local minima, and exposing agents to situations needed for robustness against human strategies.Source: O Exploration · p. 59, §O.2 · p. 62

O.1 Loss function

• Entropy bonus 0.01 was the best tested setting, while team spirit 0 performed best initially before team spirits 0.3 and 0.5 overtook it.Lower entropy impaired exploration, higher entropy made actions too random, and the authors hypothesized that team spirit 1.0 would become best later as training optimized the actual reward.Source: §O.1 · pp. 59, 62, Figure 28 · p. 61, Figure 29 · p. 61

O.2 Environment Randomization

• Environment randomization covered initial states, lane assignments, Roshan health, hero lineups, and item selections to broaden exploration and robustness, although lane randomization provided little benefit.Hero and item variation improved exposure to alternative strategies and human play, while additional heroes caused only a modest training slowdown and lane assignments were not necessary in the ablation.Source: §O.2 · pp. 62–63, Figure 30 · p. 62

P Hero Pool Size

• The agent could not play all game heroes, and hero-pool comparisons sampled five heroes independently for each team while evaluating only on the smallest pool.This setup means smaller-pool runs are closer to the five-hero test distribution and may perform better.Source: P Hero Pool Size · p. 63
• Training with 80 heroes ran approximately 20% slower than with the base 17 heroes during early training, suggesting similar high skill may require approximately 20% more training time.The experiment compared only the very early stages of training, so the speedup factor could become worse later.Source: P Hero Pool Size · p. 63

Q.1 Manually Tuned Hyperparameters

• Under time pressure before The International, manually searching hyperparameters produced high-variance experiments, and the team concluded human intuition was not the best tuning method.The agent was already very good but lost both games at the event, motivating exploration of options with potentially surprising improvements.Source: §Q.1 · p. 65

Q.2 Zero Team Spirit Embedding

• Zeroing 128 learned team-spirit embedding parameters increased performance to about a 55% winrate, despite leaving shaped reward approximately unchanged.The optimizers may have missed this direction because the shaped reward did not reflect the winrate improvement.Source: §Q.2 · p. 65
• Random perturbations of the embedding parameters consistently hurt performance, whereas moving specifically toward zero improved it, indicating that zero was unexpectedly special.The embedding had originally been retained as a vestigial fixed observation component after randomized team-spirit values were abandoned.Source: §Q.2 · pp. 65–66

Q.3 Learning path dependency

• Adding Divine Rapier in Rerun triggered a negative feedback loop that reduced skill, while banning it allowed TrueSkill to continue improving.The effect may depend on learning order: Rapier was added only after team spirit reached 1, and its transferable high-value item increased reward variance enough to hinder reliable value-function learning.Source: §Q.3 · p. 66