ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero

1. Introduction

• The work targets a research-access gap: AlphaGo-family advances require millions of selfplay games and thousands of TPUs, while unavailable code and models make reproduction and extension difficult.The authors aim to provide resources and exploratory insights for AI research and Go communities.Source: §1 · pp. 1–2
• A 20-block ELF OpenGo model reached superhuman performance after training on 2,000 GPUs for 9 days, with performance described as arguably comparable to Silver et al. (2017) and Silver et al. (2018).The release includes training code, pretrained models, 20 million selfplay games over 1.5 million minibatches, and auxiliary data.Source: §1 · p. 1
• Training strength varies substantially against other models even at reduced learning rates, and ladder moves requiring lookahead are learned slowly and never fully mastered.The model improves at learning mid-game and end-game moves at nearly identical rates, unlike typical tabular RL behavior.Source: §1 · p. 1
• Doubling gameplay rollouts still adds approximately 200 ELO to the final model, indicating that model capacity constrains its strength.The authors also report significant, sustained rollout-related improvement when playing white but diminishing returns when playing black.Source: §1 · p. 2

2. Related work

• Go uses alternating stone placement on a 19-by-19 grid, with captures based on liberties and Chinese scoring based on occupied or surrounded positions.White typically receives komi to compensate for moving second.Source: §2 · p. 2

2.1. Early work

• Classical game-tree search faces Go’s branching factor of up to 362 and hundreds of moves per game; MCTS instead balances exploration and exploitation, while deep-learning approaches use convolutional networks and selfplay.AlphaGo combined policy networks trained with human games and reinforcement learning, value networks trained via selfplay, and distributed MCTS.Source: §2.1 · p. 2

2.2. AlphaGo Zero and AlphaZero

• AlphaGo Zero and AlphaZero train from game rules without external information, using a residual network for value and move-prior predictions alongside MCTS to improve policies.MCTS selects moves using visit frequencies, with proportional selection in early training and greedy selection in late training and gameplay inference.Source: §2.2 · pp. 2–3

2.3. Contemporary implementations

• ELF OpenGo was publicly verified as superhuman and, under equal hardware constraints, was described as the strongest open-source Go AI at the time of writing.Other open-source implementations cited include LeelaZero, PhoenixGo, AQ, and MiniGo.Source: §2.3 · p. 3

2.4. Understanding AlphaGo Zero and AlphaZero

• Prior work sought to analyze AlphaGo Zero and AlphaZero and apply similar algorithms to other domains; the authors offer ELF OpenGo as an accelerator for that research.The related-work overview also describes training data sampled from a replay buffer and model updates evaluated through selfplay matches.Source: §2.4 · p. 3, §2.2 · p. 3

3. ELF OpenGo

• ELF OpenGo follows AlphaZero-style training with GPU selfplay and training, using colocated workers and asynchronous selfplay to improve throughput and handle per-game latency.The system aims to faithfully reimplement AGZ and AZ while addressing ambiguities and enabling operation on commodity hardware.Source: §3 · p. 3
• A direct total-resource comparison with AGZ is difficult because its hardware details are not provided.The authors note that their hardware setup uses NVIDIA V100 GPUs, and suggest comparable performance on most NVIDIA GPUs with Tensor Cores.Source: §3 · p. 3
• The implementation uses 2,000 selfplay GPUs and 8 training GPUs, a 500,000-game replay buffer, 1,600 selfplay rollouts per move, cpuct 1.5, and virtual loss 1.0.Training uses temperature 1 for all moves; the authors note that several settings were unspecified in prior papers.Source: §3 · p. 4
• The 256-filter, 20-block model trained for 1.5 million minibatches, or roughly 3 billion game states, with about 20 million selfplay games; it achieved superhuman performance in 9 days.The complete run took around 16 days of wall-clock time.Source: §3 · p. 4

4.1. Prototype model

• 20:0 was the prototype model’s record against four top-30 professional players.The final model was approximately 150 ELO stronger than the prototype.Source: §4.1 · p. 4, §4 · p. 4
• 98.2% was the prototype model’s win rate against LeelaZero, corresponding to an approximately 700 ELO skill differential.Both AIs played under a 50-second-per-move time limit.Source: §4.1 · pp. 4–5

4.2. Training analysis

• Nash averaging captured skill degradations between minibatches 300,000 and 400,000 that selfplay ELO missed, while denser selfplay comparisons produced more inflated ratings.The ratings compared models sampled at different minibatch intervals.Source: §4.2 · p. 5
• The initial value-loss dip followed white-win overestimation and premature black resignations; balancing black- and white-win replay sampling restored the loss, which then stayed constant.The imbalance reduced game diversity and could lead to a negative feedback loop and overfitting.Source: §4.2 · p. 5
• Strength increased overall, reaching parity with the prototype after roughly 1 million minibatches and a 60% win rate at training’s end, but remained highly variable even after learning-rate reductions.Similar trends against two earlier prototype variants made the upward trend robust to opponent choice.Source: §4.2 · p. 5
• Similar performance could potentially be achieved with 250,000 rather than 500,000 minibatches at each learning rate.This possibility was inferred from win rates against the prototype model.Source: §4.2 · p. 6
• Early-game move agreement lagged mid- and end-game agreement, initially because of overfitting, while mid- and end-game match-rate progressions were nearly identical.Professional-game match rates converged to different levels because earlier moves allow more ambiguity.Source: §4.2 · p. 6
• Ladder handling fluctuated during training, worsened after learning-rate reductions, and still produced mistakes late in training, although 6,400 rather than 1,600 rollouts generally improved play.Increasing rollouts improved ladder play but did not eliminate mistakes.Source: §4.2 · p. 6

5. Ablation study

• The ablations probe MCTS and AlphaZero-style training; training analyses also show earlier learning of end-game than opening moves and persistent ladder mistakes.The ladder examples illustrate that whether a ladder is advantageous can depend on stones distant from its start.Source: §5 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

5.1. PUCT constant

• Among the tested PUCT constants, cpuct = 1.5 was the most performant; values that are too low limit exploration, while values that are too high reduce effective search depth.The authors describe 1.5 as a suitable choice based on preliminary experimentation.Source: §5.1 · p. 7

5.2. MCTS virtual loss

• A virtual loss constant of 1 is motivated as each thread temporarily assuming a loss along its current rollout, and a sweep found this default reasonable.Virtual loss adds a temporary fixed loss to a node to discourage concurrent threads from visiting it.Source: §5.2 · p. 7, Table 1 · p. 7

5.3. AlphaGo Zero vs. AlphaZero

• AZ-trained models won 100:0 against AGZ-trained models after both trained for 12 hours, indicating better performance in this comparison.The experiment used 64-filter, 5-block models trained from scratch.Source: §5.3 · pp. 7–8

5.4. MCTS rollout count

• Doubling rollouts produced an 80–90% win rate as White (≈250–400 additional ELO) but only 55–75% as Black (≈35–200 additional ELO).The benefit for Black shrank toward 50% at higher rollout counts; a separate prototype/final-model comparison also found an ≈200 ELO boost.Source: Figure 7 · p. 8, §5.4 · p. 8

6. Discussion

• Ladder moves are learned slowly and never fully mastered, despite being easy for human beginners to understand after a few examples.The authors suspect convolutional networks lack the right inductive bias for ladders and instead memorize specific situations.Source: §6 · p. 8
• ELF OpenGo requires orders of magnitude more games than professional human players to reach the same performance level.Source: §6 · p. 8
• Significant training variance persists even when learning rates are reduced, suggesting the method remains incompletely understood and may require substantial tuning.At a learning rate of 10^-5, training to 2 million minibatches yielded minimal strength improvement and high variance; the authors suggest insufficient selfplay diversity as one possible cause.Source: §6 · p. 8
• Moves 61–180 are learned only slightly faster than moves 1–60, a weak endgame-to-opening progression compared with typical RL behavior.The authors suggest the model may quickly predict easy situations and then focus on difficult cases.Source: §6 · p. 8

7. Conclusion

• The paper provides an AlphaZero reimplementation and superhuman Go engine, with code, models, selfplay datasets, and auxiliary data publicly available.It examines training-strength variance, ladder learning, game-stage learning rates, and previously ambiguous parameters.Source: §7 · p. 8
• Further work on sample efficiency and reducing training volatility remains an opportunity for deep reinforcement learning research.Source: §7 · p. 8

A. Detailed system and software design of ELF OpenGo

• ELF OpenGo builds on the DarkForest Go engine and the ELF reinforcement learning platform.Source: ELF OpenGo · p. 11

A.1. Distributed platform

• The modified ELF platform, called ELF++, adds distributed asynchronous workflows supporting up to 2,000 clients.Source: §A.1 · p. 11
• ELF++ also offers a Python adapter for environments such as OpenAI Gym and autobatching for shared neural-network inference across game simulators.Source: §A.1 · p. 11

A.2. Go engine

• The integrated DarkForest engine handles Go game dynamics at approximately 1.2 microseconds per move.Source: §A.2 · p. 11
• The engine executes the game logic for ELF OpenGo.Source: §A.2 · p. 11
• Terminal game states are scored using Tromp-Taylor rules.Source: §A.2 · p. 11

A.3. Training system

• ELF OpenGo largely replicates AlphaZero’s training architecture, but its training setup differs in response to compute capabilities.Source: §A.3 · p. 11
• With GPU selfplay and half precision, ELF OpenGo’s throughput is roughly 1 move per second, versus 2.5 moves per second for AlphaGo Zero’s TPU worker.The comparison is for a 256-filter, 20-block model with 1,600 MCTS rollouts.Source: §A.3 · p. 11
• Switching to asynchronous training yielded over 5x the synchronous selfplay throughput and increased the selfplay-games-to-minibatches ratio by roughly 1.5x.The higher ratio helps prevent overfitting.Source: §A.3 · p. 12
• Asynchronous selfplays can combine consecutive models, reducing within-game move coherence; the authors report this effect is mild against a replay buffer containing games from over 25 models.Source: §A.3 · p. 12

A.4. Miscellany

• The replay buffer holds 500,000 games in 50 queues, with a 200-game minimum per queue to reduce the risk of rapid overfitting on too few games.Source: §A.4 · p. 12
• Dynamic resignation calibration targets a 5% false-positive rate; resignation saves computation and shifts selfplay toward midgame and opening positions.Source: §A.4 · p. 12

• MCTS augmented with only a value heuristic has a relatively low skill ceiling in Go.Source: B. Auxiliary dataset details · p. 13

B.1. Human games dataset

• 1,000 professional games from the Gogod database, spanning 2011 to 2015, were randomly sampled for the human game dataset.Source: §B.1 · p. 13

B.2. Ladder dataset

• 100 CGOS games containing ladder scenarios were reduced to decisive ladder-related states and augmented eight-fold through rotations and reflections.Source: §B.2 · p. 13

C. Practical lessons

• Stale batch-normalization moment buffers biased estimates and hurt performance; averaging moments from 50 minibatches every 1,000 training minibatches removed the bias and improved training performance and consistency.The moment calculation is a variant of postprocessing suggested by Ioffe & Szegedy (2015); the passage says this technique was configurable in PyTorch as of December 2018.Source: C. Practical lessons · p. 13
• An unintentional ablation with cross-entropy coefficient set to 1 362 let the value network reach around amateur dan strength, but further progress was extremely slow, likely due to minimal policy-network gradients.Source: C. Practical lessons · p. 13