PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Self-Play Learning in Go

David J. Wu

arXiv:1902.10565v5cs.LGstat.ML

TL;DR

KataGo addresses the computation required for strong Go self-play learning. It combines general and Go-specific improvements to neural-net-guided MCTS self-play, and surpasses ELF OpenGo after 19 days on about 27 GPUs. The results indicate an efficiency gap between general AlphaZero-like methods and what self-play can achieve with domain-specific techniques.

  • Problem

    AlphaZero and ELF OpenGo achieved strong Go performance but required thousands of accelerators, motivating more computation-efficient self-play learning.

    Source: §1 · p. 1

  • Method

    KataGo introduces general and domain-specific improvements to neural-net-guided MCTS self-play, while learning from random play without outside strategic knowledge or preexisting data.

    Source: §1 · pp. 1–2

  • Results

    About 1.4 GPU-years on 27 V100 GPUs over 19 days let KataGo surpass ELF OpenGo, a roughly 50x reduction in computation.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Domain-specific gains alongside general improvements indicate a significant efficiency gap between AlphaZero-like methods and what is possible from self-play.

    Source: §1 · p. 2, §6 · p. 14

    The authors could not replicate ELF or AlphaZero infrastructure or run longer ablations, limiting precise comparisons.

    Source: §5.2 · p. 14

Abstract

from arXiv · show

By introducing several improvements to the AlphaZero process and architecture, we greatly accelerate self-play learning in Go, achieving a 50x reduction in computation over comparable methods. Like AlphaZero and replications such as ELF OpenGo and Leela Zero, our bot KataGo only learns from neural-net-guided Monte Carlo tree search self-play. But whereas AlphaZero required thousands of TPUs over several days and ELF required thousands of GPUs over two weeks, KataGo surpasses ELF's final model after only 19 days on fewer than 30 GPUs. Much of the speedup involves non-domain-specific improvements that might directly transfer to other problems. Further gains from domain-specific techniques reveal the remaining efficiency gap between the best methods and purely general methods such as AlphaZero. Our work is a step towards making learning in state spaces as large as Go possible without large-scale computational resources.

1 Introduction

KataGo introduces efficiency improvements to AlphaZero-style self-play, surpassing ELF OpenGo after 19 days on about 27 GPUs. Its domain-independent techniques may transfer to other learning problems, while domain-specific gains expose an efficiency gap in general methods.

1 Introduction

  • About 1.4 GPU-years on 27 V100 GPUs over 19 days let KataGo surpass ELF OpenGo, a roughly 50x reduction in computation.The comparison is against ELF OpenGo’s approximately 74 GPU-years.Source: §1 · p. 1
  • KataGo starts from random play and uses no outside strategic knowledge or preexisting data.Its learning remains based on self-play rather than supplied strategic knowledge or data.Source: §1 · p. 1
  • Domain-independent improvements include playout cap randomization, policy target pruning, global pooling, and auxiliary policy targets.The authors suggest these techniques might transfer to other AlphaZero-like learning or reinforcement learning.Source: §1 · p. 2
  • Domain-specific ownership and score targets, plus game-specific input features, provide further gains beyond basic AlphaZero-like methods.The authors frame these findings as evidence of a remaining efficiency gap in self-play learning.Source: §1 · p. 2
  • The paper reports comparisons and ablations to show that its techniques each improve learning and contribute to the final result.The authors note that Leela Zero training data availability and possible use of stronger ELF data complicate comparison.Source: §1 · p. 2

2 Basic Architecture and Parameters

KataGo trains a convolutional residual network from MCTS self-play, using policy and outcome-value targets. Its search combines neural policy priors and subtree utilities, while training uses auxiliary implementation choices including weight averaging and staged network scaling.

2 Basic Architecture and Parameters

  • KataGo self-play uses neural-net-guided MCTS, growing a game tree through repeated playouts.Search selects children using predicted utility, policy prior, and prior visit counts.Source: §2 · p. 2
  • The network grows from (6, 96) to (10, 128), (15, 192), and (20, 256), with the final size matching AlphaZero and ELF.The dimensions are residual-block count and channel count; the transitions occur as training progresses.Source: §2 · p. 3
  • The convolutional residual network predicts probable good moves and the eventual win-or-loss outcome from sampled self-play positions.The loss compares the policy prediction with a playout-derived target and the value prediction with the game outcome; auxiliary-head terms are also added.Source: §2 · p. 3
  • Weight snapshots are exponentially averaged every four snapshots, and a candidate must win at least 100 of 200 games against the current net to become the self-play net.The exponential moving average uses decay 0.75.Source: §2 · p. 3
  • The main run lasted 19 days, used at most 28 V100 GPUs, and generated about 241 million samples across 4.2 million games.The reported peak GPU count is a maximum, not an average.Source: §2 · p. 3

3 Major General Improvements

KataGo’s general improvements change how self-play searches produce training targets and how the network uses context. The techniques address policy–value tension, remove exploration-driven noise from policy targets, condition convolutions on global information, and regularize policy learning with future-action targets.

3.1 Playout Cap Randomization

  • Playout cap randomization outperforms a variety of fixed playout counts while balancing value learning’s need for more games against policy learning’s need for larger searches.A small fraction of turns use full searches; most use faster searches, and only full-search turns enter training.Source: §3.1 · p. 4

3.2 Forced Playouts and Policy Target Pruning

  • Policy target pruning subtracts exploration-driven forced playouts unless a move’s PUCT urgency would otherwise justify them, decoupling the policy target from MCTS exploration.Forced playouts ensure noisy root moves receive search; pruning removes extra bad-move playouts from the target, and Figure 1 shows reduced policy mass on many bad edge moves.Source: §3.2 · p. 5, Figure 1 · p. 5

3.3 Global Pooling

  • Global pooling lets convolutional layers condition on global context that their limited perceptual radius would otherwise make hard or impossible to access.KataGo pools channel means, width-scaled means, and maxima, then uses a learned bias structure; the authors report improved later-stage training.Source: §3.3 · p. 6

3.4 Auxiliary Policy Targets

  • An auxiliary policy target predicts the opponent’s next-turn reply to regularize learning, weighted only 0.15 relative to the actual policy and never used for play.The authors report a modest but clear benefit and suggest the approach could apply to broader reinforcement-learning tasks.Source: §3.4 · p. 7

4 Major Domain-Specific Improvements

KataGo’s auxiliary ownership and score targets improve learning efficiency, while Go-specific input features and endgame optimizations contribute additional gains. The authors also propose that predicting subevents can provide more localized feedback, though this heuristic may not always apply.

4.1 Auxiliary Ownership and Score Targets

  • Decomposing binary game outcomes into ownership and score-difference targets is intended to improve regularization and provide more localized feedback for credit assignment.The authors suggest that ownership errors direct gradients toward mispredicted board regions, potentially reducing the samples needed for correct updates.Source: §4.1 · p. 8
  • Auxiliary ownership and score targets noticeably improve learning efficiency, including in shorter runs reaching strength similar to human-professional play.The authors report that the benefit persists beyond the earliest learning stages.Source: §4.1 · p. 8
  • The authors propose predicting subevents of a target when they can be expressed as, or are highly correlated with, separate components, but note this heuristic may not always apply.Their proposed mechanism is sharper, more localized gradients from mispredicted subevents.Source: §4.1 · p. 9

4.2 Game-specific Features

  • Go-specific input features and endgame optimizations noticeably contribute to learning speed, but account for only a small fraction of KataGo’s total improvement.The ablation disables the endgame optimizations and all input features except stones, previous-move history, and game rules.Source: §4.2 · pp. 9–10

5 Results

KataGo achieved substantially greater learning efficiency than ELF and Leela Zero, while ablations found gains from both general and Go-specific techniques. The comparisons also carry limits: Leela Zero’s compute estimate omits some costs, and the ablation runs were shorter than the full run.

5.1 Testing Versus ELF and Leela Zero

  • Leela Zero’s compute comparison is approximate: some training data was unavailable, and the cost of stronger ELF prototype data it used was not counted.The authors also note that these stronger ELF nets likely caused Leela Zero to learn faster than it would unaided.Source: §5.1 · p. 10
  • KataGo outperformed ELF in learning efficiency by about 50x and Leela Zero by a factor of 10, despite matching ELF’s network size and similarly short training time.The Figure 4 comparison uses estimated self-play compute; Leela Zero’s costs are highly approximate.Source: §5.1 · pp. 10–11, Figure 4 · p. 11
  • Across three 400-game matches spanning fixed-visit and fixed-wall-clock settings, KataGo defeated ELF in all three.The matches tested both low and high playouts, as well as fixed-search and fixed-time settings.Source: Table 1 · p. 11, §5.1 · p. 11

5.2 Ablation Runs

  • Playout cap randomization clearly outperformed a wide range of fixed playout caps, consistent with relieving tension between value and policy targets.The ablation runs were shorter than the full run.Source: Figure 5 · p. 12, §5.2 · p. 12
  • Global pooling noticeably improved learning efficiency, while forced playouts with policy target pruning and auxiliary policy targets delivered smaller clear gains.The gains were little evident early in the run and appeared to increase later.Source: Figure 6 · p. 13, §5.2 · p. 14
  • Removing ownership and score targets or Go-specific input features reduced learning efficiency, with domain-specific methods accounting for only part of KataGo’s total speedup.The ownership and score results also support the value, at least in Go, of regularizing by predicting target subcomponents.Source: Figure 7 · p. 13, §5.2 · p. 14
  • Ablation acceleration factors multiply to approximately 9.1x, which the authors suspect underestimates the true speedup because some effects grew during training.The ablation runs were shorter than the full run, and more extensive comparisons were beyond the authors’ resources.Source: Table 2 · p. 14, §5.2 · p. 14

6 Conclusions And Future Work

KataGo reaches strength competitive with leading AlphaZero replications while using no external data, and does so with much greater efficiency. Its domain-specific gains also point to a remaining gap between basic AlphaZero-like training and possible improvements to general methods.

6 Conclusions And Future Work

  • KataGo begins from random play without external data, yet reaches a level competitive with top AlphaZero replications at much greater efficiency.The authors present its domain-specific gains as evidence of a remaining gap between basic AlphaZero-like training and what may be possible.Source: §6 · p. 14
  • The authors hope this work lays a foundation for further improvements in reinforcement-learning data efficiency.They also suggest principles and possible avenues for improving general methods.Source: §6 · p. 15

Appendix A Neural Net Inputs and Architecture

The appendix details KataGo’s neural-net inputs, residual-block trunk, and policy and value heads.

Appendix A Neural Net Inputs and Architecture

  • The network has two input tensors feeding a residual-block trunk, with policy and value heads attached at the end.Each head has several outputs and subcomponents.Source: Appendix A · p. 17

A.1 Inputs

KataGo processes spatial and overall game-state inputs through a residual trunk that includes global pooling, then predicts policies for both players.

A.1 Inputs

  • The network receives 18 binary features at each board location and a 10-value vector describing overall game state.The spatial tensor has dimensions b × b × 18 for board widths from 9 to 19.Source: §A.1 · p. 17

A.2 Global Pooling

  • Global pooling summarizes channels using their means and maxima, then uses pooled values to form biases for spatial channels.The pooled output has 3c values; its board-size adjustment supports weights shared across board sizes.Source: §A.2 · pp. 17–18

A.3 Trunk

  • The trunk begins with spatial convolution combined with a bias from the overall game-state input, then stacks residual blocks, some using global pooling.Residual blocks include skip connections that add the convolution result elementwise to the block input.Source: §A.3 · pp. 18–19

A.4 Policy Head

  • The policy head outputs board-location move distributions for the current player and the opponent’s subsequent turn, with separate logits for passing.A global-pooling bias structure links its G and P branches.Source: §A.4 · p. 19

A.5 Value Head

The value head predicts game outcomes, score-related quantities, and board ownership; the appendix also describes network sizes and training losses.

A.5 Value Head

  • The value head predicts game outcome, final-score mean and uncertainty, ownership at each board location, and a distribution over final score differences.The variance prediction for MCTS root values is included for future research but has negligible training weight.Source: §A.5 · pp. 19–20

A.6 Neural Net Parameters

  • Four neural-net sizes were used, with two or three global-pooling residual blocks replacing ordinary blocks at regular intervals.Table 5 summarizes the architectural constants for each size.Source: §A.6 · p. 21, Table 5 · p. 21

Appendix B Loss Function

  • Mean and standard-deviation self-prediction losses regularize score estimates toward the corresponding moments of the network’s full score-belief distribution.The Huber loss avoids some divergence cases from large errors just after initialization.Source: Appendix B · p. 23
  • The MCTS root-value variance prediction term was used only with negligible or zero weight.The authors intended the term for future MCTS research.Source: Appendix B · p. 23
  • Auxiliary loss-term coefficients were mostly guessed rather than carefully tuned because only a limited number of test runs were affordable.The chosen weights were intended to yield gradients and loss values around 10–40% of those from the main policy and value terms.Source: Appendix B · p. 23

Appendix C Training Details

KataGo’s main run generated training data across four network sizes over 19 days, while training used a moving sample window and adjusted learning rates. Tables 6 and 7 summarize training time and strength for the strongest network of each size.

Appendix C Training Details

  • 19 days of training generated about 241 million samples across 4.2 million games and four neural-net sizes.Table 6 summarizes training time and Table 7 the approximate strength of the strongest net at each size; the supplied captions do not give their values.Source: Appendix C · p. 24, Table 6 · p. 24, Table 7 · p. 24
  • Training used a per-sample learning rate of 6e-5, lowered to 2e-5 for the first 5 million samples and to 6e-6 after about 17.5 days.The early reduction was intended to reduce instability from large initial gradients; the later reduction was for final-strength tuning.Source: Appendix C · p. 24
  • Samples were drawn uniformly from a moving window whose size followed a sublinear curve, with c = 250,000, α = 0.75, and β = 0.4.The curve was rescaled so that f(c) = c and f′(c) = β.Source: Appendix C · p. 24

Appendix D Game Randomization and Termination

KataGo randomizes game conditions and play to broaden training experience, and continues self-play games without resignation using reduced search effort in near-decided positions. Candidate networks pass a lightweight gating test before becoming the self-play net.

Appendix D Game Randomization and Termination

  • Rules, board size, komi, and handicap starts were randomized to broaden the range of game conditions represented in training.Rules varied between superko and suicide-move options; 37.5% of games used 19x19 boards, and 5% were handicap games.Source: Appendix D · p. 25
  • Opening moves used policy-based sampling, while later moves followed a temperature-scaled, target-pruned MCTS distribution.The temperature began at 0.8 and decayed smoothly to 0.2 with a board-width-based half-life, approximating AlphaZero or ELF scaling in the first 30 moves.Source: Appendix D · p. 25
  • In 2.5% of positions, KataGo searched an alternative move to add training examples for responding to or refuting moves a full search might not choose.The branched position received a full search for a policy sample; game outcome supplied the outcome target, while score and ownership targets were left unconstrained.Source: Appendix D · p. 25
  • Five percent of games branched into several randomly chosen opening moves, ensuring some training games had highly unusual openings.The game then continued to completion, with komi adjusted for fairness.Source: Appendix D · p. 25
  • Game randomization was intended to expose KataGo to alternate rules, board sizes, komi, handicaps, and unusual positions, primarily for human amateur game analysis.The authors state these settings likely have only a limited effect on overall learning efficiency and strength, apart from introducing a minimum amount of entropy.Source: Appendix D · pp. 25–26
  • Games continued to completion without resignation; after five consecutive turns below a 5% losing-side winrate, visit caps were reduced and samples downweighted.This costs slightly more than resignation but produces cleaner, less biased targets and makes final ownership and score targets easy to compute.Source: Appendix D · p. 26

Appendix E Gating

  • Candidates needed at least 100 wins in 200 games against the current net to pass lightweight gating.Gating retained randomized rules and board size, fixed komi at 7.5, and disabled handicaps and branching; resignation was enabled.Source: Appendix E · p. 27

Appendix F Score Maximization

KataGo supplements win/loss utility with a dynamic score-difference utility, supporting score-sensitive play for game analysis and handicap games. At search start, it recenters this utility on the neural net’s predicted score and maximizes the sum of score and win utilities.

Appendix F Score Maximization

  • KataGo assigns nonzero utility to maximizing a dynamic function of score difference, in addition to win/loss utility, to support human game analysis and handicap play.The function saturates far from its center, limiting utility for unlikely large gains or losses while rewarding likely score improvements near the center.Source: Appendix F · p. 28
  • Figure 8 plots total utility against score difference with x0 = 0, b = 19, and cscore = 0.5.cscore controls the relative importance of score, while b is board width.Source: Figure 8 · p. 28, Appendix F · p. 28
  • At search start, x0 is set to the neural net’s predicted mean score, and search maximizes the sum of win and score utilities.The net’s predicted score mean and variance are used to estimate score utility, with the integral computed by interpolation in a precomputed lookup table.Source: Appendix F · p. 28

Related papers

Loading 1902.10565v5…