Bayesian Optimization in AlphaGo

1 Introduction

• Tuning focused on game-playing parameters because neural-network tuning strategies were more robust and human knowledge was more limited for game playing.The tuned components included MCTS controls, roll-out/value-network mixing, network evaluation parameters, and search time per move.Source: §1 · p. 1
• Bayesian optimization reduced manual tuning effort, improved AlphaGo’s playing strength, and revealed insights into component contributions.One example was the relative value of fast Monte Carlo roll-outs versus value-network board evaluation.Source: §1 · p. 1
• The win-rate has no analytically tractable relationship to hyper-parameters, so it was estimated by playing self-play games against a fixed baseline.Win-rate is a function of the agent’s hyper-parameters, and each game produces a win-or-loss Bernoulli outcome.Source: §1 · pp. 1–2
• BayesElo was also used across games with different AlphaGo versions to estimate a scalar strength value for each agent.Source: §1 · p. 2
• A grid search over six hyper-parameters with five values each would have taken 8.3 days using 400 GPUs.Each value required 1,000 self-play games, with an estimated 6.7 hours per value.Source: §1 · p. 2

2 Methods

• A Gaussian-process posterior over the unknown objective feeds expected improvement, which balances exploitation and exploration when selecting the next query.The method uses Gaussian-process priors and expected improvement; the target is typically the best past observation or posterior mean at past queries.Source: §2 · p. 2
• Bayesian optimization was chosen because win-rate is non-differentiable, costly to evaluate, and has a moderate number of hyper-parameters.Source: §2 · p. 2
• The modified Spearmint procedure updates the GP with completed self-play outcomes, launches games at an EI-maximizing setting, and returns the setting with the highest posterior mean.Source: §2 · p. 3
• Self-play evaluations use win-rates from 50 games to reduce evaluation cost at the price of higher noise.The win-rate is an average of Bernoulli outcomes, and the GP receives the observed win-rate and an estimated standard deviation under a nonstationary Gaussian observation-noise model.Source: §2 · p. 3
• The visualization plots posterior win-rate mean and variance for individual parameters and pairs, helping identify influential parameters and correlations for later tuning.Figure 2 shows individual-parameter views with other parameters fixed and a paired-parameter posterior mean.Source: §2 · p. 3, Figure 2 · p. 3
• Figure 3 tracks observed and maximum expected win-rates across optimization steps.Source: Figure 3 · p. 4

3.1 Task 1: Tuning MCTS hyper-parameters

• MCTS tuning raised win-rate from 50% to 63.2% and 64.4% (94 and 103 Elo gains) in two pre-match design iterations.Optimized hyper-parameters were often correlated and not reachable through element-wise hand-tuning; tuning also increasingly favored value-network estimates over roll-outs.Source: §3.1 · p. 4

3.2 Task 2: Tuning fast AlphaGo players for data generation

• Tuning four key fast-player versions yielded Elo gains of 300, 285, 145, and 129.Under the short-search-time setting, optimal hyper-parameters differed substantially, making manual tuning prohibitive without prior knowledge.Source: §3.2 · p. 5

3.3 Task 3: Tuning on TPUs

• TPU migration changed existing hyper-parameter optima and introduced new distributed-implementation parameters; Bayesian optimization yielded further large Elo improvements.TPUs provided faster network evaluation than GPUs, and AlphaGo’s performance was boosted after the hardware migration.Source: §3.3 · p. 5

3.4 Task 4: Developing and tuning a dynamic mixing ratio formula

• Reparameterizing the dynamic mixer around its ratio at move 150 decorrelated r0 and a; tuning set the visit-count coefficient b to zero.The resulting formula weighted value networks more at the beginning of a game and less toward the end.Source: §3.4 · p. 6

3.5 Task 5: Tuning a time control formula

• 66.5% win-rate against the default time setting with a fixed 30-second search time per move followed time-control tuning.The optimized formula was for a 2-hour main play time and 60-second byoyomi.Source: Figure 5 · p. 5, §3.5 · p. 6

• Bayesian optimization made automatic tuning of AlphaGo’s game-playing hyper-parameters possible, contributing significantly to win-rate and insights useful in developing later MCTS self-play agents.Source: §4 · p. 6