Scaling Laws for a Multi-Agent Reinforcement Learning Model

• Power-law evidence has come mostly from supervised learning, with considerably less investigation of reinforcement-learning performance scaling.Source: §1 · p. 1
• Optimal network size follows a shared power law with available compute in both games.Source: Figure 1 · p. 2, §1 · p. 2
• AlphaZero and AlphaGo Zero stand below the compute-optimal size trend shown for Connect Four and Pentago.Source: §1 · p. 2
• The study tests fully connected AlphaZero agents on Connect Four and Pentago, games that differ in branching factors and game lengths.Source: §1 · p. 2
• Playing strength scales as a power law with model size under abundant compute and with compute for optimally sized agents, with similar exponents in Connect Four and Pentago.Source: §1 · p. 2

• Prior MARL scaling evidence covered MuZero reward versus data, AlphaZero compute on Hex, and football-agent Elo versus environment frames.Source: §2 · p. 2
• Atari Transformer model-size evidence was difficult to assess because it used few data points, each from one training seed.Source: §2 · p. 2
• Neumann & Gros (2022) offered first indications of the model-size scaling law studied here.Source: §2 · p. 3

3.1 LANGUAGE MODEL SCALING LAWS

• Language-model loss scales as a power law with model size, dataset size, and compute when the other resources are not bottlenecks.Source: §3.1 · p. 3

3.2 POWER LAW SCALING AND ELO RATING

• The Bradley-Terry model assigns each player a strength γ_i that determines expected game outcomes, an assumption underlying Elo ratings.Source: §3.2 · p. 3
• Because Elo is a log-scale representation of player strength, logarithmic Elo scaling with a variable implies power-law scaling of strength with that variable.Source: §3.2 · p. 3

• AlphaZero agents use Monte Carlo tree search guided by a multilayer perceptron with two hidden torso layers and fully connected policy and value heads.Source: §4 · pp. 3–4
• Most hyperparameters are fixed to OpenSpiel’s example settings, while temperature drop is tailored to each game.Source: §4 · p. 4
• Connect Four and Pentago enable benchmarking against perfect or near-perfect solvers, and differ in branching factor and average game length.Source: §4 · p. 4
• Training is repeated with six random seeds per game, and matches among the trained agents provide Elo ratings.Source: §4 · p. 4

5.1 NEURAL NETWORK SIZE SCALING

• With abundant compute, Elo scales logarithmically with network size in both games until performance approaches the optimal-play ceiling.Pentago's size-scaling exponent is strikingly close to Connect Four's; Figure 2 shows the trends and solver benchmarks.Source: §5.1 · pp. 4–5, Figure 2 · p. 5

5.2 EFFECT OF TRAINING TIME ON SIZE SCALING

• The network-size scaling exponent depends on training time: slopes increase with training steps, with diminishing returns, and converge toward a limiting value.Log-linear scaling appears at early training stages but with lower slopes.Source: §5.2 · p. 6, Figure 3 · p. 6

5.3 COMPUTE SCALING

• The compute Pareto front follows log-linear performance scaling up to optimal play, with similar compute-scaling exponents for Connect Four and Pentago.The relation also holds for average agent performance across training runs, not only the best-case training scenario.Source: §5.3 · p. 6

5.4 ESTIMATING THE COMPUTE OPTIMAL MODEL SIZE

• The predicted power law for optimal network size as a function of compute fits the observed data for both games.Its exponent is derived from the parameter-count and compute-scaling exponents; the scale constant is fitted empirically.Source: §5.4 · pp. 7–8

5.4.1 EXTRAPOLATING TO STATE OF THE ART MODELS

• The predicted compute-optimal curve places AlphaGo Zero and AlphaZero well below the optimal trend for the studied MLP agents, but transfer to Go, chess, or shogi is uncertain.Those games differ in observation size, branching factor, win conditions, and mechanics; ResNets may also matter.Source: §5.4.1 · p. 8

5.5 SAMPLE EFFICIENCY

• At equal amounts of self-play data, larger networks achieve higher Elo, while larger models require more data to reach the compute Pareto front.Training data is measured as the number of game states seen during self-play.Source: Figure 4 · p. 7, Figure 5 · p. 8, §5.5 · p. 9

6 DISCUSSION

• Across Connect Four and Pentago, playing strength follows power-law scaling with similar exponents, including when compute is abundant or model sizes are compute-optimal.The paper suggests this may indicate universality classes for performance scaling in two-player open-information games.Source: §6 · p. 9
• Larger AlphaZero agents are more data efficient than smaller ones, achieving better Elo scores with the same amount of training data.This may matter especially for reinforcement-learning studies where data generation is expensive.Source: §6 · p. 9
• Optimal model size scales as a power of available compute, and optimally sized agents can reach maximal performance within a training budget before convergence.The authors report that optimal training stops long before convergence, unlike the training of state-of-the-art models.Source: §6 · p. 9
• The observed exponents may be affected by AlphaZero hyperparameter tuning, especially MCTS exploration, optimization frequency, and data-generating-model update frequency.The study leaves the impact of these hyperparameters for future work.Source: §6 · p. 9
• Scaling-law insights can emerge from medium-sized models and even from a toy problem run on a desktop with four CPU cores.The authors argue that meaningful insights can sometimes be obtained through small-scale analysis.Source: §6 · p. 9

A HYPERPARAMETERS

• The training hyperparameter table covers Connect Four, Pentago, and Oware, while the chosen values generally follow OpenSpiel’s Python AlphaZero example.The stated aim of avoiding extensive hyperparameter tuning was to keep results as general as possible.Source: Table 3 · p. 12, A HYPERPARAMETERS · p. 12
• Temperature drop is 15 for Connect Four and 5 for Pentago because the term, which should correlate with game length, can drastically affect performance after 10^4 optimization steps.The paper does not give a temperature-drop value for Oware in these passages.Source: A HYPERPARAMETERS · p. 12
• Replay-buffer reuse is set to 10 rather than the example’s suggested 3 to keep states in the buffer longer and speed training.The passage attributes the suggested value to Lillicrap et al. (2015).Source: A HYPERPARAMETERS · p. 12
• During matches, noise is zero and temperature remains at 0.25; the low temperature yields roughly 5% or fewer repeated games.The number of MCTS simulations per move at test time matches the training count.Source: A HYPERPARAMETERS · pp. 12–13

B INFERENCE-TIME COMPUTE SCALING

• The comparison uses fully converged agents and scales MCTS search inversely with forward-pass cost to hold inference-time compute fixed.The paper’s main results instead use 300 MCTS steps at both training and test time.Source: B INFERENCE-TIME COMPUTE SCALING · p. 13
• Under fixed inference-time compute, Elo improves with model size at smaller sizes, but larger agents depart from the trend at budget-dependent points.A power law appears at large inference-time compute and breaks off only for the largest agents; its exponent is significantly smaller than in Figure 2.Source: B INFERENCE-TIME COMPUTE SCALING · p. 13
• At large inference budgets, smaller agents can use up to two orders of magnitude more MCTS steps per turn than the largest agents.The paper notes this difference when explaining why the inference-constrained scaling exponent is smaller than in Figure 2.Source: B INFERENCE-TIME COMPUTE SCALING · p. 13

C MEASURING ELO AGAINST OTHER PLAYERS

• Agents far from optimal play are unreliable in solver-only Elo estimates; the worst agent was nearly 1,200 Elo points below the solver.That gap corresponds to an expected win rate of about one in 1,000, requiring many matches; some agents never won and received minus-infinity Elo.Source: C MEASURING ELO AGAINST OTHER PLAYERS · pp. 13–14
• Solver agents vary the strength of a Connect Four solver prior by applying a temperature transformation before matching against AlphaZero agents.At zero temperature the prior is optimal; as temperature approaches infinity it becomes uniform.Source: C MEASURING ELO AGAINST OTHER PLAYERS · p. 14, Figure 7 · p. 14
• Solver-agent matches reproduce the model-size scaling exponent found using the broader player pool, supporting robustness to player-pool bias.The calculation excludes matches between two AlphaZero agents.Source: C MEASURING ELO AGAINST OTHER PLAYERS · pp. 13–14
• Solver-only Elo is less accurate because all solver agents share a strategy, which biases match outcomes and widens the Elo spread for smaller agents.Despite this bias, the model-size scaling law remains visible.Source: C MEASURING ELO AGAINST OTHER PLAYERS · p. 14

D.1 AVERAGE COMPUTE SCALING

• Average Elo scaling with compute has similar exponents whether using Pareto-optimal agents or scores averaged across model-size groups.This supports compute log-scaling when training a single agent rather than selecting the best of many seeds.Source: §D.1 · p. 15, Figure 8 · p. 15

D.2 SIZE SCALING CONVERGENCE

• The parameter-count scaling exponent converges slowly during training, reaching its final value only after extended training.The size-scaling law applies in the limit of infinite compute, when agents train to convergence.Source: §D.2 · p. 15, Figure 9 · p. 15

E OWARE SCALING

• Oware exhibits power-law scaling with model size and compute, but insufficient convergence and underperformance by the largest models limit exponent estimates.The large-model issue mostly affects estimation of the compute exponent.Source: E OWARE SCALING · p. 16

E.1 SIZE SCALING

• Oware's small- and medium-sized agents show size scaling, but the largest agents underperform despite having greater theoretical strategy capacity.The medium-agent fit has a smaller exponent than Connect Four and Pentago, and its fitted exponent appears to keep increasing with training time; training could not be extended another order of magnitude.Source: §E.1 · pp. 16–17, Figure 10 · p. 16

E.2 COMPUTE SCALING

• Oware's compute Pareto front follows a log-linear trend over several orders of magnitude, with a smaller exponent than Connect Four and Pentago.Large models that failed to train well do not appear on the Pareto front.Source: §E.2 · p. 17, Figure 12 · p. 17

E.3 ESTIMATING THE COMPUTE OPTIMAL MODEL SIZE

• Oware's smaller compute-scaling exponent yields a smaller optimal-size exponent; its predicted trend still places earlier state-of-the-art models below optimal size.The Oware trend fits Connect Four and Pentago poorly, despite that undersizing prediction.Source: §E.3 · p. 17

E.4 TRAINING ISSUES

• The largest Oware agents failed to converge to optimal playing ability, underperforming smaller agents they should at least match.The authors suggest integer-valued rather than Boolean input tensors may be a cause; a similar failure appeared in a toy problem with floating-point inputs.Source: §E.4 · pp. 17–18
• Even the lower-exponent Oware scaling law predicts that previous state-of-the-art models were undersized, though by a lesser degree.Oware has smaller size and compute exponents and a larger optimal-size scaling exponent than the other games.Source: Figure 13 · p. 18, Table 5 · p. 18
• A modified AlphaZero version tested each candidate data-generating model against the current model and updated only when the candidate performed better.The authors report that large models trained without issues on their toy problem using this modification, which imports a feature from AlphaGo Zero.Source: §E.4 · p. 18

F TEST SET LOSS

• The test set contains random Connect Four states labeled by a perfect solver with state values and a policy prior over optimal moves.Optimal moves lead to the fastest win, a draw if possible, or the slowest loss; suboptimal moves receive zero prior probability.Source: F TEST SET LOSS · p. 18
• The authors compare fully trained agents' value estimates and policy priors with solver targets using value and policy losses from AlphaZero's full loss function.The solver's values and priors replace the usual game-outcome and MCTS-improved-policy targets.Source: F TEST SET LOSS · p. 18
• Policy loss decreases with model size but remains far from its minimum, while value loss decreases and then increases; neither tracks performance scaling well.The value and policy losses are measured against solver targets, and the policy loss is plotted above its irreducible component.Source: F TEST SET LOSS · pp. 18–19, Figure 14 · p. 19