PaperlayerView exact revision on arXiv

Source-linked AI summary

Acquisition of Chess Knowledge in AlphaZero

Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, Vladimir Kramnik

arXiv:2111.09259v3cs.AIstat.ML

TL;DR

The paper asks whether a superhuman neural network trained only by chess self-play acquires human-understandable knowledge. It probes human chess concepts across AlphaZero’s layers and training, analyzes behaviour, and examines activations directly. The authors find evidence that many human concepts emerge in the network, while emphasizing that probing establishes correlation rather than causation.

  • Problem

    The paper asks whether capable neural networks trained without human-labelled data can acquire representations corresponding to human concepts, which matters for interpreting their decisions.

  • Method

    The study combines sparse concept probing across layers and training, behavioural analysis including opening play, and direct activation analysis with unsupervised methods.

  • Results

    Many human chess concepts can be reliably reconstructed from AlphaZero’s internal representations, with material prominent early and king safety and mobility becoming important value predictors later.

  • Takeaways & Limitations

    The findings provide evidence that human-understandable concepts of surprising complexity can exist in AlphaZero despite self-play training without an objective promoting human-like play or activations.

  • Takeaways & Limitations

    The probes measure proximate correlation rather than causation, and high regression scores cannot establish that AlphaZero is genuinely computing the human concept.

Abstract

from arXiv · show

What is learned by sophisticated neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.

1 Introduction

The paper asks whether AlphaZero acquires human-understandable chess concepts despite learning solely through self-play. It combines probing, behavioural analysis, and direct activation analysis to track what is learned, when it emerges, and where it is represented.

  • 1 Introduction: AlphaZero offers a test of human-understandable representations in a superhuman system trained without human-labelled chess data.Its self-play training avoids human-generated examples and imposed classification categories.
  • 1 Introduction: The study probes human chess concepts, tracks behavioural changes during training, and examines internal activations directly.The three analyses cover concept decodability, evolving play, and unsupervised factorisation of representations.
  • 1 Introduction: Many human chess concepts can be reconstructed from AlphaZero’s internal representations more accurately than from its input, though reconstruction remains incomplete.This indicates relevant information is computed within the network while distinguishing probes from complete conceptual equivalence.
  • 1 Introduction: Concept probing produces what-when-where plots showing concept regression accuracy across training time and network depth.A second training run shows remarkable consistency, and many concepts emerge near a rapid change in move selection.
  • 1 Introduction: Early training emphasizes material, while king safety and mobility become important predictors of the value function relatively late.The authors use concept weights as a surrogate for AlphaZero’s style.
  • 1 Introduction: AlphaZero’s development shares some similarities with human chess history but does not recapitulate it, while self-play policy shows striking similarities.The paper also assesses changing playstyle qualitatively across training.
  • 1 Introduction: Unsupervised analyses use non-negative matrix factorisation and activation–input covariance to inspect representations beyond predefined human concepts.The factorisation identifies components related to move selection.

2 Past and related work

The paper situates its post-hoc concept-based analysis within neural-network interpretability and chess research. It contrasts concept probing with alternative explanation and mechanistic approaches while emphasizing unresolved fidelity and causality challenges.

  • 2 Past and related work: Interpretability includes inherently interpretable models and post-hoc explanations for already-trained systems.The paper uses the latter because AlphaZero is a fixed, pre-existing reinforcement-learning system.
  • 2 Past and related work: Concept-based methods use human-understandable concepts to explain network decisions and can incorporate complex domain-specific knowledge.Domain experts may specify concepts that probe network operation in greater detail.
  • 2 Past and related work: Other post-hoc approaches include feature-importance methods, saliency maps, natural-language commentary, and question-answering benchmarks for chess.These approaches target different forms of explanation or chess knowledge.
  • 2 Past and related work: Mechanistic interpretability seeks low-level understanding of the algorithms and representations implemented across network layers.Prior work includes matrix factorisation and feature visualisation for intermediate representations.
  • 2 Past and related work: Post-hoc concept explanations may lack fidelity because alignment between representations, model reasoning, and human mental concepts remains unresolved.The paper’s probes measure correlation rather than causation, and intervention is difficult to specify.
  • 2 Past and related work: Reinforcement-learning explainability must address both complex environments and complex agent architectures.Existing work includes transparent models, post-hoc explanations, auxiliary goals, causal action models, and reward-difference explanations.
  • 2 Past and related work: Chess serves as a model system because it combines extensive human knowledge, verifiable positions, intuition, and move-by-move calculation.The paper presents chess as a testing ground for AI research and interpretability.

3 AlphaZero: Network structure and training

AlphaZero combines a deep neural network for policy and value estimates with MCTS, and this paper examines how its residual network transforms chess states during self-play training.

  • 3 AlphaZero: Network structure and training: AlphaZero uses a deep neural network to estimate game value and next-move probabilities, while MCTS repeatedly evaluates states and updates action selection.These outputs are called the value and policy heads.
  • 3 AlphaZero: Network structure and training: The network is trained from games generated by repeatedly playing against itself, with stronger MCTS-generated games replacing older buffered data.Training begins from randomly initialized parameters.
  • 3 AlphaZero: Network structure and training: The standard input uses h = 8 plies, encoding the current and seven preceding positions as an 8×8×119 tensor oriented toward the playing side.Channels encode pieces, repetition, side to move, castling rights, and move counters.
  • 3 AlphaZero: Network structure and training: The residual backbone indexes layers at skip-connection points, progressively transforming z0 through activations zl before the policy and value heads.Each residual block adds convolutional processing to copied activations and applies ReLU clipping.
  • 3 AlphaZero: Network structure and training: The torso comprises layers 1–20; the value head computes v and the policy head computes a distribution p over moves from z20.The paper restricts its layer analysis to the torso.
  • 3 AlphaZero: Network structure and training: The policy representation is an 8×8×73 tensor encoding legal move patterns and promotion options for each square.The 73 options include sliding moves, diagonals, knight moves, and promotions.
  • 3 AlphaZero: Network structure and training: Figure 1 depicts 3×3 convolutions with 256 filters and residual blocks containing two rectified batch-normalized convolutional layers with skip connections.The figure also shows the 8×8×119 input configuration.
  • 3 AlphaZero: Network structure and training: Experiments update network parameters for 1,000,000 gradient-descent steps and refresh self-play networks every 1,000 steps.Training uses a batch size of 4096, while self-play uses 800 MCTS simulations per move.

4 Encoding of human conceptual knowledge

The paper tests whether human chess concepts are represented in AlphaZero by probing internal activations with sparse regressors across layers and training time. Concepts span engine evaluations and custom chess features, with controls and information-theoretic qualifications.

  • 4 Encoding of human conceptual knowledge: Concepts are defined as user-defined functions mapping the network input position z0 to real-valued outputs.They may be binary, integer-valued, or continuous, including bishop pairs, pawn differences, and mobility.
  • 4 Encoding of human conceptual knowledge: The study uses Stockfish 8 evaluation sub-functions as concepts, focusing on phase-adjusted total values used in position evaluation.These concepts cover high-level features such as material, imbalance, pawns, and related terms.
  • 4 Encoding of human conceptual knowledge: The authors add 116 custom concepts covering forks, pins, contested files, and pawn-structure features.These concepts target more specific lower-level chess properties.
  • 4 Encoding of human conceptual knowledge: Concept values and AlphaZero activations are computed for positions from 105 ChessBase games, then deduplicated and split into training, validation, and test sets.The resulting dataset is organized around unique positions rather than only games.
  • 4 Encoding of human conceptual knowledge: Sparse linear probes predict continuous concepts with linear regression and binary concepts with logistic regression from activations at each layer and training step.Probe accuracy on held-out positions measures how much concept-related information activations carry.
  • 4 Encoding of human conceptual knowledge: The probes use regularization and validation-based λ selection because each residual layer contains 16,384 activations.Separate λ candidate sets are used for continuous and binary concepts.
  • 4 Encoding of human conceptual knowledge: Controls compare regression from the network input with regression to normally distributed random concepts.These controls assess added structure from network computation and probe behaviour on random targets.
  • 4 Encoding of human conceptual knowledge: The information-theoretic framing distinguishes making information available to a computationally bounded probe from creating new Shannon information.The data-processing inequality still constrains transformations through the network.

4.3 Visualising concept learning with what-when-where plots

AlphaZero’s human-defined chess concepts are probed by sparse linear regression across network layers and training checkpoints, producing what-when-where plots. The results show that many concepts emerge during training, are distributed across multiple blocks, and are often most linearly accessible in intermediate layers.

  • Scoring and visualisation: Continuous concepts use coefficient of determination r^2, while binary concepts use accuracy improvement over random guessing on balanced datasets.For continuous concepts, r^2 = 1 indicates perfect linear predictability and r^2 = 0 corresponds to always predicting the mean concept value.
  • Scoring and visualisation: What-when-where plots track concept regression accuracy across network depth and training time for each chess concept.Each grid contains r^2 values for continuous concepts or s values for binary concepts, with axes for layer l and training step t.
  • Evolution of concepts: Many human-defined concepts increase in regression accuracy during training, often emerging around 32,000 steps and being computed across multiple blocks.The same pattern is observed in a second AlphaZero training run, indicating stability of the results.
  • Stockfish 8 score: Stockfish 8 total score reaches r^2 > 0.75 after 10 layers at 64,000 training steps and beyond, following a substantial increase after 16,000 steps.Accuracy plateaus at 128,000 training steps, while early-training accuracy is very low.
  • Threat-related concepts: Threat-related concepts emerge progressively through early layers, while has_mate_threat continues rising later, indicating representation of opponent moves and their consequences.The authors interpret this pattern as evidence of at least single-move lookahead being implemented in the network rather than deferred entirely to MCTS rollouts.
  • Drop in linearly-available information: For well-trained networks, threats, piece-count difference, and imbalance peak in early-layer regression accuracy before dropping sharply in later layers.The decline may reflect information being discarded after use or encoded nonlinearly, so last-layer visualisations can miss important intermediate computations.

4.4 Systematic semantic differences in Stockfish score outliers

AlphaZero’s regression errors can expose systematic semantic differences between its evaluations and Stockfish’s, while concept-based value regression tracks when human chess concepts influence its value output.

  • Systematic semantic differences: Prediction-error outliers can reveal systematic differences between AlphaZero and Stockfish rather than merely failed regression.The analysis treats structured residuals as possible evidence of differing engine evaluations.
  • Systematic semantic differences: In all 12 extreme outlier positions favoring White in regression, Black’s queen could be taken, often without an exchange.These positions occurred around queen-exchange sequences and suggest the probe captured White’s queen-taking ability.
  • Systematic semantic differences: The outlier positions also cluster in AlphaZero activation space, indicating that the residual pattern corresponds to structured internal representations.The clustering was visualized with a t-SNE projection from the same layer and checkpoint.
  • Challenges and limitations: Sparse linear probes may underrepresent spatial concepts because useful information can be distributed across board locations and network layers.The paper also notes that local convolutions gradually propagate spatial information, affecting regression accuracy with depth.
  • Challenges and limitations: High regression scores do not establish that AlphaZero genuinely computes a concept, because probes may recover confounders and concepts can be entangled.The paper describes intervention as a possible remedy but notes that suitable interventions are not obvious.
  • Relating concepts to the value function: 16,000 steps: material begins influencing AlphaZero’s value prediction; 32,000 steps and beyond: king safety, threats, and mobility begin contributing.These stages coincide with the emergence of accurate concept regression from some network layers.

5 Progression through AlphaZero and human history

AlphaZero’s opening preferences narrow from an initially uniform prior during training, whereas recorded human chess history shows increasing diversification. The comparison uses historical games and a Ruy Lopez case study to illustrate divergent paths.

  • Progression through AlphaZero and human history: AlphaZero narrows plausible first moves from a uniform opening prior, while human opening preferences broaden over five centuries.The paper frames this as a marked difference between the progression of AlphaZero training and recorded human chess understanding.
  • Five centuries of data: Historical opening data span 135 recorded games from 1400–1800, 1,175 from 1800–1850, 9,615 from 1850–1900, and roughly 10,000 games per later decade.Later sampling used minimum game lengths before 1970 and highly rated players after 1970.
  • First move progression: Human play shifted from early dominance of 1. e4 toward more balanced use of 1. d4, 1. c4, and 1. Nf3.The paper contrasts this historical pattern with AlphaZero’s more simultaneous exploration of openings.
  • Five centuries of data: 2.76, 2.81 and 3.02 bits: AlphaZero’s first-move prior entropy after 1 million training steps across three seeds, versus 4.32 bits initially.The initial value corresponds to a uniform prior over 20 legal first moves.
  • Five centuries of data: 0.33 bits between 1400 and 1800 and 1.87 bits at the end of the twentieth century: recorded human first-move preferences became more informative.These values quantify the historical increase in first-move preference concentration.
  • First move progression: After the Ruy Lopez sequence, different AlphaZero runs converged either to 3...Nf6 or 3...a6 as their main reply.The paper describes these alternatives as theoretically similar, despite taking different paths from human opening history.
  • Scope of comparison: The historical comparison is narrow: it addresses initial opening options, not middle-game themes or broader endgame understanding.Those human-development dimensions require different estimation approaches.

6 Rapid increase of basic knowledge

AlphaZero acquires basic chess knowledge in a staged process: material value appears first, recognizable openings emerge rapidly, and opening theory is refined over much longer training. Behavioural evidence also suggests tactical skills may develop before positional skills, although that interpretation remains circumstantial.

  • 6 Rapid increase of basic knowledge: Material value and piece mobility are largely learned between training steps 10k and 30k, then refined.Stockfish’s material evaluation is strongly indicative of AlphaZero’s value assessment.
  • 6.1 Discovery of standard openings: Recognizable opening theory develops between 25k and 60k training steps, after basic material knowledge.The prior shifts from roughly uniform to commonly played moves during this period.
  • 6.1 Discovery of standard openings: 80% of the prior mass goes to 1...c5, c6, e5 or e6 after 25k training steps, versus 20% at initialization.Individual opening moves and continuations are adopted within short windows of roughly 30k training steps.
  • 6.1 Discovery of standard openings: Opening theory is progressively refined through repeated updates of the training buffer with fresh self-play games.The Ruy Lopez is refined over a million training steps after the initial opening transition.
  • Limitations: The conclusions depend on the hyperparameter settings selected as strongest, and weaker training configurations were not examined.The analysis therefore does not establish whether the same acquisition pattern holds across substantially weaker configurations.
  • 6.3 Are tactical skills acquired before positional skills?: Tactical positions show slightly larger checkpoint ELO differences than positional openings, providing circumstantial evidence that tactical skills may be acquired earlier or faster.The reported differences are 365 versus 343 points from 32k to 64k, and 152 versus 123 points from 64k to 128k.

7 Exploring activations with unsupervised methods

Unsupervised analyses reveal interpretable structure in AlphaZero’s activations, including persistent local detectors, evolving global features, and factors related to threats and move selection. Many later-layer factors remain unexplained, limiting the current account of the network’s representations.

  • 7.1 Methodology: NMF decomposes AlphaZero activations into non-negative spatial weights and global factors, providing a concept-agnostic complement to supervised probes.Each square is represented by K non-negative weights over global factors, approximating the original activation matrix.
  • 7.1 Methodology: Using 36 factors per block yields 720 block/factor pairs, but most factors in later layers remain unexplained.The full factorisation dataset is available online, and explaining later-layer factors is identified as future work.
  • 7.2 Covariance between inputs and activations: Interpretable factors show that potential move computations develop across multiple blocks because convolutional layers connect only spatially adjacent neurons.The gradual increase in predictive power for threat-related concepts over initial layers is partly explained by this multi-layer computation.
  • 7.2 Covariance between inputs and activations: Layer-3 factors encode the number of opponent pieces able to move to a square and apparent move quality for the current player.The opponent-mobility factor may support potential-exchange computations, while the move-quality factor highlights stronger candidate moves.
  • 7.3 Limitations: Interpreting activation factors is constrained by possible factor-count errors, broken spatial correspondence, and later-layer complexity.Relating complex later-layer factors to understandable early-layer factors is identified as an interpretability priority.
  • 7.2.1 Feature detectors: Early activations detect localized piece patterns and move types, including long-range diagonal, horizontal, and vertical movement.These detectors distinguish bishop, queen, and rook movement patterns from particular board squares.
  • 7.2.2 Persisted feature detectors: Some feature detectors persist from layer 1 through layer 20, while others evolve from local feature propagation toward global patterns.Persistence is illustrated by an activation detecting opposing central pawns; other activations encode changing global features across layers.

8 Conclusions

The study combines concept probing, behavioural analysis, and direct activation examination to study how AlphaZero’s representations and play evolve. It finds that many human chess concepts can be accurately regressed from the trained network despite no human games or human-like objective.

  • 8 Conclusions: The study combines concept probing, behavioural analysis, and examination of AlphaZero’s activations to investigate evolving representations and play.The analyses cover human concepts, behavioural changes, and direct activation structure.
  • 8 Conclusions: Many human chess concepts can be accurately regressed from AlphaZero’s trained network, despite training without human chess games or an objective promoting human-like play or activations.This conclusion concerns the trained network’s representational content.

A Full concept list

The paper’s concept inventory combines 93 Stockfish 8 API concepts with additional custom chess concepts, including pawn-related concepts. The custom tables define shorthand for player, opponent, and value differences.

  • A Full concept list: Table 2 contains 93 concepts drawn from Stockfish 8’s public API.The paper uses Stockfish 8 rather than a later version because prior observations compare AlphaZero with Stockfish 8.
  • A Full concept list: Tables 3 and 4 add programmatically implemented custom concepts, including concepts specifically related to pawns.These custom concepts are not taken from Stockfish 8’s API.
  • A Full concept list: In the custom concept tables, m means mine, o means opponent, and diff means the difference between corresponding player and opponent values.These abbreviations apply to the self-implemented concepts in Tables 3 and 4.

B Regression results for all concepts

The appendix reports regression results for the paper’s extended concept inventory, covering Stockfish-derived, custom, capture-related, and pawn-related concepts. This inventory remains only a starting point because it cannot explicitly capture the full breadth of chess knowledge.

  • B Regression results for all concepts: The concept inventory is far from capturing the vast chess knowledge accumulated over centuries and the many board patterns that can occur.The authors describe the inventory as a starting point for further exploration.
  • B Regression results for all concepts: The custom-concept results exclude capture-related concepts in Figures 19–22, while Figure 23 reports capture-related concepts.The figures continue across multiple panels for the respective concept groups.

C AlphaZero policy progression through training for different training seeds

Figures 26–31 track AlphaZero’s highest-probability opening moves and sequences across training iterations and different initial seeds. The visualizations restrict attention to moves made at least once by human players and show how probability mass is distributed over time.

  • Policy predictions: Figures 26–31 show the highest-probability move sequences or moves predicted by AlphaZero’s policy network from several chess positions.The displayed starting positions include common openings such as 1. e4 e5 2. Nf3 Nc6 3. Bb5 and the start position.
  • Policy predictions: Each figure compares probability distributions across different initial training seeds over training iterations.The x axis records training iteration for the relevant seed, allowing opening-policy changes to be followed during training.
  • Policy predictions: The plots include only moves made at least once by human players, while the white area represents cumulative probability assigned to omitted move sequences.This presentation separates the displayed high-probability sequences from the remaining policy mass.

D Outlier positions in score regression

The negative outlier positions from the score-regression analysis have a concrete tactical interpretation: Black’s queen is capturable in every displayed position.

  • Negative outliers: Black’s queen can be captured in every negative outlier position shown in Figure 3.These are positions highlighted in red with true value less than zero.
  • Negative outliers: The outlier analysis therefore examines positions where score-regression errors coincide with a recurring queen-capture feature.The section identifies the shared tactical property across all displayed negative outliers.
  • Negative outliers: Figure 32 presents all negative outliers from Figure 3 rather than a selected subset.The figure caption explicitly characterizes the collection as all negative outliers.

E Concept regression results for second seed

The concept-regression analysis was repeated on a second AlphaZero training run to test whether the reported findings extend beyond one seed. The resulting regression plots were obtained from seed 1 and matched the corresponding plots from the initial analysis.

  • Second training run: The concept-regression methodology was applied to a second AlphaZero training run.The initial analysis used seed 3, whereas the additional regression plots were obtained from seed 1.
  • Second training run: The plots from seed 1 match those in Figures 14–25 of Appendix B.The passage presents this agreement as evidence that the regression results generalize beyond a single training run.
Loading 2111.09259v3…