Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero

1 Introduction

• The paper reframes AI systems as learning machines whose hidden knowledge could extend human expertise.It contrasts this goal with treating AI only as efficient problem-solving machinery.Source: §1 · pp. 1–2
• The authors distinguish shared knowledge (M ∩H), human-only knowledge (H −M), and machine-only knowledge (M −H), focusing on the last set.They argue that existing interpretability often analyzes machine knowledge through human concepts.Source: §1 · p. 2
• Chess provides a test bed because its knowledge base is mature, play quality is measurable by Elo, and AlphaZero learned super-human chess without human knowledge.The paper uses AlphaZero’s self-play training to investigate knowledge not fully accessible to humans.Source: §1 · pp. 2–3
• The framework discovers dynamic, data-efficient concepts from AlphaZero’s policy-value network and MCTS, filters for novelty and teachability, and analyzes relations to human concepts.Spectral analysis selects information unique to AlphaZero’s game space, while a teachability metric evaluates downstream usefulness.Source: §1 · p. 3
• The study collaborated with four world-top chess grandmasters and former World Champions to test whether they could learn and apply new concepts from prototypical examples.Source: §1 · p. 3
• Grandmasters improved at finding concept-based moves aligned with AlphaZero, whose discovered concepts often combine chess ideas differently from traditional human principles.One prototype contrasts the common Rxh5 continuation with AlphaZero’s Qc1 plan to regroup pieces toward the queenside.Source: §1 · p. 4, Figure 2 · p. 4

2.1 Concept-based explanations

• Concept-based explanation methods use high-level abstractions to explain models, but probe-set dependence, linearity, and alignment with human concepts remain limitations.Prior work also investigates discovering concepts beyond human labels, generally to explain models rather than teach humans new concepts.Source: §2.1 · p. 5

2 Related work

• This paper differs by discovering and teaching new concepts rather than only identifying existing human concepts or explaining agent behavior.The distinction links concept discovery with the human-learning objective of the paper.Source: §2.1 · p. 5, §2.3 · p. 6

2.2 Generating explanations in Reinforcement Learning

• Reinforcement-learning interpretability addresses temporal state-action dependencies through methods including saliency, trees, memory visualization, finite-state models, and counterfactuals.These methods can expose overfitting, poor out-of-distribution performance, inter-agent dynamics, and spurious correlations.Source: §2.2 · pp. 5–6

2.3 Chess and Artificial Intelligence

• Chess-engine research spans computationally powerful systems, neural-network and reinforcement-learning approaches, and explanations based on saliency, search trees, and language.AlphaZero-related concept studies largely identified pre-existing human concepts and did not incorporate search and move sequences.Source: §2.3 · p. 6

3 What are concepts?

• A concept is defined as a unit of knowledge that contains useful information and removes irrelevant information through minimality.The beak example illustrates that a concept can be transferred and used to solve a task.Source: §3 · p. 7
• Concepts are operationalized by showing that they can be transferred to another agent to help solve a task or follow a represented strategy.Successful transfer implies that the concept is self-contained and useful for the task.Source: §3 · p. 7
• The framework assumes concepts are linearly encoded in neural-network latent space, while acknowledging that nonlinear concepts may be missed.The authors nevertheless report finding useful concepts with purely linear representations.Source: §3 · p. 7
• In reinforcement learning, the target concepts give rise to plans: deliberate action sequences motivated by context-specific instrumental goals.The framework assumes that similar contexts with similar instrumental goals produce similar concepts.Source: §3 · p. 7

4.1 Excavating concept vectors

• The method formulates concept discovery as convex optimization over both binary and real-valued inputs, covering the policy-value network and MCTS.Sparsity via the L1 norm encourages minimal concept vectors.Source: §4.1 · p. 8

4.1.1 Concept constraints for static concepts

• Static concepts are learned from labeled positive and negative positions by finding sparse vectors whose inner products separate concept-present from concept-absent activations.Performance is evaluated on held-out activations to test whether the learned concept constraints generalize.Source: §4.1.1 · pp. 8–9

4.1.2 Concept constraints for dynamic concepts

• Dynamic concepts contrast AlphaZero’s optimal MCTS rollout with subpar trajectories across depths, filtering equal-value alternatives and reducing noise and polysemanticity.The contrasting rollout pairs produce convex constraints on latent activations over time.Source: §4.1.2 · pp. 9–10, Figure 3 · p. 10

4.2 Filtering concepts

• Concepts are filtered for usefulness by testing whether a student agent improves after learning from concept prototypes.The procedure measures baseline performance, teaches with prototypes, and evaluates a concept-related task afterward.Source: §4.2 · p. 11, §4.2.1 · p. 11

4.2.1 Teachability

• Concept-specific teaching improved holdout performance more than random-position teaching, with 50 epochs on few prototypes matching 10K–100K epochs of self-play.Students trained on concept prototypes improved most on concept-specific tests, and also improved on random prototypes; holdout evaluation ruled out memorization.Source: §4.2.1 · p. 12, Figure 4 · p. 12

4.2.2 Novelty

• AlphaZero games had higher latent-representation rank than human games at layers 19 and 23, motivating novelty discovery in those final layers.The rank comparison suggests AlphaZero games may contain concepts absent from human games, although the supplied passage reports no table values.Source: §4.2.2 · p. 13, Table 1 · p. 14
• Novelty is quantified by comparing concept reconstruction losses using basis vectors from AlphaZero and human games, with higher scores indicating closer alignment to AlphaZero.The method searches for concepts better explained by AlphaZero’s latent-space basis than by humans’.Source: §4.2.2 · p. 14
• Concepts are accepted when AlphaZero’s basis vectors reconstruct them with lower error than human-game basis vectors for every k.Figure 5 visualizes novelty scores for 120 layer-19 concepts, distinguishing accepted from rejected concepts.Source: §4.2.2 · p. 14, Figure 5 · p. 15
• 27.1% of concepts remaining after teachability-based filtering are removed using the novelty metric.Source: §4.2.2 · p. 15

5 Method evaluation

• The method evaluation covers datasets ranging from concepts more familiar to humans to concepts associated with AlphaZero.Table 2 distinguishes strategic and tactical datasets using S and T.Source: §5 · p. 15, Table 2 · p. 15

5.1 Evaluation of the proposed convex optimisation framework for finding concept vectors

• The evaluation uses layers 19, 20, 21, and 23 to examine concept vectors near AlphaZero’s policy and value outputs.Layer 19 is used elsewhere because spectral analysis suggests potential novelty, while this evaluation includes additional output-proximal layers.Source: §5.1 · pp. 15–16

5.1.1 Do the concept constraints hold for a test dataset?

• Most datasets achieve high accuracy on held-out concept constraints, indicating that the learned vectors represent the supervised concepts well.Accuracy is measured on an 80/20 train/test split and reported in Table 3.Source: §5.1.1 · p. 16, Table 3 · p. 16

5.1.2 How many data points do we need to learn a concept?

• The method often reaches close to full-set accuracy with as few as 10 training examples on the pieces dataset.Performance is lower in the value head than the policy head, possibly because value representations compress or combine piece-related concepts.Source: §5.1.2 · p. 16, Figure 6 · p. 17

5.1.3 Does amplifying concept vectors increase concept-related behavior?

• Concept amplification nudges latent representations toward a concept vector to test whether the concept influences AlphaZero’s predicted moves.The experiment compares normalized policy performance with and without amplification across layers 18, 19, and 23.Source: §5.1.3 · p. 17
• Higher concept test accuracy produces larger performance improvements on strategic-tactical puzzles when concept vectors are amplified.This suggests test accuracy is a useful proxy for how well a concept vector captures the concept’s semantic meaning.Source: §5.1.3 · p. 18, Figure 7 · p. 18

6 Human evaluation

• The evaluation measured baseline solving, exposed grandmasters to AZ’s suggested top lines based on MCTS calculations for the same puzzles, and tested unseen puzzles sampled from the same concepts.Source: §6 · p. 18

6.1 Grandmaster performance

• All four grandmasters improved notably from Phases 1 to 3, suggesting they learned and applied represented AlphaZero concepts; improvement did not correlate with Elo rating.Participants rated 2600–2700 or 2700–2800, and performance was measured on unseen puzzles from previously studied concepts.Source: §6.1 · p. 19, Table 4 · p. 19, §6 · p. 19

6.2 Qualitative analysis of the concepts and illustrative examples

• Grandmasters generally valued AlphaZero’s concepts as clever, novel, and difficult to spot, often describing the resulting positions as highly complex.Their comments included that some moves were “something new” or “not natural,” even after seeing AlphaZero’s solutions.Source: §6.2 · p. 20

6.2.1 Concept example: positive knowledge transfer

• One strategic-prophylactic concept improved from 0/4 correct identifications in Phase 1 to 2/4 in Phase 3, including plans combining space, tactics, and unconventional sacrifices.The examples involved inducing weaknesses before a strategic queen sacrifice and expanding space despite king exposure; the grandmaster later identified AlphaZero’s prophylactic ideas.Source: §6.2.1 · pp. 20, 21, 24

6.2.2 Concept example: unsuccessful learning

• A different space-and-activity concept was not learned: the grandmaster understood its general plan but missed its finesses and later rejected AlphaZero’s move.AlphaZero prioritized activity and pawn advances over immediate material recovery or king safety, whereas human heuristics favored safer, more familiar priorities.Source: §6.2.2 · pp. 25–27

6.2.3 Differences between humans and AZ

• The examples suggest that AlphaZero assigns different concept relevance than humans, flexibly applying concepts across the board while prioritizing space and piece activity over material or king safety.The authors speculate that self-play gives AlphaZero fewer human heuristic priors, enabling rapid plan changes and potentially super-human concept application.Source: §6.2.1 · p. 23, §6.2.3 · p. 27, §6.2.2 · p. 27
• AlphaZero’s self-play objective assumes optimal play and information symmetry, whereas humans often choose practical moves that exploit opponents’ imperfect play.Humans may complicate drawn positions or choose less clear continuations to increase their chances against another human, while AlphaZero seeks the optimal plan regardless of complexity.Source: §6.2.3 · pp. 27–28
• AlphaZero’s unconstrained game duration leads it to choose slow strategic wins, unlike humans who often simplify winning positions to secure results quickly and reduce risk.This lack of a time-related penalty may yield super-human concepts but can also produce concepts that are difficult for humans to learn.Source: §6.2.3 · p. 28
• AlphaZero’s greater computational capacity lets it select computationally expensive moves and defend complicated positions that humans may avoid or handle with safer, slightly suboptimal choices.Although computational capacity itself cannot be transferred, AlphaZero can still discover new ideas that can be taught to humans.Source: §6.2.3 · p. 28

• The framework extracts unsupervised chess concepts from AlphaZero, verifies their novelty against human games, and tests their informativeness through transfer to another agent and human puzzles.Source: §7 · p. 28
• The work provides a first step toward using super-human AI knowledge to extend human knowledge and improve expert understanding.Source: §7 · p. 28
• All four world-top grandmasters improved their performance after learning AlphaZero concepts compared with baseline performance.Source: §7 · p. 28
• The study examines only linear sparse concept vectors and generally models a plan with a single concept, leaving nonlinear and multi-concept structures for future work.Source: §7 · p. 29
• Human learning conditions remain underexplored because grandmasters received a fixed time budget and non-interactive MCTS-based explanations.Source: §7 · p. 29

8.1 Human experiments: more concept puzzle examples

• The appendix adds further concept-puzzle examples from the human experiments.Source: §8.1 · p. 38

• The grandmaster improved by +2/4 between Phases 1 and 3, indicating that AlphaZero’s unconventional centre-control concept was human-learnable.Source: Additional concept example: positive… · p. 38
• AlphaZero’s 5...Bf5 concept controls e4 and supports unconventional follow-ups such as ...Bb4 and ...h5, contrasting with the grandmaster’s more natural alternatives.Source: Additional concept example: positive… · p. 38
• In the paired positions, AlphaZero prioritizes centre control, space, and piece activity over conventional principles such as completing development, king safety, and preserving the bishop pair.Source: Additional concept example: positive… · p. 39
• AlphaZero’s knight manoeuvres 10.Ndf4 and 12.Ng1 improve activity and central control despite conflicting with common development and king-safety rules.Source: Additional concept example: positive… · p. 40
• The grandmaster initially missed 12.Ng1 but recognized its value after seeing AlphaZero’s continuation.Source: Additional concept example: positive… · p. 40

• AlphaZero’s Qc1 plan is described as the only way to maintain an advantage, repositioning White’s pieces from the kingside toward queenside control.Source: Informative puzzles · p. 40
• In Figure 20, AlphaZero uses prophylactic tactics such as 21.Nc5 and 18.Rad1 to prevent central or queenside advances while preserving a space advantage.Source: Informative puzzles · p. 41, Figure 20 · p. 41
• The grandmaster eventually identified Nc5 as the strongest move after considering several alternatives, highlighting the complexity of the prophylactic idea.Source: Informative puzzles · p. 41
• A grandmaster described the alternative attacking choice as emotional after AlphaZero’s quieter f3 continuation maintained White’s advantage.Source: Informative puzzles · p. 42
• AlphaZero favors slower, longer-dominance play, whereas grandmasters tend to consider more forcing sequences and can remain committed to kingside plans.Source: Informative puzzles · pp. 41–42

• AlphaZero’s calculated line starts with 18.f3, includes 18...Ng6, 19.Bc5, Bh4, 20.Bxg6 hxg6, and 21.g3, after which White is better.Source: Human vs AI Play: AZ opts for less… · p. 42

• The authors note that some concepts may be intrinsically hard for human players because their abstract thinking, capabilities, and computational budgets differ.Source: Difficult or non-instructive puzzles · pp. 42–43
• AlphaZero’s Qa1 activates the queen but requires careful calculation to rule out kingside counterplay, making the move appear risky to humans.Source: Difficult or non-instructive puzzles · p. 43
• The grandmaster described AlphaZero’s tactical solution after 33.Ne6 Nxe6 34.Bh3 as a very nice idea that was hard to spot.Source: Difficult or non-instructive puzzles · p. 43

• AlphaZero forces a draw with g4 because it treats equalising moves alike when the position’s final result is already determined.Source: Differences in motivation in human and… · pp. 43–44
• The grandmaster judged g4 practically inferior because it removes Black’s winning chances, whereas other drawing moves preserve practical chances for both sides.Source: Differences in motivation in human and… · p. 44
• AlphaZero assumes an equally strong opponent, while humans try to exploit suboptimal moves and therefore choose moves that maximise practical outcomes.Source: Differences in motivation in human and… · p. 44

Shortcomings of method for generating prototypes

• A prototype puzzle exposed a method shortcoming because it admitted several viable moves, making the underlying concept difficult for humans to understand from the move sequence.AlphaZero chose Rd1, whereas the grandmaster preferred Re1 or Ke1 and described alternative drawing plans.Source: Shortcomings of method for generating… · p. 45

8.2 Background: AZ policy value network

• AlphaZero combines a policy-value network with Monte Carlo tree search (MCTS) to produce policy and value estimates and select moves.The network uses a ResNet body with 19 residual blocks, while the value and policy heads are indexed as layers 20–22 and 23–24, respectively.Source: §8.2 · p. 46

8.3 Background: how humans and AZ play chess

• The method parallels human chess analysis by extracting position features, weighting candidate moves with the policy, and using MCTS to calculate an optimal move.Humans identify critical aspects, consider candidate moves, and calculate continuations; calculation remains necessary when principles do not determine the best continuation.Source: §8.3 · pp. 47–48

8.4.1 Dynamic concept hyperparameters

• Dynamic concepts distinguish single-player from both-player representations by sampling alternating or all rollout latents, with rollout depths T = 5 or T = 10.Subpar variations require a minimum value difference of 0.20 and/or a visit-count difference of 10%.Source: §8.4.1 · p. 48

8.4 Further details: convex optimisation formulation for concepts

• Concept discovery uses convex optimization across labelled and unlabelled chess datasets, relating human-taught concepts to AZ-specific positions.Datasets vary by concept type, human knowledge, and complexity, and include pieces, Stockfish concepts, strategic puzzles, openings, and 30,000 AZ self-play positions selected where checkpoints differed by 75 Elo.Source: §8.4.2 · pp. 48–50

8.4.3 Convex formulation for different datasets

• Dataset-specific formulations use human-labelled positions, Stockfish score extremes, strategic optimal-versus-subpar MCTS rollouts, selected openings, and AZ games where models 75 Elo apart choose different moves.For strategic concepts, subpar rollouts differ by at least 0.2 in estimated value and 10% in visits; the analysis uses maximum depth T = 5.Source: §8.4.3 · pp. 50–51

8.5 Beta hyperparameter tuning

• Concept amplification results were not very sensitive to the hyperparameter β in validation experiments using two randomly chosen STS concept sets.The supplied passage reports the qualitative robustness finding but no numerical values from Table 6.Source: §8.5 · p. 52, Table 6 · p. 52

8.6 Teachability implementation

• Teachability prototypes are top-2.5% concept-scoring positions, while students are earlier AZ checkpoints with top-1 policy overlap below 0.2 and are trained to match the teacher’s policy.Training minimizes KL divergence for five epochs; performance is compared with students trained on randomly sampled concept vectors and prototypes.Source: §8.6 · p. 52

8.7 Graph analysis

• The graph contains 29% of possible directed edges (180,609 of 625,681), and performance on related concepts is significantly better than on unrelated concepts at the 5% significance level.Source: §8.7 · pp. 53–54

8.8 Human experiments

• The human experiments recruited four grandmasters and evaluated whether they found AlphaZero’s selected move while requiring correct reasoning in their free-form comments.Prototypes were filtered for value-estimate concordance, late-training complexity, nontrivial solutions, and reliable predicted outcomes; filters based on value or policy differences were deliberately omitted.Source: §8.8 · pp. 54–55

8.9.1 Concept constraint satisfaction

• Constraint-satisfaction results are reported for piece, Stockfish, STS, opening-configuration, and opening-concept datasets, with Stockfish concepts extracted from Stockfish 8’s public API.The supplied material identifies the relevant tables and defines Stockfish result notation, including white/black/total differences and middle-game/endgame/phased values.Source: Table 7 · p. 56, §8.9.1 · p. 56, Table 8 · pp. 57–59

8.10 Sample Efficiency Results

• Near-full-set accuracy is reached with as few as 10 or 25 data points on the STS dataset, with similar performance across layers and slightly lower performance for policy head layer 23.The convex optimisation method shows this sample efficiency across the bottleneck, value, and policy layers; the concepts may be more important for value than policy.Source: §8.10 · p. 60, Figure 29 · p. 61