PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Chess and Go in Natural Language

Ori Press, Alexander Golubev

arXiv:2610.23166v1cs.CLcs.AIcs.LG

TL;DR

Chess engines often make choices whose ideas are hard for people to understand, and engine evaluations rank moves without explaining those ideas. This paper evaluates language-model explanations and introduces BinProp, a training-free method for discovering chess concepts and using them to explain moves. BinProp improves explainability across tested models and also discovers concepts in other games without additional tuning.

  • Problem

    Engine evaluations show which move an engine prefers, but not the chess ideas behind that choice.

    Source: §1 · p. 2

  • Method

    BinProp iteratively discovers binary motifs that occur more often in winning than losing positions, then uses them to explain engine moves in natural language.

    Source: §3 · p. 4, §7 · p. 9

  • Results

    BinProp improves explainability for every tested model, raising Fable 5 from 3.1% with direct prompting to 29.6% and GPT-5.6 Sol from 6.2% to 20.5%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Without additional tuning, BinProp also discovers useful concepts in Go, draughts, and shogi.

    Source: §1 · p. 2

Abstract

from arXiv · show

Modern chess engines play far beyond human ability, often choosing moves whose underlying ideas are difficult for humans to understand. We propose an automatic metric for evaluating chess explanations, grounded in chess-engine evaluations. Evaluating state-of-the-art language models using this metric, we find that they perform surprisingly poorly, often relying on surface-level board observations rather than robust, transferable chess concepts. We then introduce BinProp, a training-free method that enables these same models to discover reusable chess concepts and use them to explain moves in natural language. BinProp improves explainability for every model tested, nearly tripling the average score compared with direct prompting. Beyond chess, the same method discovers useful concepts in Go, draughts, and shogi without additional tuning.

1 Introduction

Chess engines can choose moves far beyond human ability, yet their evaluations reveal preferences rather than the ideas behind them. The paper frames explanations as contrasts between candidate moves and introduces an engine-grounded metric and BinProp for producing them.

1 Introduction

  • Engine decisions are difficult to understand despite engines’ superiority to human players.The paper’s motivation is that an engine evaluation identifies its preferred move but not the chess ideas behind that choice.Source: §1 · pp. 1–2
  • Move quality depends on the continuations it enables and prevents, so explanations compare what becomes true after one candidate move but not the alternative.The target explanations use concrete motifs such as passed pawns, restricted pieces, and superior piece activity.Source: §1 · p. 2
  • The paper proposes an engine-grounded metric and finds that language-model explanations often rely on surface observations rather than robust, transferable chess concepts.It introduces BinProp, a training-free method for discovering reusable concepts and explaining moves in natural language.Source: §1 · p. 2
  • BinProp improves explainability for every tested model, raising Fable 5 from 3.1% to 29.6% and GPT-5.6 Sol from 6.2% to 20.5%.The method also extends to Go, draughts, and shogi without additional tuning.Source: §1 · p. 2

2 Evaluating Explanations

The evaluation tests whether a binary chess motif captures why an engine prefers one move, rather than merely separating example boards. It uses engine-generated continuations and nearby negatives to assess generalization.

2 Evaluating Explanations

  • A motif must return True on positions from the engine’s top-move continuations and False on positions from weaker alternatives; continuations are supplied so the test targets motif quality rather than move calculation.The framework focuses on binary motifs that may emerge several moves after the move being explained.Source: §2 · p. 2
  • Nearby negatives share a positive continuation’s early moves but replace a later move with a weaker one, exposing brittle motifs that separate ordinary examples using irrelevant differences.Figure 2’s “Black has a pawn on d6” motif also fires on a nearby negative and therefore earns a low score.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Evaluation Setup

  • The test samples 100 master-game positions, excluding overwhelming advantages and forced mates, with the best move evaluated at +75 to +125 centipawns and the alternative at least 75 centipawns worse.The alternative must also be within 50 centipawns of the second-best move.Source: §2.1 · p. 3
  • Continuation sampling retains quiet positions across multiple plausible lines, with positives preserving the best move’s advantage and negatives beginning with the weaker alternative.Stockfish plays both sides for at least six plies; the sampling branches across up to 12 additional plies and considers up to six highly ranked moves per turn.Source: §2.1 · p. 3
  • Nearby negatives are constructed by replacing a move along a positive continuation with a weaker move and retaining the resulting quiet position only when the advantage is lost.Eight nearby negatives are collected per starting position.Source: §2.1 · p. 3
  • Each starting position contributes eight shown positives and eight shown negatives, while evaluation uses 24 unseen positions: eight positives, eight negatives, and eight nearby negatives.The explainability score is the lowest correct-output fraction across the three groups, and nearby negatives are withheld from the model.Source: §2.1 · p. 4

3 Method

BinProp discovers binary chess motifs that occur more often in winning than losing positions, then uses useful and move-relevant motifs to explain engine preferences. It ranks motifs by separation and fit and translates selected motifs into natural language.

3 Method

  • The study constructs winning and losing sets from 200k Lichess Elite Database positions, with player-to-move evaluations: 100k at +125 to +250 and 100k at −250 to −125 centipawns.Source: §3 · p. 4
  • At each of 100 iterations, a language model proposes a Python function that maps a board to True or False; BinProp retains motifs whose winning-versus-losing firing-rate gap exceeds 0.01.For example, the material-advantage motif fires on 61.5% of winning positions and 5.5% of losing positions, a 56-percentage-point gap.Source: §3 · p. 4
  • BinProp limits redundancy by measuring how often candidate motifs overlap on winning positions, rejecting containment above 70% unless the candidate has a larger separation gap and replaces the older motif.The loop gives feedback to the model as it iteratively proposes motifs.Source: §3 · p. 4
  • For a specific move, BinProp evaluates each discovered motif’s general winning-versus-losing separation and its fit on that move’s positive and negative positions.It ranks motifs using ∆(f) · fit(f), returns the top k, and uses an additional language-model call to express them in natural language.Source: §3 · p. 4

4 Results

BinProp improved explainability for all eight evaluated models, while direct prompting often produced brittle motifs. Results also show variation in the concepts models discover and in how motif selection affects scores.

4 Results

  • Across all eight models, BinProp improved explainability: its lowest score (0.174) exceeded the strongest Baseline (0.173), and its highest was 0.296.Table 1 reports average scores across 100 test positions.Source: Table 1 · p. 5, §4 · p. 5
  • Baseline motifs often relied on shallow board details and scored zero on every nearby negative, though two examples reached 87.5% explainability.Examples include motifs checking piece locations and a motif requiring an advanced bishop to attack near the opposing king.Source: §4 · p. 5, Figure 4 · p. 6
  • BinProp discovered 192 motifs across models, with counts ranging from 11 for Kimi-K3 to 44 for GPT-5.5 and median ∆(f) from 2.5% to 9.0%.Fable 5 had the lowest median ∆(f), while Qwen had the highest.Source: §4 · p. 5
  • Models varied in motif complexity: most Opus 5 motifs expressed one idea, while Fable 5 and GPT-5.5 often combined several ideas.Fable 5’s concrete target motif can identify profitable captures, weak-pawn pressure, advanced passed pawns, or king attacks under material and piece-safety conditions.Source: §4 · p. 5
  • Material advantage and the bishop pair were common motifs, but models differed in their material thresholds and in conditions attached to bishop-pair motifs.Requirements ranged from a one-pawn to a two-pawn advantage; some bishop-pair motifs added material, piece-safety, or king conditions.Source: §4 · pp. 5–6
  • Among four models’ motifs with ∆(f) ≥5%, Opus 5 supplied the two most unique; its most unique accepted motif had ∆=1.16%.That motif, Line pressure with pins and x-rays, compares pins, x-rays, batteries, and potential discovered attacks between sides.Source: §4 · pp. 6–7
  • Using test labels, the cross-model BinProp oracle scored 0.660, more than twice the best individual BinProp score of 0.296.The passage identifies better ranking, stronger motifs, or both as possible sources of further improvement.Source: §4 · p. 7

5 Analysis

The analysis tests whether explainability tracks objective move quality, compares BinProp with concept-guided commentary, and evaluates its transfer to Go. BinProp’s Go results improve on direct prompting across all eight models.

5.1 Explainability Tracks Objective Move Quality

  • Explainability strongly correlates with Stockfish agreement across sampled motif sets (ρ = 0.66, p < 10^-4).The study samples sets of 10 motifs and uses them as features in a position evaluator.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 Comparison to Concept-Guided Chess Commentary

  • On the test set, CCC scores 0.051, compared with 0.031 for the baseline and 0.296 for BinProp.Fable 5 translates commentary into Boolean Python functions, which undergo meaning checks before scoring.Source: §5.2 · p. 7, Figure 6 · p. 8

5.3 Using BinProp to Explain Go, Draughts, and Shogi

  • On Go, BinProp improves explainability scores for all eight models, while discovering reusable concepts about territory and threatened groups.Examples include local stone density territory estimates and motifs comparing low-liberty groups.Source: §5.3 · pp. 7–8, Table 2 · p. 8

6 Related Work

Related work spans language-based explanations, contrastive concept discovery, chess commentary, and efforts to identify concepts in engine representations. BinProp combines concept discovery with executable board functions.

Language-Based Model Explanations

  • Language-based explanation methods justify classifier outputs, identify counterexamples or absent features, and use user-defined concepts; some convert explanations into labeling programs.Related approaches include jointly learning language interpretation and classification and generating Python labeling functions.Source: §6 · p. 8

Contrastive Concept Discovery

  • Contrastive concept-discovery methods propose and score descriptions of differences across samples, while BinProp compiles each proposed description into a deterministic board function.Related approaches address text, images, model outputs, and paired video or audio.Source: §6 · p. 9

Explaining Chess

  • Chess explanation work generates commentary from engine evaluations or plans, while ACT-Eval checks claims and coverage against expert annotations.BinProp instead automatically discovers motifs as Python functions and tests them on engine continuations.Source: §6 · p. 9

Engine Concepts

  • Prior engine-concept work probes internal activations or compares strong and weak search lines; BinProp needs neither engine activations nor training.One cited approach also tests whether grandmasters can infer discovered concepts from example positions.Source: §6 · p. 9

7 Conclusion and Future Work

The conclusion presents BinProp as a training-free way to find concepts for explaining moves across four games, and suggests extending the approach to long agent rollouts. The appendices describe its iterative motif-discovery harness and conversion of executable motifs into checked natural-language explanations.

7 Conclusion and Future Work

  • BinProp finds concepts for explaining moves in chess, Go, draughts, and shogi; the authors suggest it could explain long agent rollouts beyond games.The proposed use is to compare trajectories that succeed with similar ones that fail when manual review is impractical.Source: §7 · pp. 9–10

A Iterative Prompting Harness

  • The prompting harness runs 100 iterations, proposes one motif per iteration, and scores candidates on the full 200,000-position dataset after screening.It includes ranked current motifs and examples from buckets the current set does not separate.Source: A Iterative Prompting Harness · p. 14

A.1 Prompt

  • The prompt asks for short Python functions that fire more often on winning than comparison positions, avoid overlap, and work from either side to move.The side to move in each FEN is the player being evaluated.Source: §A.1 · p. 14

A.2 Proposal Format

  • A valid proposal consists of a motif name, a one-sentence description, and a side-to-move function.The prompt requires Python code with these three items.Source: §A.1 · p. 14, §A.2 · p. 16

A.3 Low-Separation Feedback

  • For low-separation candidates, feedback reports winning-set activation and losing-set false-positive rates, with examples of erroneous positives.The examples include the FEN and up to 25 executed source lines.Source: §A.3 · p. 17

A.4 Overlap Feedback

  • A candidate with high overlap is rejected when its separation is lower than the existing motif’s; a more discriminative candidate can replace it.The example describes a candidate fully contained in the existing net-material motif.Source: §A.4 · p. 17

A.5 Compile and Runtime Feedback

  • For parse, compile, or runtime failures, the evaluator returns the relevant source excerpt; runtime errors can identify unsupported APIs.The appendix gives an unsupported python-chess API as an example.Source: §A.5 · p. 19

B Turning Python Functions into Text

  • Displayed motifs are executable binary functions; explanations are generated from recorded board facts, rewritten for readability, and checked against the board.Motifs that cannot be tied to concrete board facts are not displayed.Source: B Turning Python Functions into Text · p. 20

C Data Collection

The chess data collection builds engine-scored winning, losing, and nearby-negative continuations from distinct master-game positions. The bootstrap procedure estimates uncertainty in the Baseline-versus-BinProp gains.

C.1 Chess Data Collection

  • The collection uses 200,000 engine-evaluated positions for motif discovery and distinct master-game starting positions for validation and testing.Continuations branch from the best and a weaker move; nearby negatives share the positive path until a later weaker move loses the advantage.Source: §C.1 · p. 21

C.2 Chess Bootstrap Confidence Intervals

  • Chess gains are estimated with 100,000 paired bootstrap resamples of 100 test positions, using the 2.5th and 97.5th percentiles as 95% confidence limits.Baseline and BinProp results for all eight models remain paired at each resampled position; failures score zero.Source: §C.2 · p. 22, Table 3 · p. 22

D.1 Data Collection

The Go evaluation set pairs engine-winning and engine-losing positions with continuations that preserve or lose the advantage. Nearby negatives share most of a positive line, testing whether motifs distinguish similar boards.

D.1 Data Collection

  • The 200k-position discovery pool contains 100k winning and 100k losing positions from 19 × 19 no-handicap JGDB games between players rated at least 7 dan.Winning evaluations range from +7.5 to +15.0 points and losing evaluations from −15.0 to −7.5, balanced by side to move.Source: §D.1 · p. 23
  • KataGo proposes up to ten moves with 512 visits, then rescored child positions determine eligible starting positions and best and negative continuation seeds.Accepted starts require a best-move value of +7.5 to +12.5, a second-best value of −12.5 to +5.0, and a gap of at least 7.5 points.Source: §D.1 · p. 23
  • Separate continuation trees are searched 8–12 plies from each seed, and only endpoints meeting strict evaluation bands are retained as positive or negative candidates.Intermediate search nodes may temporarily fall up to 5 points outside the final bands; retained endpoints must have the starting player to move.Source: §D.1 · p. 23
  • Each retained Go starting position contributes 12 positives, 12 negatives, and eight nearby negatives, separated by at least 7.5 points.Four positives and four negatives are shown to the model; the remaining positions are held out for evaluation.Source: §D.1 · p. 23
  • Nearby negatives share a positive path until a late decision, then differ in the final two plies and must enter the negative score band.This construction retains positions that are similar in their earlier play but lose the advantage after a lower-ranked move and best reply.Source: §D.1 · p. 23

D.2 Results

The Go results are reported as mean explainability for eight models, with qualitative examples and an examination of AlphaGo’s Move 37. The merged material also describes draughts data and evaluation construction.

D.2 Results

  • The Go runs use 100 proposal steps, and final motif sets contain 35 motifs for Opus 5 and 30 for Kimi-K3.Invalid code counts as a step; provider failures and empty responses are retried.Source: §D.2 · p. 24
  • Table 4 reports mean explainability on Go test positions for all eight models, comparing Baseline and BinProp with maximum reasoning.The supplied passages give the table’s scope but not its cell values.Source: Table 4 · p. 24, §D.2 · p. 24

D.3 Qualitative examples

  • Figure 9 compares Baseline and BinProp explanations for Go, with panels showing held-out positive positions and bars for KataGo’s evaluation.Explainability is measured over eight positive, eight negative, and eight nearby-negative positions.Source: Figure 9 · p. 25, §D.3 · p. 25

D.4 AlphaGo’s Move 37

  • Figure 10 presents eight-ply continuations after P10 and three alternatives, alongside three BinProp properties true after P10 but not after those alternatives.The bars show KataGo’s estimated lead for Black immediately after each candidate move.Source: §D.4 · p. 26, Figure 10 · p. 26

E.1 Data Collection

  • The draughts dataset uses 22,621 recorded games and Ponder depth-8 evaluations scaled so one unit represents one uncrowned piece.The winning and losing sets each contain 8,000 positions, balanced by side to move and mutually exclusive across discovery, validation, and test.Source: §E.1 · p. 27
  • Draughts episodes retain 12 positives, 12 negatives, and eight nearby negatives, with every positive at least 0.75 uncrowned-piece equivalents above every negative.Four positives and four negatives are shown; the remaining positions are held out, and motif scoring follows the chess protocol including nearby-negative specificity.Source: §E.1 · p. 27

E.2 Results

The draughts results are reported as mean explainability on the test set, with qualitative examples. The merged material also describes shogi data and evaluation construction.

E.2 Results

  • Table 5 reports mean explainability on the draughts test set for Baseline and BinProp explanations.The supplied passages identify the comparison but do not provide its cell values.Source: §E.2 · p. 27, Table 5 · p. 28

E.3 Qualitative examples

  • Figure 11 compares Baseline and BinProp explanations on held-out draughts positions, with bars showing Ponder evaluations in uncrowned-piece equivalents.The examples are sampled without ranking their held-out scores.Source: Figure 11 · p. 29, §E.3 · p. 29

F.1 Data Collection

  • The shogi set contains 100,000 winning and 100,000 losing positions, balanced by side to move and separated across discovery, validation, and test.Their side-to-move evaluations are +800 to +1600 and −1600 to −800 centipawns, respectively.Source: §F.1 · p. 30
  • Shogi continuation trees start from the best move and qualifying alternatives; retained endpoints are 8–12 plies away and meet positive or negative evaluation bands.Nearby negatives share at least the first six plies of a positive line, then differ in the final two plies; positive-negative separations must be at least 800 centipawns.Source: §F.1 · p. 30

F.2 Results

The shogi results are reported as mean explainability on the test set, followed by qualitative examples. The merged material also applies an expanded chess motif set to expert-game decisions and lists the motifs used in chess results.

F.2 Results

  • Table 6 reports mean explainability on the shogi test set for Baseline and BinProp explanations.The supplied passages identify the comparison but do not provide its cell values.Source: §F.2 · p. 30, Table 6 · p. 31

F.3 Qualitative examples

  • Figure 12 compares Baseline and BinProp explanations on held-out shogi positions, with bars showing YaneuraOu’s evaluation in pawns.The examples are sampled without ranking their held-out scores.Source: Figure 12 · p. 32, §F.3 · p. 32

G Explaining Moves from Expert Games

H Chess Motif Sets

  • The chess motif tables list all 192 BinProp motifs used for Table 1 results and define ∆(f) as the winning-versus-losing frequency difference in percentage points.Motif names and descriptions are model-generated and refer to the side to move.Source: H Chess Motif Sets · p. 36

H.1 Fable 5

This section catalogs chess motifs discovered by language models, spanning tactical exchanges and material, piece activity, king safety and attacks, pawn structures, positional pressure, and endgame advantages.

H.1 Fable 5

  • Fable 5 motifs include advanced outposts, favorable check refutations, material-repaying captures, and exchange sequences that restore balance while retaining an edge.The listed motifs also include trapped enemy pieces and favorable minor-piece imbalances.Source: §H.1 · p. 37
  • Fable 5 also identifies parity-based advantages from king shelter, open-file pressure, queenless endgame edges, and attacks on a vulnerable king.Its motifs include a safer shelter or king alongside pressure on an enemy king or a combined edge in passers, king, rook activity, and pawn health.Source: §H.1 · p. 37

H.2 Opus 5

  • Opus 5 motifs capture positional advantages such as a bad opposing minor piece, passive defenders, dangerous passers, favorable piece imbalances, weak-color control, and active kings or lines.They also include pressure on weak pawns and enemy camps with multiple structural or king weaknesses.Source: §H.2 · pp. 39–40
  • Opus 5 also identifies tactical and initiative motifs, including profitable captures without counterplay, unsafe opponent moves, safe checks, tempo-gaining threats, and overloaded or hanging enemy pieces.Other motifs concern harassed queens, double-attack threats, and concrete pressure points such as pins and unstable guards.Source: §H.2 · pp. 39–40

H.3 GPT-5.5

  • GPT-5.5 motifs combine concrete gains and initiative—such as favorable captures, queen tempos, bishop-pair activity, loose-piece pressure, and advanced passers—with material close to parity.The listed concepts also include queenless rook activity, castling against a lagging king, queen imbalances, and long-range compensation from bishops, rook lifts, or queen pressure.Source: §H.3 · p. 42
  • GPT-5.5 also identifies positional pressure from space clamps, backed outposts, pawn targets, safer kings, and coordinated pieces attacking restricted enemy positions.Other motifs include overextended-pawn cleanup, central pawn binds, and pressure on key defenders or weakened shelters.Source: §H.3 · p. 42
  • GPT-5.5 motifs include king pressure, deep pins and checks, queen-minor batteries, enemy-camp wedges, and queenless endgame conversion.The descriptions condition these motifs on material balance or only a small material edge.Source: §H.3 · pp. 42–43
  • GPT-5.5 also identifies minor-piece swarms, safe major-piece tactics, supported queen lifts, central-file pressure, rook-lift spears, and sixth-rank activity.These motifs are described as occurring without a large material edge or when material is close.Source: §H.3 · p. 43

H.4 GLM-5.2

  • GLM-5.2 motifs include having no backward pawns while the opponent has one, and placing a rook or queen in the opponent’s back two ranks to attack a pawn.The latter motif contrasts that invasion with the opponent lacking a corresponding major-piece attack on the side-to-move’s pawns in its back two ranks.Source: §H.4 · p. 45

H.5 Kimi-K3

  • Table 11 presents chess motifs discovered by Kimi-K3.The supplied passage gives the table caption but no motif descriptions or results.Source: Table 11 · p. 46

H.6 DeepSeek-V4-Pro

  • DeepSeek-V4-Pro motifs include a queenside pawn majority, supported or unstoppable passed pawns, a kingside pawn storm, a bishop pair, and a knight outpost.Other described motifs concern queen attacks near the opposing king, rook invasion or placement behind a passed pawn, and attacks on pieces defended only by the king.Source: §H.6 · p. 48

H.7 GPT-5.6 Sol

  • GPT-5.6 Sol identifies a two-minor-piece coalition against an extra rook and a deployment lead with matched non-pawn forces and at most a one-pawn deficit.The coalition description specifies balancing or strengthening conditions; the deployment lead may involve active minors or a safe supported advanced rook.Source: §H.7 · p. 50

H.8 Qwen3.5-397B-A17B

  • Table 14 presents chess motifs discovered by Qwen3.5-397B-A17B.The supplied passage gives the table caption but no motif descriptions or results.Source: Table 14 · p. 51
Loading 2610.23166v1…