Self-Play Search Distillation for Large Language Model Reasoning

1 Introduction

• Scaling reasoning requires synthetic supervision that can exceed the student’s capabilities, because final answers omit process and human process labels are costly, difficult to verify, and bounded by annotator expertise.Prior work is cited for what final answers fail to expose and for the limits of human process supervision.Source: §1 · p. 1
• The central challenge is to generate structured supervision that reveals comparisons and consequences beyond the learner’s current policy and can be checked independently.Self-generation remains coupled to the learner’s current errors and limitations.Source: §1 · p. 1
• Executable board games expose legal actions, exact transitions, and delayed outcomes, while search records retain alternatives, opponent replies, branch values, and replayable consequences.The introduction presents external search experts as a source of supervision not inherently capped by human expertise.Source: §1 · p. 1
• SPSD converts self-play search into replay-grounded supervision that exposes selected actions, alternatives, and their consequences for LLM post-training.The framework also includes complementary state questions; the LLM does not participate in expert self-play or receive search information at inference.Source: §1 · p. 2
• The paper evaluates whether self-play search supervision transfers across game play, controlled rule variation, and mathematical reasoning.The introduction states that mathematical gains and preserved game competence vary in magnitude by backbone.Source: §1 · p. 2

Game reasoning

• Game-focused work evaluates competence, trains game policies, or places language models in the planning loop, whereas SPSD exports decisions, alternatives, and continuations from a separately trained search expert as offline supervision.The cited work includes chess and strategic-game benchmarks, game-playing policies, and LLM-guided search.Source: §2 · p. 2

Synthetic supervision

• Synthetic reasoning work ranges from verifier-based task generation to search-trace training, while SPSD supervises one planning-expert root decision with ranked alternatives and replayable continuations rather than the full tree or planner computations.The related methods include AlphaGeometry, Enigmata, Absolute Zero, Tree of Thoughts, ASTRO, and Stream of Search.Source: §2 · p. 2

Self-play and post-training

• Unlike online LLM self-play or methods that optimize a learner through generated tasks, SPSD uses self-play to construct a reusable offline corpus of environment-grounded reasoning data.The section contrasts SPSD with self-play methods for games and mathematics and with adaptive environment-design approaches.Source: §2 · p. 3

Expert Iteration

• Expert Iteration alternates planning that improves decisions at individual states with apprentice fitting that generalizes those improvements across states.The apprentice guides expert search, and the next apprentice fits the resulting policy and optional value targets; setting λv = 0 gives policy-only training.Source: §3 · pp. 3–4

Neural-guided MCTS

• Neural-guided MCTS uses apprentice priors and backed-up action values to guide simulations, with root visits defining a search-improved policy after a fixed budget.The root-visit policy target’s sharpness is controlled by Tmcts.Source: §3 · p. 4
• A completed root search yields a selected action and value estimate, while retained paths preserve concrete continuations behind those summaries.These outputs capture the expert decision and evidence accumulated while comparing legal actions.Source: §3 · p. 4

Search-derived supervision

• Search-derived reasoning chains can expose intermediate decision structure, and executable environments allow narrated transitions and outcomes to be checked.On-policy self-distillation can guide a teacher along prefixes sampled by the current student.Source: §3 · p. 4

4 Method

• SPSD is an offline, three-stage bridge from fixed self-play experts to post-training: export decision records, replay and verify them, then create prompts and reference responses.The expert generating the supervision is independent of the student trained on it.Source: §4 · p. 4

4.1 Executable decision records

• Each expert record pairs a selected root action with visit-derived policy, root value, and retained branch evidence replayed to recover successor states and terminal outcomes.Trajectories begin with a uniformly random legal prefix; the expert then controls both seats and runs 50 MCTS simulations at each selected state.Source: §4.1 · p. 5

4.2 Replay-grounded supervision

• A supervision instance uses the board-game state as input, the best action as target, and a target-hidden expert trace derived from the verified MCTS record.Prompts include the rules, board, side to move, legal options, and answer contract; the student sees only the state and must reconstruct the comparison.Source: §4.2 · p. 5
• Move-choice traces replay the selected action, up to one visit-ranked alternative, and an available opponent reply, preserving legal prefixes and terminal outcomes reached in replay.Equivalent failed branches are deduplicated, and each trace ends with the selected legal action.Source: §4.2 · p. 5
• Six environment-derived question types supervise state reading, with answers computed from the board, legal actions, transitions, and terminal rules.Trajectory-grouped splits keep each decision state in exactly one row family.Source: §4.2 · pp. 5–6

4.3 Language-model post-training

• SFT trains directly on expert traces, while OPSD uses the expert trace to guide the teacher on prefixes sampled from the student’s own responses.The forward KL objective penalizes low student probability on tokens the grounded teacher deems likely; RuleBot-Distill provides a non-search control under the same student-facing interface.Source: §4.3 · p. 6

Training corpus and search experts

• The main SPSD corpus draws on Connect4, Domineering, Simplified Othello, and Tic-Tac-Chess, with 5,000 examples per game except 600 for Tic-Tac-Chess.Each game allocates 80% to move choice and 20% to six state-question tasks.Source: §5 · p. 6

Models and post-training

• Experiments compare Qwen3-4B-Base, two Qwen3-8B thinking settings, and Llama-3.1-8B; SFT fits SPSD traces, while OPSD uses them as privileged teacher traces.RuleBot-Distill keeps OPSD unchanged and replaces search-derived data with heuristic-policy completions.Source: §5 · pp. 6–7

Rule diversity

• The separate rule-diversity study compares curricula from five base games and nine variants each, creating 50 environments with local rule changes.Variants preserve observation shape, board topology, and action-space size.Source: §5 · p. 7

Evaluation Benchmarks

• Game transfer is evaluated on 15 held-out Ludii games using mean FIDE score, episode win rate, and move legality; mathematics transfer uses six benchmarks.Mathematics uses pass@1 on MATH500, Olympiad Bench, and Minerva Math, and 32-response average accuracy on AIME24, AIME25, and AMC23.Source: §5 · p. 7

Search distillation analysis

• Search-distillation analysis uses GPT-5.6 to recover considered positions and committed legal moves, then compares those moves with expert-search values and preferences.Per-episode Q50 averages expert values for committed moves; oracle agreement measures matches to the expert-preferred action.Source: §5 · p. 7

Transfer across model families and decision metrics

• 15% to 45% held-out-game win rate and 24.1 to 36.6 six-benchmark mathematics mean show OPSD transfer for Qwen3-4B-Base.The table reports mathematics benchmarks and aggregate and game metrics.Source: §6 · p. 7, Table 1 · p. 8

Expert source

• Against RuleBot-Distill, OPSD raises held-out-game win rate from 35% to 45% and six-benchmark mean from 33.9 to 36.6, though RuleBot leads OlympiadBench 35.7 to 35.1.Source: Table 1 · p. 8, §6 · p. 8

Training trajectories

• At step 1000, variants curriculum reaches 42.4% FIDE, 77.3% legality, and math mean 39.2, versus 39.9%, 75.1%, and 36.6% for base.SFT peaks early and declines, whereas OPSD keeps improving during training.Source: §6 · pp. 8–9, Figure 3 · p. 8

MCTS structure and reasoning sharpness

• Wins concentrate where selected-action Q50 and MCTS agreement are high; successful SPSD play is associated with choosing actions favored by MCTS.Search values and traces remain hidden at inference, so agreement reflects alignment in the model's choices.Source: Figure 4 · p. 9, §6 · p. 9

7 Conclusion

• OPSD provides the most stable overall improvements across studied budgets, while SFT loses earlier gains with longer training and variant-enriched curricula strengthen held-out-game and mathematics transfer.The conclusion identifies expanding executable-environment curricula as a direction for future continual training.Source: §7 · p. 9

A Grounded MCTS Linearization

• Each decision-state record becomes a natural-language trace of the selected action, retained alternatives, and replayed consequences, ending with exactly one boxed legal handle.Numerical search telemetry determines which evidence is retained but does not enter the trace.Source: A Grounded MCTS Linearization · p. 15
• Replay checks the restored state and selected action; an unreplayable selected action rejects the record, while an unreplayable optional branch is removed.Retained branches record legal action prefixes, recomputed successor states, and any terminal outcome reached.Source: A Grounded MCTS Linearization · p. 15
• Linearized traces use fallback, target-only, or contrast modes, with added detail gated by search values or replayed outcomes.Illegal replies are omitted, and non-terminal prefixes are described without outcome language.Source: A Grounded MCTS Linearization · p. 15

B Corpus Materialization

• Materialization creates one move-choice family and six state-QA families, retaining rows only after executable checks.Each row stores student-facing input separately from the reference response and hidden teacher reasoning or verifier context.Source: B Corpus Materialization · p. 15, Table 2 · p. 16
• Each game contributes 100 trajectories opened by up to eight randomly selected legal moves; those prefix actions provide state diversity but are excluded from training rows.Candidate states are ordered by a seed-keyed hash, while RuleBot-Distill and SPSD retain distinct teacher and decision-state provenance.Source: B Corpus Materialization · pp. 15–17

• Tables 3–4 report settings for expert-data generation, language-model post-training, benchmark decoding, and hardware allocation.The passage identifies the scope of the settings, not their specific values.Source: C Experimental Configuration · p. 17
• Replay geometry is evaluated with 50-simulation Q50 estimates.Source: C Experimental Configuration · p. 17
• The reported configuration spans both data generation and model evaluation, including benchmark decoding and hardware allocation.Source: C Experimental Configuration · p. 17

C.1 MuZero-family Expert Training

• The four language-data teachers are EfficientZero agents, a MuZero-family algorithm trained by self-play.The retained data comes from a separate 50-simulation search run from each expert.Source: §C.1 · p. 17, Table 5 · p. 18
• Data generation applies each trained expert’s deterministic 50-simulation policy after a randomized opening prefix.Training uses undiscounted returns, deterministic argmax evaluation, and no reanalysis.Source: §C.1 · p. 17

C.2 Replay-Geometry Expert Selection

• Expert selection compares checkpoints at 50, 100, 200, and 400 training steps on shared scenario indices.Game-macro oracle agreement selects the expert, with game-macro Q50 breaking ties; episodes with missing Q50 are omitted from scoreable-turn averages.Source: §C.2 · p. 18

D Controlled Variants

• Each curriculum receives the same 1,000-step budget, with nine variants per rule family changing one local factor while preserving observation shape, board topology, and action-space size.The model, optimizer, schedule, prompt contract, and evaluation protocol stay fixed; variants must preserve observation shape and legal-handle space and pass executable-state checks.Source: D Controlled Variants · p. 18, Table 6 · p. 18

E Game Corpus

• The controlled-diversity corpus contains five base games and 15 held-out games, with the held-out set used for the main-text evaluation.Games were selected from Ludii and admitted after rule/environment cross-validation, schema and legality checks, invariant rollouts, and smoke training; the base games are Connect4, Tic-Tac-Chess, 3 6 9, Domineering, and First Attack.Source: E Game Corpus · pp. 18–19

E.1 Game Cards

• 3 6 9 is a shared-stone game on an empty 9 × 9 board: players place stones on empty cells, and lines of exactly three, six, or nine occupied cells score one, two, or three points; the higher final score wins.Source: §E.1 · p. 19

A Simple Game

• The simple game is an orthogonal-movement race to form three in a row, with no captures or diagonal movement.Players start with four stones each on alternating cells along the top and bottom edges; a line can be horizontal, vertical, or diagonal.Source: A Simple Game · p. 19

Agapi

• Agapi combines bishop movement on a 4 × 4 board with a permanent orthogonal blocker shot, and wins when the opponent has no legal move-plus-shot.Each player has two bishops; a turn moves one bishop diagonally to its farthest reachable square, then shoots a blocker orthogonally to the farthest reachable empty cell.Source: Agapi · p. 20

All Queens Chess

• All Queens Chess is a queen-slide race to form a line before the board locks: each player controls three queens on a 4 × 4 board, sliding orthogonally through empty cells and leaving permanent blockers.Alquerkonane has checkers-like forward movement with orthogonal jump captures.Source: All Queens Chess · p. 20

Aralzaa / Bajr

• Aralzaa is a forward-leap race to occupy the opponent’s home row, whereas Bajr is a directional camp race using one-step orthogonal moves.Bajr also awards a win for occupying all three cells of the opponent’s camp or leaving them without a legal move.Source: Aralzaa · p. 21, Bajr · p. 21

Chains of Thought / Connect 4 / Domineering / First Attack

• The passages describe a hex-chain game with placement or movement turns, legality tied to local group sizes, and capture of smaller adjacent opposing chains after every move, plus a vertical connection game on a 6 × 7 grid.Source: Chains of Thought · p. 21, Connect 4 · p. 22

First Attack

• First Attack varies forbidden-neighbor geometry, opening locks, and normal-versus-misere last-move outcomes across orthogonal, diagonal, and knight offsets.A legal placement shares no row, column, or diagonal with an occupied cell, and the player making the last legal placement wins.Source: First Attack · p. 23

Groups

• Groups is a connection game where friendly jumps help assemble one group and do not capture; a player wins immediately when at least six own pieces form a single orthogonally connected group.Source: Groups · p. 23

King’s Valley

• King’s Valley is a sliding-piece race in which a king wins by reaching one of four central throne cells or leaving the opponent without a legal slide.Each player starts with one king and three guards on a 4 × 4 board.Source: King’s Valley · p. 24

Moxie

• Moxie combines placement and movement with mandatory orthogonal jump captures when available; wins come from making three in a row, reducing the opponent to two stones, or immobilizing them.Captures do not chain.Source: Moxie · p. 24

Othello

• Othello uses straight-line outflanks to flip opposing discs; the higher disc count wins when the board is full or both players are stuck.A player with no legal placement passes.Source: Othello · p. 24
• Tapatan is a placement-then-sliding alignment game on a nine-point graph, with a win for forming a line of three along the board lines.Each player has three stones; after placement, stones slide to adjacent connected empty intersections.Source: Othello · p. 25

Tic-Tac-Chess

• Tic-Tac-Chess starts with three queen pieces per player, then permits queen slides and non-capturing hops, while varying placement timing, winning-line direction, movement, and locks.A player wins by forming a horizontal, vertical, or diagonal line of three.Source: Tic-Tac-Chess · p. 25

F Corpus Statistics

• The controlled-diversity pool is separate from the four-game SPSD corpus; its 15 held-out games supply no training rows, and its 45 variants and 50 environments are summarized by rule family and structural range.Figure 5 groups the variants by edited rule family, while Table 7 reports structural-property statistics for the environments.Source: F Corpus Statistics · p. 25, Figure 5 · p. 26, Table 7 · p. 26

G Training Hyperparameters

• Table 8 covers controlled game variants, ExIt experts, trace construction, and Qwen3-4B SPSD post-training; trace parameters stay fixed across ablations.The ablations vary the source game, downstream training objective, or teacher MCTS budget.Source: G Training Hyperparameters · p. 26, Table 8 · p. 26

G.1 Teacher and Optimization Selection

• Table 9 marks the matched main-experiment configuration with * and the highest-mathematics-transfer configuration with †; runtime settings are outside the selection grid.The search records explored training choices and settings held fixed.Source: G Training Hyperparameters · p. 26, Table 9 · p. 27
• 36.6 mean accuracy over six mathematics benchmarks, versus 31.9 for the matched fixed-teacher run, is reached by EMA decay 0.998 on Qwen3-4B-Base.The decay follows an inverted U; larger adapters beyond r=128, larger batches, a weaker divergence anchor, and twice the step budget did not reach that level, while forward KL alone collapses training.Source: Table 10 · p. 27, §G.1 · p. 27

H Prompt Templates

• Move prompts use rules, board/state text, legal-action handles and descriptions in fixed field order, ending in a boxed legal move; state questions replace the move instruction.The shared contract supplies the game-specific rules and state, and requires exactly one legal action.Source: H Prompt Templates · pp. 27–28, Figure 6 · p. 28

I RuleBot-Distill Example

• RuleBot-Distill keeps target selection out of the student prompt: rule-bot context verifies a reference completion teacher-side, and inference supplies only the prompt.This arm uses rule-bot supervision without MCTS search.Source: I RuleBot-Distill Example · p. 28, Figure 8 · p. 29

J Grounded MCTS Example

• Grounded MCTS records show the student the board task and legal-action contract while comparisons and replay evidence remain teacher-only; raw search telemetry is hidden.The reference completion ends with the boxed handle.Source: J Grounded MCTS Example · pp. 28–29, Figure 9 · p. 30

K SPSD-dataset Task Taxonomy

• The six state-question tasks share a replay-backed answer contract but probe different state facts, with the taxonomy comparing question type, replay-computed evidence, and compact expected answer.Source: Table 11 · p. 30, K SPSD-dataset Task Taxonomy · p. 30