PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess

Szymon Miłosz, Piotr Duch, Szymon Grabowski

arXiv:2608.27757v1cs.LGcs.AI

TL;DR

Searchless chess networks imitate search despite being deployed without it, leaving a mismatch between training targets and direct play. The paper uses self-play reinforcement learning with forward-KL prior-directed exploration and an uncertainty-adaptive temperature. In about 2,000 steps, tactics improve while searchless strength holds at or slightly above the base, but tactical accuracy and playing strength dissociate.

  • Problem

    Searchless networks imitate MCTS despite deployment without search, creating target and state-distribution mismatches with direct single-pass play.

  • Method

    The paper fine-tunes Chessformer with self-play reinforcement learning, forward mass-covering KL toward its MCTS prior, and value-uncertainty-adaptive sampling temperature.

  • Results

    About 2,000 steps improve tactical accuracy while searchless strength holds at or slightly above the base; tactical accuracy and playing strength dissociate.

  • Takeaways & Limitations

    Anchoring exploration to the prior supports tactical improvement without requiring a strength increase, while unregularized self-play collapses and forward KL retains harder solutions.

  • Takeaways & Limitations

    Forward KL requires the reference’s full distribution, which is straightforward for enumerable legal moves but expensive for language-model vocabularies.

Abstract

from arXiv · show

Searchless chess networks reach human master strength from a single forward pass by imitating a stronger teacher: the strongest, Leela Chess Zero's (Lc0) released Chessformer, distills the visit counts of an AlphaZero-style Monte Carlo Tree Search (MCTS). Imitating a search is a poor proxy for playing without one, so we fine-tune for single-pass strength with self-play reinforcement learning (RL). Its exploration is usually supplied by an entropy bonus, the reverse Kullback-Leibler (KL) divergence to uniform. We replace it with a forward, mass-covering KL toward the network's own MCTS prior (prior-directed exploration), so exploration covers the moves the prior judges promising, and pair it with an entropy-adaptive sampling temperature, set by the value head's outcome uncertainty, that sharpens once a position is decided. In about two thousand steps it raises puzzle accuracy from 93.9% to 94.9% on a 100,000-puzzle suite and mate-in-four accuracy from 77% to 81% while holding searchless strength at or slightly above the base. Measuring tactical accuracy and playing strength together across a matched-compute sweep, we find the two dissociate: accuracy gains fall in a one-point band while ratings straddle the base, and a control fine-tuned on puzzles alone posts the study's largest tactical gains while shedding roughly 260 Elo; a better puzzle-solver is not thereby a stronger player. Distribution-level measurements show what anchoring buys: without a regularizer self-play collapses onto a single line of play, and the puzzles newly solved are the near misses whose winning move the prior kept alive. The forward-KL prior tops the rating ladder, statistically tied with a reverse-KL anchor that concentrates twice as hard and drops the hardest solutions the mass-covering prior keeps in support.

1 Introduction

Searchless chess networks inherit mismatches from imitating search, motivating self-play reinforcement learning that directly optimizes single-pass play. The paper replaces uniform entropy exploration with prior-directed coverage of promising moves and evaluates its effects on tactics, strength, and distributional collapse.

  • Motivation: Chessformer distills MCTS visit counts, but those targets and the resulting training states can mismatch direct searchless play.The target is only a noisy proxy for the move maximizing win probability under direct play, while the state distribution also differs.
  • Motivation: Entropy-based exploration spreads probability across all legal moves, including many that strong players would reject.With large branching factors, this produces an undirected exploration signal and can waste the exploration budget.
  • Method: Self-play fine-tuning optimizes Chessformer’s single-pass strength using truncated importance sampling and a one-step TD(0) value head with an EMA target.The policy, rollouts, and value bootstrap use one network evaluation per position, matching searchless deployment.
  • Method: Prior-directed exploration replaces the entropy bonus with forward, mass-covering KL toward the network’s MCTS-derived prior.It keeps moves judged promising by the prior in support, while search-derived policies provide breadth over several candidates.
  • Method: An entropy-adaptive temperature uses value-head win/draw/loss uncertainty to remain exploratory in contested positions and sharpen play once positions are decided.This avoids spending rollout and learning capacity on already settled positions.
  • Results: About 2,000 steps improve tactics while preserving or slightly improving searchless strength, and the gains depend on retaining promising prior-supported moves.The study also finds tactical accuracy and playing strength dissociate, while unregularized self-play collapses onto one line and the forward prior retains the hardest solutions.

2 Method

The method replaces undirected entropy exploration with forward-KL coverage of a search-derived prior and uses outcome uncertainty to adapt sampling temperature. Single-step self-play updates combine policy-gradient learning with TD(0) value bootstrapping, a frozen prior, and EMA targets.

  • Policy-gradient background: Unregularized policy gradients can become deterministic because unsampled moves lose both reward observations and gradient pathways.Entropy regularization sustains exploration, but its uniform reference spreads probability across all legal moves regardless of promise.
  • Prior-directed exploration: Forward KL toward πbase keeps probability on every promising move in the prior while permitting mass outside low-prior regions.Unlike reverse KL, it is mass-covering and zero-avoiding, so it resists collapse onto a single prior mode; its coverage depends on the prior’s support.
  • Prior-directed exploration: The approach requires an exploratory reference, and the MCTS-derived prior supplies breadth because search visit distributions retain mass on several reasonable moves.The released network inherits this breadth by matching those visit-count distributions, while the regularization strength also depends on β and reward scale.
  • Prior-directed exploration: Forward KL requires the reference’s full distribution, which is straightforward for enumerable legal chess moves but expensive for language-model vocabularies.This computational cost helps explain why reverse KL is more common in large-scale post-training.
  • Single-step self-play recipe: Self-play fine-tuning uses one network evaluation per position, with πbase frozen, πθ online, and EMA weights providing slow targets for a one-step TD(0) update.The sampled behavior policy is tempered by τ(s)=H(p̄θ(s)); decisive positions sharpen toward deterministic play, while uncertain positions preserve exploration.
  • Single-step self-play recipe: The entropy-adaptive temperature controls decisiveness within games: it reduces stochasticity in decided positions while retaining sampling diversity in contested ones.Raw WDL entropy spans approximately [0,1.1], and the temperature is applied through μ ∝ πθ^(1/τ).

3 Experiments

Across matched-compute experiments, prior-anchored self-play improves tactical behavior while preserving or modestly improving searchless strength, but tactical accuracy and playing strength can dissociate. Distributional analyses attribute the gains to retaining probability on promising winning lines rather than merely sharpening the top move.

  • Searchless playing strength: +16.6±8.3 Elo: the fixed-temperature forward-KL prior tops the searchless rating ladder, with a slender 0.52 head-to-head score against the base.The reverse-KL anchor is statistically tied at +16.2±8.3 Elo, so the study does not claim the forward direction is stronger.
  • Tactical accuracy, retention, and coverage: 5.13% error at β = 10−2: the forward-KL coefficient sweep improves over both the frozen base’s 6.15% and the unregularized run’s 5.94%.The response is U-shaped in logβ, with its trough near β ∈ [3·10−3, 3·10−2].
  • Tactical accuracy, retention, and coverage: 33% higher solution probability on ≥2400 puzzles: the forward prior increasingly outperforms the reverse anchor as puzzle difficulty and mate depth rise.The reverse anchor retains solutions better on shorter lines but falls below the base on the hardest sets.
  • Tactical accuracy, retention, and coverage: 22,000 draws to cover 95% of depth-4 mass for the frozen base versus 1,400 for forward-KL and 700 for reverse-KL, showing sharply different policy concentration.The sampled strength ladder also spans approximately 670 Elo and tracks move entropy inversely.
  • Searchless playing strength: 0.22 forward-KL divergence to the base versus 0.60 for reverse-KL and 1.26 for entropy, indicating that forward KL covers the prior’s mass more faithfully.The comparison uses fixed-temperature evaluation policies, making divergence comparable across arms.
  • Tactical accuracy, retention, and coverage: 1,405 and 1,378 newly solved puzzles: the two prior-anchored models recover failures whose winning moves the base still assigns non-negligible probability.This supports the retention-gated mechanism: improvement must be sampled before it can be reinforced.

4 Related Work

Prior work removes test-time search through distilled policies, shallow procedures, or implicit search, while exploration research largely regularizes toward uniform or uses forward KL for non-exploration purposes. This paper instead positions forward-KL prior anchoring as a search-derived exploration regularizer.

  • Searchless chess: Searchless chess methods replace large test-time searches with single-pass policies, one-ply lookups, or fixed shallow decoding procedures.
  • RL for games and pretrained policies: Self-play methods such as AlphaZero and MuZero use search to guide learning, whereas this work starts from a search-trained network that does not optimize its own searchless play.
  • Exploration regularizers: Entropy bonuses and related exploration regularizers spread action probability toward uniform, while intrinsic-reward methods alter rewards rather than action-distribution regularization.
  • Exploration regularizers: Trust-region methods use KL in either direction to stay near a reference, but the exploration regularizers discussed here point toward uniform distributions.
  • Forward-KL regularization: Forward KL is established for smoothing, preference optimization, and distillation, but these applications use it for regularization, alignment, or distillation rather than exploration.

5 Conclusion

The paper concludes that self-play RL with a forward, mass-covering KL toward the MCTS prior improves tactics while preserving searchless strength, but tactical accuracy and playing strength dissociate. The mechanism depends on a strong, enumerable prior and remains limited to one domain and base network, with additional risks from action-space cost, prior mismatch, and self-play distribution shift.

  • Conclusion: Forward-KL prior anchoring makes self-play exploration cover moves judged promising by the MCTS prior, avoiding both uninformed spreading and mode collapse.The reward sharpens within the prior’s supported moves.
  • Conclusion: In about two thousand steps, puzzle accuracy and deeper forced-mate performance improve while searchless playing strength remains at or slightly above the base.The forward-KL prior tops the rating ladder, statistically tied with the reverse-KL anchor.
  • Conclusion: Tactical accuracy and playing strength dissociate: improving puzzle accuracy does not by itself establish a stronger player.Without regularization, self-play collapses onto a single line; anchoring preserves reachable near-miss solutions.
  • Scope and requirements: The recipe is scoped to settings with an enumerable reference and an exploratory prior worth covering, especially games and combinatorial planning.Language models can meet the enumerability condition only by paying for the reference’s full next-token distribution.
  • Limitations: The evidence covers one domain and one base network, while forward and reverse KL remain tied on strength and the absolute gains are small.The two-thousand-step procedure adjusts the base network rather than remaking it.
  • Limitations: Exact forward KL requires a modest categorical action set, and a weak or biased prior may be faithfully covered rather than corrected.The authors also require agreement between tactical and playing-strength evaluations before claiming improvement.
  • Limitations: Self-play can narrow opening diversity, while the frozen prior is least reliable on off-distribution states reached during training.Persistent root randomization and annealing the KL weight are identified as safeguards or extensions.

A Notation, MDP, and networks

The paper models chess as a deterministic episodic two-player zero-sum MDP and uses a network with policy and categorical win/draw/loss value heads. Training maintains online, frozen-reference, and EMA networks, with the EMA target stabilizing value learning.

  • MDP: Chess is represented as an episodic, two-player, zero-sum Markov decision process with deterministic legal-move transitions.States include board, side to move, and auxiliary rule information; rewards are terminal with γ = 1.
  • Network heads: The policy head defines a distribution over legal moves, while the value head outputs a categorical win/draw/loss distribution.
  • Value perspective and model copies: After a move, the opponent’s score is represented by 1 − v(s′), with win and loss probabilities swapped.The setup keeps an online model, a frozen Chessformer prior, and an EMA model.
  • Value learning: The EMA model provides slowly moving bootstrap targets, decoupling value targets from rapidly changing online parameters to stabilize value learning.

B Base network: Chessformer

The adapted model is an Lc0-released Chessformer transformer checkpoint, while interpretability evidence comes from a different smaller transformer. The base network was supervised-distilled from AlphaZero-style MCTS visit counts and game outcomes rather than trained through online RL.

  • Architecture: The fine-tuned checkpoint is a 15-layer encoder-only transformer with 1024-dimensional embeddings, 32 attention heads, and 1536-wide MLPs.It matches the reported 191M-parameter Leela-CF dimensions.
  • Interpretability scope: Interpretability findings come from a different, smaller 768-dimensional, 24-head Lc0 transformer rather than the exact adapted checkpoint.They are therefore read as properties of the broader square-token policy lineage.
  • Training provenance: The released base network was supervised-distilled by fitting its policy to AlphaZero-style MCTS visit-count distributions and its value head to win/draw/loss outcomes.Its data come from searched self-play, but the network itself imitates recorded search statistics rather than learning through an online RL loop.

C Why the base network’s prior is exploratory

The base network’s single-pass prior remains exploratory because it distills MCTS visit distributions, which spread probability across promising moves and sharpen when search is decisive.

  • Why the base network’s prior is exploratory: MCTS exports a visit-count distribution rather than only its best move, preserving mass on several moves with appreciable prior and value.PUCT exploration keeps moves with non-negligible prior in consideration, while temperature controls sharpening when the search is confident.
  • Why the base network’s prior is exploratory: Cross-entropy distillation transfers the breadth of MCTS visit-count targets to the network’s single-pass policy.Forward KL penalizes dropping moves visited by the search, making the training objective mass-covering.
  • Why the base network’s prior is exploratory: The resulting πbase is exploratory by construction and sharpens only where the distilled search was decisive.This provides the search-derived breadth that prior-directed forward-KL exploration is designed to cover.

D Self-play data generation

Self-play generates training data from parallel games advanced one ply per iteration, with only initial random openings used to stagger positions across game phases.

  • Self-play data generation: N games run in parallel, and each game advances by one ply per training iteration.At training start, each slot plays k ∼Uniform{0,1,...,60} full moves sampled from πbase before self-play proceeds.
  • Self-play data generation: After termination, games restart from the standard initial position so the batch continuously supplies fresh self-play transitions.

E Importance sampling, advantage, and the single-step update

The update corrects temperature-induced off-policy sampling with truncated importance weights, uses an unnormalized TD advantage and value regression, and reduces PPO to a single-step policy gradient.

  • Importance sampling: Moves sampled from tempered µ create an off-policy gap relative to πθ, corrected with per-sample truncated importance weights.The reference distribution is explicitly enumerable over legal chess moves, avoiding the unbounded-weight issue that arises when reference mass is dropped.
  • Advantage: The one-step TD(0) advantage uses the EMA successor value and is applied without advantage normalization.The method separates advantage normalization from mean-centering of the value target.
  • Single-step update: With one gradient step per fresh batch, the PPO clipping term never activates because r(θ) ≡1.The resulting surrogate is an advantage-weighted policy gradient, while truncated sampling weights remain necessary because µ differs from πθ.
  • Value learning: The value head regresses toward a point-of-view-flipped EMA target, with ρ-weighting correcting the action expectation from µ to πθ.This makes the regression target on-policy for the online policy’s one-step value.
  • Importance sampling: The forward KL requires an expectation under the reference distribution, which is tractable because chess has only a few legal actions.

F Strength evaluation protocol

Strength is evaluated through a closed BayesElo round-robin of searchless agents using paired openings, with ratings anchored to the released Chessformer network.

  • Strength evaluation protocol: Searchless Elo is fit with BayesElo from a closed round-robin in which every move is the argmax of one policy forward pass.The evaluation pool includes the frozen base and fine-tuned agents.
  • Strength evaluation protocol: Each model pair plays 200 games from 100 paired UHO openings, with each side additionally playing 1,000 games against the base.Openings are played from both colors and games are capped at 512 plies.
  • Strength evaluation protocol: The base is fixed at zero and mapped to 2374±37 using the Chessformer paper’s rating, while reported ∆Elo intervals are approximately ±8.3.The external anchor shifts all ratings together but leaves within-pool differences unchanged.
  • Strength evaluation protocol: Ratings reflect game-sampling variance but not training-run variance because fine-tuning is not repeated across seeds.Consequently, rankings among the top cells should be treated as indicative rather than seed-stable.

G Training configuration

The study fine-tunes searchless chess networks for 2,000 optimizer steps using specified AdamW, layerwise learning rates, and large-scale parallel self-play.

  • Optimization: 2,000 optimizer steps use AdamW with β1 = 0.9, β2 = 0.98, zero weight decay, and a 200-step linear warmup.The peak learning rates are held constant for the remaining 1,800 steps.
  • Optimization: Layerwise learning rates are 3×10−6 for the backbone, 1×10−5 for the final block and policy head, and 3×10−5 for the value head.
  • Self-play, data, and sweep: 2,000 self-play steps consume approximately 2.05M transitions from 1,024 parallel games, with games capped at 512 plies and capped games scored as draws.
  • Self-play, data, and sweep: The headline forward-KL coefficient is β = 10−2, selected on a 10,000-puzzle suite and confirmed at scale by Figure 5.Each Table 1 baseline is reported at its own swept-best coefficient.

H Additional results

Additional results connect policy concentration with tactical recovery across opening positions and exclusive puzzle cases. The forward prior retains broader support than the reverse anchor while recovering distinct near-miss solutions.

  • Exclusive recoveries: In five of six exclusive-recovery positions, the base value head separates the winning move from the played move by 17–50 percentage points, matching the Stockfish oracle.The queen-sacrifice panel illustrates this most clearly: the value head scores the sacrifice 100% while the policy ranks it third.
  • Puzzle accuracy: The 10,000-puzzle coefficient-selection table reports frozen-base accuracy of 93.52±0.48 and unregularized accuracy of 93.99±0.47 with adaptive τ.The unregularized τ ≡1 run scores 94.04±0.46.
  • Coverage beyond the initial position: At three opening positions, the unregularized policy collapses to one line, while the reverse anchor reaches 95% coverage in fewer draws than the forward prior.The forward prior requires 61–73, 19–24, and 142–217 draws across the Ruy Lopez, Najdorf, and QGD positions, versus 26–34, 7–8, and 36–61 for the reverse anchor.
  • Exclusive recoveries: The forward-KL prior recovers a puzzle that the reverse-KL anchor does not, while another listed recovery belongs to the reverse anchor.The figure materials present exclusive recoveries in both directions rather than a single uniformly superior solver.
Loading 2608.27757v1…