PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

Allen Nie, Anirudhan Badrinath, Nicholas Tomlin, Timothy Dai, Carissa Yip, Rose E Wang, Emma Brunskill, Chris Piech

arXiv:2608.14851v1cs.AIcs.LG

TL;DR

Automatically generated chess puzzles may have uncertain long-term learning value, making effective practice-set selection difficult. This paper uses offline reinforcement learning on 1.5 billion puzzle-solving histories to learn puzzle-selection policies, finding roughly 110.1% improvement over the existing system for users under 1500 Elo.

  • Problem

    It remains unclear whether automatically generated chess puzzles produce meaningful, lasting learning gains, motivating methods to identify and filter low-value practice content.

  • Method

    Using 1.5 billion puzzle-solving histories, the authors train an offline reinforcement-learning policy to select among 441,113 puzzles and evaluate it offline alongside expert-quality assessments.

  • Results

    Roughly 110.1% improvement over the existing system was observed for users under 1500 Elo, with statistically significant gains for beginners rated 100–1000.

  • Takeaways & Limitations

    The results suggest that general user interaction data can reveal practice-item pedagogical value and support automated chess-puzzle selection.

  • Takeaways & Limitations

    Because randomized trials were impractical, direct human evaluations are still needed to fully validate the system’s impact on chess learners.

Abstract

from arXiv · show

Learning and skill mastery require extensive and deliberate practice. In many learning settings, producing high-quality pedagogical materials can require a high level of domain expertise and be very time-consuming. Pedagogical materials often need to train students to engage in different thinking patterns. In some domains, such as chess, puzzles are used to help students practice their skills in calculating the next moves and recognizing known patterns on a board. Giving students a practice set of puzzles to help them learn different modes of thinking is challenging because the teacher needs to carefully balance between different motifs and how many look-ahead steps a student needs to perform. Popular online platforms like Chess.com and Lichess offer players millions of puzzles. Unlike chess tactics puzzles procured by human experts, where chess beginners can learn valuable insights, these puzzles are automatically generated and often regarded as having low pedagogical value. These platforms also rely on a heuristic to recommend puzzles to users for practice. Using the user history data over an entire year, a total of 1.5 billion puzzle-solving histories, we learn the pedagogical value of a puzzle and how to automatically choose a set of puzzles to better support chess learners using insights from offline reinforcement learning. We show that using offline policy evaluation, our trained policy has significant impact on beginners with puzzle-solving Elo range of 100--1000, particularly for the group of beginners whose learning growth was stagnant. We also performed a qualitative analysis of the puzzles discovered by our model by collecting annotation ratings from expert chess players. The success of our pipeline shows promise for a future where we can understand the pedagogical values of practice items given general user interaction data.

1 Introduction

Chess puzzles can support deliberate practice by isolating difficult concepts, but automatically generated online puzzles may lack the careful curation of expert-authored materials. This work uses 1.5 billion Chess.com puzzle-solving attempts to learn and evaluate an offline-RL policy for puzzle selection, while distinguishing growing and stagnant learners.

  • Motivation: High-quality, curated learning materials can help students learn at similar rates through extensive deliberate practice.The introduction frames material quality as a central factor in learning and skill mastery.
  • Chess puzzles: Chess tactics puzzles isolate difficult concepts and train players to plan multiple moves in sequence.These subgames are presented as common and useful learning materials for beginners.
  • Problem: Online platforms automatically generate puzzles from played games, assign Elo ratings, and randomly present puzzles within a fixed difficulty range.This contrasts with tactics books, whose puzzles are curated through expert deliberation, review, and editing.
  • Approach: 1.5 billion puzzle-solving attempts from 3.1 million users over one year support learning and evaluating an offline-RL puzzle-selection policy.The policy is evaluated using a holdout portion of the historical interaction dataset.
  • Learner groups: Users separate into a growth group whose Elo gradually increases with more puzzles and a stagnant group whose Elo remains flat.This grouping is reported as consistent with the finding in Wang et al. (2022).

2 Related Work

Prior work has used chess to evaluate AI and begun uncovering chess knowledge and playing styles, but has rarely leveraged such knowledge to teach humans. Related educational systems use reinforcement learning to adapt instruction, including offline RL and subgroup-aware personalization in math tutoring.

  • Chess AI and teaching: Chess has long served as a benchmark for evaluating AI progress, whose resulting systems have achieved superhuman game-playing abilities.The passage contrasts this progress with limited work on using AI knowledge for human teaching.
  • Chess AI and teaching: Schut et al. and McGrath et al. investigated chess knowledge learned in AlphaGo and AlphaZero, while McIlroy-Young et al. modeled chess playing styles.These studies represent early efforts to interpret or characterize knowledge learned by chess systems.
  • RL for education: Reinforcement learning has been used in automated teaching systems to adaptively select instructional materials for students.Examples include adaptive tutoring for college mathematics and systems deciding when and how to provide hints.
  • Terminology and scope: The paper defines “rating” and “Elo” as Chess.com puzzle Elo scores assigned to both users and puzzles, using no information outside tactics.User and puzzle scores initially change, but puzzle Elo eventually becomes fixed.
  • RL for education: Offline RL has augmented fraction learning in math games, while other work studies subgroup-specific impacts, meta-exploration, and feedback personalization in math tutoring.The cited systems span fraction learning, differential subgroup effects, interactive student programs, and hint provision.

3 Data

The dataset contains 1,536,254,297 puzzle histories from 3,132,428 active Chess.com users over one year. It characterizes puzzle serving, puzzle structure and motifs, and relationships between puzzle experience and Elo growth.

  • Dataset: 1,536,254,297 puzzles from 3,132,428 unique active users were recorded over one year, covering 441,113 unique puzzles.Users played an average of 490.4 puzzles, and 96.9% occurred within three minutes of another puzzle.
  • Puzzle Serving: Chess.com serves each user a puzzle sampled uniformly from puzzles within ±200 points of the user’s puzzle-solving Elo.Both users and puzzles receive Chess.com tactics or puzzle-solving Elo scores, rather than ratings from human games.
  • Chess Puzzles: Puzzle move counts range from 1 to 16 and average 2.6, with each puzzle presenting an initial position and a required solution.Solutions require finding the correct move or moves while adhering to the puzzle’s main goal or task.
  • Chess Puzzles: 26% of the 441,113 puzzles have no motifs, while puzzles average 5.6 motifs and range from zero to 56.Motifs describe the primary strategy or skill required, such as Exchange Sacrifice or Promotion, but provide nonexhaustive insight.
  • Analyzing User Elo Growth: Higher puzzle counts generally correlate with higher Elo, while growth trajectories differ across Elo groups, with the lowest-Elo Growth Group initially dipping before gaining.The analysis covers players’ first 50 recorded puzzle plays and defines Growth and Stagnant Groups by within-range Elo-change percentiles.

4 Policy Learning

The paper formulates chess-puzzle recommendation as offline policy learning from logged trajectories, using a non-Markovian transformer policy over user learning histories and a large discrete puzzle set. It trains advantage-weighted actor-critic components while addressing offline distribution shift and computational costs in the action space.

  • Offline policy formulation: The recommendation task is modeled as an offline decision process using trajectory data collected by a behavior policy, without access to transition dynamics or online interaction.Policy performance is defined by expected discounted return, with corresponding action-value and value functions.
  • Policy architecture: The non-Markovian transformer policy represents fixed-length user puzzle histories and learning progress, selecting among 441,113 chess puzzles.The state is a continuous k-dimensional embedding, while actions are individual puzzles.
  • Offline policy objective: To mitigate extrapolation error, the policy is penalized for deviating from the behavior policy through statewise KL divergence.This regularization is intended to keep the learned policy close to the data-supported behavior distribution.
  • Offline policy objective: The method uses an advantage-weighted actor-critic objective, projecting a closed-form policy proportional to πb(a|s) exp(advantage/β) into the parameterized policy space.Value and action-value networks are trained offline from dataset transitions using expectile regression and a separate action-value objective.
  • Implementation: The policy combines user and puzzle representations with a causal transformer, while minibatch puzzle embeddings approximate normalization over the approximately half-million-puzzle action space.User features include Elo and correctness; puzzle features include board positions, puzzle identities, and first moves.
  • Evaluation and reward: During evaluation, puzzle embeddings are computed once for the entire action space and reused across user sequences rather than recomputed per prediction.The reward quantifies learning benefit using correctness and relative puzzle difficulty, assigning no reward to incorrect responses.

5 Evaluation

The evaluation combines offline policy evaluation with expert human assessment because randomized trials are impractical. It analyzes policy improvements across learner groups and uses expert-calibrated LLM judges to scale puzzle-quality annotation.

  • Evaluation approach: The study uses offline policy evaluation and human evaluation instead of a randomized controlled trial because recruiting participants and measuring learning effects would be difficult and time-consuming.The evaluation therefore assesses whether the learned policy could meaningfully affect learning without running a live experiment.
  • Offline policy evaluation: The offline evaluation applies importance sampling using the behavior and trained policy distributions, with a one-step approximation to avoid impractical variance from long sequences.Traditional importance sampling over sequences of 256 actions, or even shorter sequences, is described as impractical because of induced estimator variance.
  • Offline policy evaluation: Policy improvements are analyzed by puzzle-solving Elo groups and learner-growth groups, defining stagnant growth as no more than 50 Elo points across the sequence.This breakdown examines whether improvements differ across user abilities and learning patterns.
  • Expert evaluation: Puzzle quality is assessed with a rubric covering sequential calculation, iconic-pattern recognition, and appropriateness for a player’s rating.The rubric was designed with two project chess experts rated around 1900 and 2400 USCF.
  • Expert evaluation: Eight strong chess players annotated 30 puzzles, including five officially titled players with USCF Elo ratings from 1990 to 2576.The group included two USCF Experts, one USCF National Master, one FIDE Master, one FIDE International Master, and two FIDE Grandmasters.
  • LLM-assisted evaluation: The study trains 8 LLM annotators with DSPy to imitate the judgments of the 8 chess experts and scale evaluation beyond costly expert annotation.DSPy uses cross-validation and random search to select expert examples that maximize the annotation metric.

6 Experimental Results

The learned policy is evaluated through offline policy evaluation and an automated annotation pipeline. It improves outcomes especially for lower-Elo players, while recommending puzzles judged slightly more enjoyable, harder, and better for calculation and pattern recognition, with qualitative findings treated as preliminary.

  • Evaluation approach: Offline policy evaluation and automated annotation provide objective and qualitative comparisons between the learned policy and Chess.com’s behavior policy.The evaluation uses importance sampling for quantitative comparison and annotation ratings for recommendation-quality analysis.
  • Offline policy evaluation: 100–1000 Elo buckets show statistically significant and consistent improvement over the behavior policy, while margins decrease at higher Elo.At higher Elo, the learned policy becomes neutral relative to the behavior policy.
  • Offline policy evaluation: 110.1% improvement is reported over the existing system for the more than 80% of users below 1500 Elo.The user base and evaluation data are concentrated at lower Elo levels, where puzzle selection is described as more critical to learning.
  • Offline policy evaluation: The trained policy’s relative improvement is approximately similar across growth and stagnant groups, with the smallest Elo bucket favoring stagnant users relatively.For all other Elo buckets, the trained policy performs relatively better for growth-group users.
  • Qualitative analysis: LLM judges rate the trained policy’s puzzles as slightly more enjoyable and harder, with somewhat higher calculation and pattern-recognition scores.These qualitative evaluations are preliminary because the LLM judges were calibrated on a small set of expert annotations.

7 Conclusion and Future Work

The study uses 1.5 billion puzzle-solving histories and offline reinforcement learning to select chess puzzles, while offline evaluation and LLM analysis suggest differences from Chess.com’s policy that require direct human validation.

  • Conclusion: 1.5 billion puzzle-solving histories support learning chess-puzzle pedagogical value and developing an automated selection system with offline reinforcement learning.The system is designed to learn pedagogical value from user interaction data.
  • Conclusion: Offline policy evaluation and LLM-based qualitative analysis both suggest differences between the learned policy and Chess.com’s existing policy.The passage does not specify the direction or magnitude of these differences.
  • Future Work: Direct human evaluations are still needed to fully validate the automated puzzle-selection system.The authors identify human evaluation as necessary future validation.

A Appendix · A.1 Human Annotation Guidelines

The appendix defines an expert-annotation protocol for comparing the pedagogical quality of learned-policy puzzles with Chess.com’s tactics trainer. Annotators rated puzzles across calculation, pattern recognition, informativeness, rating-appropriateness, fun, and holistic quality.

  • A.1 Human Annotation Guidelines: Eight titled and expert-level annotators evaluated puzzles supplied through spreadsheets of puzzle URLs and player Chess.com Elo information.The protocol was designed to compare puzzles recommended by the learned policy with those served by Chess.com’s existing tactics trainer.
  • A.1 Human Annotation Guidelines: Calculation uses a 1–4 scale to assess reasoning over long move sequences or lines with high branching factors.Annotators first identified whether each puzzle primarily tested calculation, pattern recognition, both, or neither, and rated only the applicable criterion.
  • A.1 Human Annotation Guidelines: Pattern recognition uses a 1–3 scale to measure the unusualness or interest of a puzzle’s tactical motif relative to common motifs.The highest score applies to genuinely unusual patterns or common patterns presented with an unusual twist.
  • A.1 Human Annotation Guidelines: Informativeness uses a 1–5 scale to assess whether the solution follows the most challenging and instructive continuation rather than an easier secondary line.An example rated 1 avoids the most challenging continuation despite requiring the solver to calculate it to justify an earlier move.
  • A.1 Human Annotation Guidelines: Rating-appropriateness uses a 1–5 scale to judge puzzle difficulty from the perspective of a player at the target rating.Annotators were instructed to assess appropriateness relative to the recommended player’s rating rather than their own.
  • A.1 Human Annotation Guidelines: Fun uses a 1–5 scale to measure how enjoyable a puzzle would likely be for a player at the target rating.This criterion evaluates expected enjoyment from the target player’s perspective.
  • A.1 Human Annotation Guidelines: Quality is a holistic 1–5 score assessing the puzzle as a whole independently of difficulty.The preceding criteria inform, but do not strictly determine, this overall judgment, and Figure 4 summarizes the rating ranges.

A.2 LLM Judge Prompt Fields

Each LLM judge is prompted to imitate an expert annotator and receives three inputs: annotation guidelines, target-user puzzle Elo, and an ASCII puzzle-board representation. It outputs valid JSON with integer scores from 1 to 100 for six categories, with DSPy supplying selected expert-rated examples during optimized execution.

  • Judge setup: Each judge receives a system message identifying the imitated expert by chess title, when applicable, and USCF Elo rating.The message also instructs the model to apply the annotation guidelines from that expert-level perspective.
  • Input fields: The annotation-guidelines field provides the complete rubric for rating calculation, pattern recognition, informativeness, rating-appropriateness, quality, and fun.
  • Input fields: The target-user puzzle Elo field supplies the current Chess.com puzzle Elo as the reference skill level for rating-appropriateness and fun.
  • Input fields: The puzzle-board field contains an ASCII grid generated from the puzzle’s FEN string, followed by its first move and a request to find the best continuation.
  • Judge output: The output field requests valid JSON containing integer scores from 1 to 100 for all six categories.During optimized judge execution, DSPy also supplies selected expert-rated examples.

A.3 Annotation Questionnaire

The annotation questionnaire asked chess experts to assess each puzzle’s emphasis, calculation demands, pattern-recognition value, informativeness, level appropriateness, quality, and fun, with optional written justification.

  • Questionnaire procedure: Each puzzle occupied one spreadsheet row, with one column per question, and annotators supplied 1–2 sentence justifications in an additional-notes field.The paper states that Table 4 reproduces two completed rows as an illustration.
  • Questionnaire items: Annotators identified each puzzle’s primary emphasis as Calculation, Pattern Recognition, Both, or Neither.The questionnaire also defined calculation as thinking through the entire solution before the first move.
  • Questionnaire items: Pattern recognition was rated by whether a puzzle taught useful or unusual motifs rather than banal or overdone ones.Annotators could select N/A for this 1–3 rating.
  • Questionnaire items: Annotators rated informativeness, target-skill-level appropriateness, overall quality, and fun on 1–5 scales.Informativeness concerned whether the solution followed the right path to clarify the puzzle’s point; quality was independent of Elo appropriateness.

A.4 Selecting Illustrative Examples

The authors contrast a policy-recommended puzzle with a Chess.com puzzle using ratings from eight annotators. The policy example was unanimously judged highly unusual, while the Chess.com example received more disputed pattern-recognition ratings despite comparable overall quality.

  • Example selection: The examples were selected from puzzles independently rated by all eight annotators, comparing average pattern-recognition scores between the authors’ policy and Chess.com’s system.The analysis excluded annotator ratings from the average quality calculation.
  • Policy-recommended example: Every annotator gave the policy-recommended puzzle the maximum pattern-recognition score, describing its move combination as unusual and non-obvious.Annotators distinguished it from puzzles that merely test recognition of a standard pattern.
  • Chess.com example: The Chess.com puzzle received pattern-recognition ratings ranging from “banal/boring pattern” to “interesting and useful,” revealing disagreement among annotators.The disagreement concerned whether the puzzle’s underlying pattern was sufficiently distinctive, while free-text notes reinforced the contrast.
  • Cross-example comparison: The two puzzles had comparable holistic quality scores, indicating that the clearest difference between them appeared in pattern recognition rather than overall puzzle quality.Table 6 presents the full distributions across all eight annotators to make agreement and disagreement visible.
Loading 2608.14851v1…