PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Prompt Optimization of Large Language Models With Chess

Timothée Lesort, Alejandra López de Aberasturi Gómez, Tristan Karch, Tom Veniat, Philippe Modard, Karl Tuyls, Ludovic Denoyer

arXiv:2610.00416v1cs.AIcs.LG

TL;DR

LLM benchmarks can saturate, risk contamination, or require costly grading and infrastructure, challenges that recur during automatic prompt optimization. This paper introduces a renewable chess-puzzle benchmark for optimizing frozen LLM prompts and evaluates six algorithms across eight target models. Four models show significant puzzle-accuracy gains, but gains do not guarantee transfer and no clear reduction in game-play regret is found.

  • Problem

    APO evaluation inherits benchmark risks including saturation, contamination, and costly grading or execution, motivating cheap, deterministic tasks that can be renewed as models improve.

    Source: §1 · p. 1

  • Method

    The paper benchmarks prompt optimization for frozen LLMs using chess puzzles, engine-based scoring, and evaluations across six algorithms and eight target models.

    Source: §1 · p. 2

  • Results

    Four models show significant puzzle-accuracy gains, reaching 7.57 percentage points, while no clear reduction in mean regret appears in evaluated game-play rollouts.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Chess supports inexpensive repeated evaluation with renewable, adjustable-difficulty puzzles and lets researchers test whether puzzle gains carry over to play.

    Source: §6 · p. 14

    The study uses a deliberately minimal common baseline rather than testing whether APO outperforms the strongest model-specific manual prompts.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55\% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around \$800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts (https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess).

1 Introduction

The paper motivates chess as an affordable, renewable benchmark for automatic prompt optimization, addressing limitations of static or costly LLM evaluations. It introduces a puzzle benchmark and compares six optimization algorithms across eight models.

1 Introduction

  • Static test sets risk leakage, frontier benchmarks can require heavy infrastructure, open-ended grading can be costly, and saturated benchmarks can become obsolete.Source: §1 · p. 1
  • Chess puzzles offer inexpensive exact-match and engine-based scoring, renewable problems with tunable difficulty, and a way to test transfer from puzzles to play.Fresh sets can be constructed from new games as models improve.Source: §1 · p. 2
  • The benchmark evaluates six APO algorithms across eight proprietary and open-weight models to study optimization behavior and prompt transfer.The analysis covers puzzle metadata, prompt composition, engine evaluation, and transfer across models and to game-play rollouts.Source: §1 · p. 2
  • The release includes 1,118 curated puzzles, optimization and evaluation code, and scripts to refresh the benchmark and adjust difficulty.Source: §1 · p. 3

2 Related Work

The paper situates its benchmark among renewable LLM evaluations, chess studies, and APO methods. It uses chess as a text-based reasoning task for frozen general-purpose models, with behavior evaluated on puzzles and through play.

Frontier LLMs Benchmarks

  • Renewable benchmarks address saturation and contamination concerns; this benchmark adds inexpensive exact-match and graded engine evaluation without an LLM judge or agentic environment.Related examples include LIVEBENCH [13], RULER [14], LLF-BENCH [15], and LOCOMO.Source: §2 · p. 3

Chess with LLMs

  • Earlier chess studies trained or probed chess-specific representations, whereas this work studies frozen general-purpose language models on text-based chess reasoning.Prior work also finds performance sensitive to board representation and prompt structure [7].Source: §2 · p. 3

Automatic Prompt Optimization (APO)

  • APO methods adapt model behavior without updating parameters, using techniques such as discrete search, gradient-free editing, and evolutionary algorithms [1, 2, 34–37].Chess complements existing APO evaluations by allowing optimized behavior to be assessed on isolated decisions and through gameplay consequences.Source: §2 · p. 3

3 Chess Puzzle Benchmark

The benchmark treats chess as a text-based reasoning task for general-purpose models and optimizes prompts while keeping model parameters frozen. It combines puzzle scoring with a renewable dataset whose difficulty can be adjusted.

Prompt Optimization with Chess

  • The benchmark optimizes prompt p for a frozen model f(x; θ, p), with θ fixed and prediction error measured against reference label y.Source: §3 · p. 4
  • Engine evaluation provides cheap, graded, reproducible scores on new positions and can supply labels for fresh puzzles as models improve.This addresses the need for scoring that generalizes to new inputs and offers more than binary feedback.Source: §3 · p. 4

Chess Puzzle Representation and Evaluation

  • A puzzle is solved only when the model matches the reference move at every position queried along its solution; exact-match error is the default loss.Appendix B.2 adds a graded engine-based evaluation.Source: §3 · p. 4
  • Inputs include FEN, the side to move, and explicit legal moves; responses must provide one legal move in SAN or UCI wrapped in JSON.Unparseable formatting is scored as incorrect; preliminary experiments found FEN effective for the evaluated models.Source: §3 · p. 4

Dataset Construction and Difficulty

  • The dataset contains 1,118 Lichess puzzles split equally into 559-puzzle training and test sets, stratified by puzzle length and Lichess rating.Because it uses public puzzles, pretraining exposure cannot be excluded; its stated advantage is renewability from newer games. Rating and length are descriptive, not independent difficulty factors or LLM playing ratings.Source: §3 · p. 4
  • Preliminary experiments found that recent models do not saturate the current puzzle database, supporting its use for APO evaluation.Source: §3 · p. 4

4 Experimental Setup

The experiments compare six prompt optimizers across eight target models, using a fixed meta-model for prompt proposals and reflection. Optimizers are run with standard configurations, and baseline results are reported alongside model characteristics.

4.1 Algorithms

  • Six optimizers span instruction-only, demonstration-only, and joint prompt optimization, using different feedback types and search strategies.The panel covers GEPA, COPRO, BFS, BRS, MIPROV2, and SIMBA.Source: §4.1 · p. 5
  • SIMBA jointly modifies instructions and examples using introspective textual feedback and numerical scores, while MIPROV2 searches instruction-demonstration combinations with Bayesian optimization.GEPA uses traces and textual and numerical feedback; COPRO uses numerical feedback to refine instructions; BFS bootstraps successful demonstrations, and BRS randomly searches demonstration sets.Source: §4.1 · p. 5
  • Optimizers use either scalar accuracy or textual feedback that classifies failures and can supply task-specific guidance; comparisons use standard recommended configurations, not equalized compute budgets or optimizer-specific tuning.Source: §4.1 · p. 6

4.2 Models and Baseline Evaluations

  • The evaluation spans providers including Google, DeepSeek, Anthropic, OpenAI, Alibaba Group, and Meta, with models selected for cost, provider diversity, and Game Arena rank.Jev 1.13 is a zero-shot classifier over legal moves rather than a free-text generator, so generative output-token columns do not apply.Source: §4.2 · p. 6
  • Gemini 3.5 Flash Lite achieves 26.30% accuracy at $0.169 per 1k calls, and Gemini 3.5 Flash serves as the fixed meta-model.The common baseline is deliberately minimal, intended to show what optimizers derive from the same starting point rather than compare against the strongest model-specific manual prompts.Source: §4.2 · p. 7

5 Results

Across 141 optimization runs and 24 baseline evaluations, the study compares six APO methods on eight models and examines puzzle gains, prompt composition, cross-model transfer, and game-play rollouts. Puzzle improvements vary by model and optimizer, do not transfer consistently, and do not significantly reduce mean regret in the evaluated rollouts.

5 Results

  • The study’s eight-model puzzle panel comprises 141 optimization runs and 24 baseline evaluations, at around $800 in recorded optimization and evaluation costs.Results are organized around model improvement, puzzle subsets, prompt content, cross-model transfer, and game-play rollouts.Source: §5 · p. 7

5.1 Optimization Gains and Evaluation Cost

  • 7.57 percentage points with SIMBA is the best improvement for Gemini 3.5 Flash Lite; four models show significant gains, while four show none.The best optimizer differs across models, and GPT-5.6 has no mean improvement over baseline.Source: §5.1 · p. 7, Table 3 · p. 8
  • GEPA has the highest median gain and improves mean accuracy for six of eight models, while no optimizer wins everywhere.SIMBA has the largest mean gain, driven mainly by Gemini 3.5 Flash Lite; optimization raises evaluation cost for every model except Spark 1.2 Contributor.Source: §5.1 · p. 8, Figure 2 · p. 9

5.2 Gains Across Puzzle Difficulty and Structure

  • Gains span Elo ranges and themes but largely stay below 2200; longer puzzles can improve per move without converting those gains into full solves.Mate-in-two and short themes capture the most aggregate improvement, but some rating bins and move indices are unchanged or worse; Qwen loses 0.8 points on 1400–1600 Elo puzzles.Source: Figure 3 · p. 9, §5.2 · pp. 9–10, Figure 4 · p. 10

5.3 What Do Optimized Prompts Contain?

  • SIMBA and GEPA produce the best prompts for seven of eight models, but the largest gain comes from a short prompt using only examples and chess guidance.Flash Lite’s prompt averages 930 characters; selected GEPA prompts average 2,109–3,626 characters, and no winning prompt uses all content types.Source: Figure 5 · p. 11, §5.3 · p. 11

5.4 Are Optimized Prompts Generic or Model Specific?

  • Transfer depends on the source–target pairing: 31 of 49 cross-model pairs improve, 16 decline, and two are unchanged.Jev 1.13’s prompt improves all seven other models; Flash Lite’s prompt, despite its largest native gain, degrades five. Jev raises DeepSeek V4 Pro from 14.49% to 17.59%.Source: §5.4 · pp. 11–12, Figure 6 · p. 12

5.5 Transfer from Puzzles to Play

  • Puzzle gains did not produce a statistically significant reduction in mean regret on accepted moves in rollouts against Stockfish.The evaluation compares each model’s base prompt with its best changed held-out puzzle prompt; regret is conditional on accepted moves and each prompt’s own visited states. Puzzle gains alone do not establish better play from standard openings.Source: §5.5 · p. 13, Table 4 · p. 13

6 Conclusion

Four models gained puzzle accuracy, with improvements reaching 7.57 percentage points, while the hardest puzzles remained largely out of reach. The study also finds that gains do not necessarily transfer to game play.

6 Conclusion

  • Four models show significant puzzle-accuracy gains, reaching 7.57 percentage points, while the hardest puzzles remain largely out of reach.The study distinguishes baseline strength, responsiveness to optimization, and transferability.Source: §6 · p. 14
  • Stronger models do not necessarily improve more, and larger puzzle gains do not guarantee better transfer to play.Some prompts help other models, but the evaluated game-play rollouts show no clear reduction in mean regret.Source: §6 · p. 14

A.1 Chess Notation Examples

  • Forsyth–Edwards Notation (FEN) represents a board state with the side to move, castling rights, en-passant target, and move counters.The example gives the position after 1. e4 e5 2. Nf3.Source: §A.1 · p. 19
  • Universal Chess Interface (UCI) records moves as source-destination squares, while Portable Game Notation (PGN) records them in standard chess notation.For 1. e4 e5 2. Nf3, UCI is e2e4 e7e5 g1f3 and PGN is 1. e4 e5 2. Nf3.Source: §A.1 · p. 19

A.2 Playing Chess Starting Positions

  • Table 5 lists starting positions for the playing-chess transfer evaluation, with model color indicating which side the language model controls.Source: Table 5 · p. 19

A.3 Puzzle Dataset Details

The dataset-details table organizes train/test puzzle counts by solver moves and Lichess rating bin.

A.3 Puzzle Dataset Details

  • Table 6 reports train/test puzzle counts by solver moves and Lichess rating bin.Each cell is presented as train/test.Source: Table 6 · p. 19

B Results Details

The examples show compiled puzzle prompts from MIPROV2 and SIMBA, including their instructions, demonstrations, and puzzle-specific inputs.

B.1 Example Compiled Puzzle Prompts (Gemini 3.5 Flash Lite)

  • The seed-42 MIPROV2 example pairs an instruction with two of four demonstrations, whereas the SIMBA example shows its complete static prompt.The MIPROV2 prompt is 1,507 characters and the SIMBA prompt is 901 characters; Table 8 values average across seeds 42–44.Source: §B.1 · p. 20
  • The MIPROV2 prompt uses green for output-format constraints, orange for chess guidance, blue for optimizer-selected demonstrations, and pink for other content.Source: §B.1 · p. 20
  • The prompt examples supply a FEN position, side to move, legal moves, and a selected move.The MIPROV2 instruction requests exactly one legal move and forbids reasoning, explanations, or labels in the response.Source: §B.1 · p. 20

B.2 Puzzle Move Quality by Regret

Median regret provides a more granular view of move quality than binary puzzle accuracy, but it is not validated as a general measure of playing strength. Prompt size results show that some optimization runs leave the base prompt unchanged.

B.2 Puzzle Move Quality by Regret

  • Regret compares the Stockfish evaluation of the best move with that of the model’s move, giving incorrect moves a graded rather than binary assessment.The analysis covers long puzzles of 3–5 moves and omits GPT-5.6 Luna because its selected prompt is unchanged from baseline.Source: §B.2 · p. 21
  • Median regret, position accuracy, and puzzle accuracy produce similar but not identical model rankings on long puzzles.Median regret limits the influence of harsh penalties, which can reach 200,000 centipawns for missing checkmate or predicting an illegal move; the text cautions that this agreement does not validate regret as a general measure of playing strength.Source: §B.2 · p. 21, Table 7 · p. 21

B.3 Prompt Size

  • 21 of 141 evaluated model–optimizer seed runs (14.9%) leave the prompt unchanged.The base prompt is 55 characters.Source: §B.3 · p. 21

B.4 Task Prompts and Signatures

The task prompt asks for one best move from a chess position, with structured inputs and a single-move output. Generative models use DSPy’s JSONAdapter to enforce structured formatting.

B.4 Task Prompts and Signatures

  • Generative-model results use DSPy’s JSONAdapter, which enforces strict structured output formatting.Source: §B.4 · p. 21
  • Table 8 summarizes average compiled-prompt size over three seeds, with a 55-character base prompt; gray cells indicate no improvement and N.E. means not evaluated.Source: Table 8 · p. 22
  • The puzzle task uses a 55-character default instruction: “Solve the chess puzzle by finding the single best move.”The instruction is distinct from the adapter-generated formatting instructions and puzzle input fields.Source: §B.4 · p. 22
  • Inputs provide the position in FEN, the side to move, and comma-separated legal UCI moves; the output is exactly one legal SAN or UCI move.Source: §B.4 · p. 22

B.5 Interpreting LICHESS Puzzle Ratings for LLMs

The study treats frozen LICHESS puzzle ratings as a descriptive difficulty axis and tests how well they track LLM solve rates. Solve rates are strongly correlated with ratings, while Figure 7 compares model performance with Elo reference curves.

B.5 Interpreting LICHESS Puzzle Ratings for LLMs

  • The benchmark freezes GLICKO-2-derived puzzle ratings and treats them as an external descriptive variable, ignoring rating uncertainty.GLICKO-2 models paired comparisons; LICHESS treats a puzzle like an opponent that wins when a user fails to solve it.Source: §B.5 · p. 23
  • Puzzle-rating correlations across pooled 200-point Elo bins are r = −0.937 (p < 0.001, 8 bins) and r = −0.996 (p = 0.004, 4 bins) for 1–2- and 3–5-move puzzles, respectively.Source: §B.5 · p. 23
  • Each 200-point Elo increase multiplies solve odds by 0.72 for 1–2-move puzzles and 0.45 for 3–5-move puzzles, corresponding to odds reductions of 28% and 55%.Source: §B.5 · pp. 23–24
  • Figure 7 plots solve rate by puzzle rating: solid curves show empirical performance, shaded regions show ±1 standard deviation, and dashed curves show Elo references anchored at 50% success for ratings 955 and 1980.Those anchor ratings approximately match the observed 50% solve-rate points of Gemini 3.5 Flash Lite and Gemini 3.5 Flash.Source: Figure 7 · p. 24, §B.5 · p. 24

B.6 Hyperparameters

The appendix identifies the models and reports configuration and optimizer-hyperparameter tables. The stated settings include different temperatures for task and meta-models, while Jev uses a decision endpoint without sampling controls.

B.6 Hyperparameters

  • Table 9 gives the exact OpenRouter model identifiers and notes that Gemini 3.5 Flash is the fixed reflection and meta-model.Source: Table 9 · p. 24
  • Task LMs use temperature=0.0, while the meta-model and COPRO prompt LM use temperature=1.0; selected LMs use a 30.0-second timeout.Source: §B.6 · p. 24
  • Jev 1.13 uses a Typesafe decision endpoint that selects among provided legal moves, with no temperature or sampling controls, so SIMBA is not evaluated for it.Source: §B.6 · p. 25
  • Table 11 lists optimizer hyperparameters, using DSPy defaults unless otherwise noted.Source: §B.6 · p. 25, Table 11 · p. 26
Loading 2610.00416v1…