PaperlayerView exact revision on arXiv

Source-linked AI summary

ChessGPT: Bridging Policy Learning and Language Modeling

Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, Jun Wang

arXiv:2306.09200v2cs.LGcs.AI

TL;DR

Research often treats historical policy data and natural-language analysis separately, despite humans drawing on both when making decisions. The paper builds chess datasets and presents ChessGPT and ChessCLIP with an evaluation framework; it reports that its models outperform other LLM baselines across evaluation tasks.

  • Problem

    Prior work often uses historical replay to learn policy or value functions, or trains language models on language corpora, while the interaction between policy learning and language modeling has received little attention.

    Source: §1 · p. 1

  • Method

    The paper builds a chess game-and-language dataset, presents ChessGPT and ChessCLIP, and evaluates modeling, value judgment, and policy abilities.

    Source: §1 · pp. 2–3

  • Results

    The authors report that ChessGPT and ChessCLIP consistently outperform other LLM baselines across all evaluation tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The dataset and benchmark are intended to make future research on policy and language alignment more accessible.

    Source: §6 · p. 10

    ChessGPT-Base shows limited move-score variation across Elo ratings, suggesting difficulty using Elo information and a risk of learning an average policy.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

When solving decision-making tasks, humans typically depend on information from two key sources: (1) Historical policy data, which provides interaction replay from the environment, and (2) Analytical insights in natural language form, exposing the invaluable thought process or strategic considerations. Despite this, the majority of preceding research focuses on only one source: they either use historical replay exclusively to directly learn policy or value functions, or engaged in language model training utilizing mere language corpus. In this paper, we argue that a powerful autonomous agent should cover both sources. Thus, we propose ChessGPT, a GPT model bridging policy learning and language modeling by integrating data from these two sources in Chess games. Specifically, we build a large-scale game and language dataset related to chess. Leveraging the dataset, we showcase two model examples ChessCLIP and ChessGPT, integrating policy learning and language modeling. Finally, we propose a full evaluation framework for evaluating language model's chess ability. Experimental results validate our model and dataset's effectiveness. We open source our code, model, and dataset at https://github.com/waterhorse1/ChessGPT.

1 Introduction

Research often treats historical policy replay and natural-language analysis separately, although human decision-making draws on both. The paper uses chess to build a dataset, models, and an evaluation framework for studying their interaction.

1 Introduction

  • Chess provides a testbed with abundant policy replay and chess-related language resources for studying their interaction.The authors argue that natural-language understanding and policy learning should not be studied in isolation.Source: §1 · p. 2
  • The dataset spans game replays, chess-language material, mixed game-language examples, and instruction or conversation data.Mixed examples align articles, discussion, or commentary with specific game replays.Source: §1 · pp. 2–3
  • ChessGPT and ChessCLIP use the datasets to explore learning from both replay data and language knowledge.Source: §1 · p. 3
  • The evaluation framework assesses modeling ability, value judgment, and policy ability; the authors report that their models outperform other LLM baselines across its tasks.Source: §1 · p. 3
  • The pipeline is intended to advance research on policy-language interplay, while the models may also have practical use as chess assistants.Source: §1 · p. 3

2 Related work

Related work covers chess policy learning and LLM-based decision-making, while identifying limited prior attention to training on the interplay between language modeling and policy learning.

2 Related work

  • Some chess-policy studies focus on small-scale datasets or limited training, while chess engines reach superhuman performance through human knowledge or self-play.Source: §2 · p. 3
  • LLM-agent work often relies on powerful models such as GPT-4, whereas this paper emphasizes training and the interplay of language modeling with policy learning.Source: §2 · p. 3
  • Prior LLM decision-making approaches use transformer representations for imitation learning or embedded common knowledge for policy learning; their dynamic interplay is scarcely addressed.Source: §2 · p. 3

3 A large-scale game and language dataset for chess

The paper assembles chess materials from online and other sources into four dataset categories, ranging from game replay and language to aligned game-language and conversation data.

3 A large-scale game and language dataset for chess

  • The dataset organizes Internet-collected chess material into game, language, mixed game-language, and instruction-tuning or conversation categories.Source: §3 · p. 3

3.1 Game dataset

  • Game records retain PGN headers and move comments, preserving player, Elo, opening, outcome, and possible engine-evaluation information for policy learning.The authors describe five months of Lichess data, 440,000 professional games, and three million CCRL engine games.Source: §3.1 · p. 4

3.2 Language dataset

  • The language collection combines chess-related corpus filtered from existing datasets with chess blogs, books, and forum discussions.Source: §3.2 · pp. 4–5

3.3 Mixed game-language dataset

  • The mixed dataset contains 245,000 annotated games and 1.3 million board-language pairs, linking game states with commentary and analysis.A separate video pipeline aligns chessboard screenshots and transcripts using OCR to produce FEN and board-language pairs.Source: §3.3 · p. 5

3.4 Instruction-tuning & conversation dataset

  • Instruction-tuning data includes GPT-4-generated chess instruction-response pairs and multi-turn chess conversations collected from Reddit.The sources describe around 4,000 instruction-response pairs and 410,000 human conversations.Source: §3.4 · p. 5

4 Large-scale pretraining

The paper presents ChessCLIP for aligning game and language data and ChessGPT, which treats chess moves as text for GPT-like policy learning. Their inputs include paired annotations, game materials, and instruction data.

4.1 ChessCLIP

  • ChessCLIP aligns chessboard or game-trajectory representations with paired annotation language and can retrieve matching games or comments.It can also select moves by searching legal actions for the one with the highest similarity to a text description.Source: §4.1 · pp. 5–6

4.2 ChessGPT

  • ChessGPT applies causal language modeling to chess materials because chess states and moves can be represented as text, covering an imitation-learning objective over game data.Source: §4.2 · p. 6
  • ChessGPT training has base-model fine-tuning on chess corpora followed by supervised fine-tuning on question or conversation responses; RLHF is left for future work.Source: §4.2 · p. 6

5 Evaluation and benchmark

The evaluation compares ChessGPT and ChessCLIP with baseline language models across chess-state modeling, value judgment, and policy tasks, alongside qualitative chess questions.

5.1 Chess modeling ability

  • On chess state-tracking tasks, ChessGPT-Base and Chat consistently outperformed baselines, while Chat performed slightly below Base.The authors suggest a potential trade-off between language capabilities and state tracking.Source: Table 1 · p. 7, §5.1 · p. 7
  • Across UCI-to-FEN and PGN-to-FEN conversion, ChessGPT averaged above 90% similarity, while both baselines stayed below 10%.The paper reports that whole-board state tracking is more challenging than tracking an individual piece.Source: Table 2 · p. 7, §5.1 · pp. 7–8

5.2 Value judgement ability

  • Across all four value-judgment tasks and settings, ChessCLIP, ChessGPT-Base, and ChessGPT-Chat surpassed both baseline language models.The authors report alignment with both the true value function and human judgment or knowledge.Source: §5.2 · p. 8

5.3 Policy evaluation

  • For checkmate-in-one, ChessGPT-Base and Chat surpassed the two LLM baselines by a large margin; ChessCLIP performed poorly, which the authors attribute to sparse checkmate annotations.An instruction suffix can substantially enhance the base model's performance.Source: §5.3 · p. 9, Table 6 · p. 9
  • In general policy evaluation, all models exceeded the random policy's approximately 50% performance at the tested high human-skill Elo ratings.The task evaluates selecting a next move within a specified skill level.Source: Table 7 · p. 9, §5.3 · p. 9
  • Although the dataset spans Elo ratings, the current training method may not sufficiently emphasize rating-specific features, and ChessGPT-Base may struggle to use Elo information.The authors identify better integration of Elo and other auxiliary metadata as a direction for improving generalization.Source: Table 8 · p. 9, §5.3 · p. 9

5.4 Qualitative results

  • Qualitatively, ChessGPT-Chat gave more factual answers and better chess-task analysis than ChessGPT-Base and the baselines, which struggled with commentary and PGN-to-FEN conversion.ChessGPT-Base performed similarly to RedPajama in these comparisons.Source: §5.4 · p. 10

6 Conclusion

The conclusion presents ChessGPT’s chess dataset, benchmark, evaluation framework, and models as an initial step toward studying policy-language interplay. The appendix explains common formats for moves, games, and board positions.

6 Conclusion

  • The paper introduces a chess dataset and benchmark, an evaluation framework for language models’ chess ability, and ChessCLIP and ChessGPT as promising models for learning language-action interplay.The authors describe the work as an early step toward understanding how to bridge policy learning and language modeling.Source: §6 · p. 10

B.1 Universal Chess Interface (UCI)

  • UCI represents an individual move by combining its starting and ending squares, as in “e2e4.”The appendix also provides a UCI sequence representing a full game.Source: §B.1 · p. 17

B.2 Standard Algebraic Notation (SAN)

  • SAN represents chess moves with piece abbreviations and destination squares, with additional symbols for checks, checkmates, and castling.Pawns have no piece abbreviation; “+” marks check and “#” marks checkmate.Source: §B.2 · p. 17

B.3 Portable Game Notation (PGN)

  • PGN records whole games using SAN moves alongside metadata such as player names, event details, and results.The appendix characterizes PGN files as human-readable and shareable.Source: §B.3 · p. 17

B.4 Forsyth–Edwards Notation (FEN)

  • FEN describes a chess position, including piece locations, active color, castling rights, en passant target, and move counters.The appendix distinguishes FEN as a representation of a specific game state.Source: §B.4 · pp. 17–18

D.1 Dataset statistics and metrics

The appendix reports dataset statistics by subset and summarizes the metrics used to characterize the chess-specific language dataset.

D.1 Dataset statistics and metrics

  • Dataset statistics are broken down by subset, including raw size, document count, and subset type.Table 9 presents the dataset statistics breakdown.Source: §D.1 · p. 18, Table 9 · p. 19
  • The language-dataset properties include average words per example, character and word repetition, special-character and stopword ratios, and first-shard perplexity.These metrics are reported in Table 10.Source: §D.1 · p. 18, Table 10 · p. 19
  • The language-dataset metrics draw on criteria employed by Bloom in preprocessing pipelines.The passage identifies Bloom as the source of some of the criteria.Source: §D.1 · p. 18

D.2 Dataset examples

The appendix illustrates the dataset’s varied chess content and describes collection from chess websites, forums, and annotated-PGN sources.

D.2 Dataset examples

  • The dataset examples span game records, chess instruction and discussion, puzzles, commentary, and question-answer material.Examples include annotated games, a chess puzzle, forum questions, instructional text, and a transcript excerpt.Source: §D.2 · pp. 19, 20, 21, 22, 24

E.1 Online Chess websites

  • The authors scraped chess-related language from around 26 websites, including blogs, news, openings, chess terms, forums, and articles.They used BeautifulSoup4 and Playwright to locate text and PGNs, converting PGNs to text for mixed game-language data.Source: §E.1 · p. 26

E.2 Online Chess forums

  • The forum corpus was collected from five chess forums using an approach similar to the website collection process.Source: §E.2 · p. 26

E.3 Annotated PGN

  • Annotated PGNs came from seven sources, including Lichess studies, Pgnlib, and Gameknot; commercial Chesspublishing and Megabase data were not opened.The passage also names Chess publishing and Megabase as commercial annotated-PGN sources.Source: §E.3 · pp. 26–27

E.4 Existing Datasets

The section describes chess-data collection and preprocessing pipelines, including a video-to-FEN workflow and synthetic modeling tasks. It also outlines ChessCLIP’s paired board/text training and move-generation capability.

E.5 YouTube transcripts dataset

  • Chess videos are filtered for chess relevance, then detected boards are converted to FEN through decomposition, tile classification, and next-turn prediction.The pipeline divides boards into 64 tiles and classifies each as one of 13 piece-or-empty categories.Source: §E.5 · p. 27, §E.5.1 · p. 27, §E.5.2 · pp. 27–28

E.6 Lichess dataset

  • The collected sources include five monthly Lichess datasets, pro-player games, chess books, CCRL games, and Lichess puzzles.Because of copyright issues, the authors do not open-source chess-book data, but do release the list of books used.Source: §E.6 · p. 28, §E.7 · p. 28, §E.8 · p. 29

E.11 Chess modeling data

  • The authors generate 11 chess-modeling tasks from sampled Lichess games, with approximately 10,000 games from each of nine Elo intervals.Tasks include legal-move generation and PGN-to-FEN conversion.Source: §E.11 · p. 29

E.12 Preprocessing

  • Preprocessing filters existing datasets, parses chess-related HTML, and uses lighter processing for notation-heavy examples than for natural-language text.The authors also anonymize names in conversation-like data and use deterministic shuffling for reproducibility.Source: §E.12 · pp. 29–30

F.1.1 ChessCLIP

  • ChessCLIP training uses annotated-PGN board/text pairs, while its board/action encoder represents positions and moves for alignment with text.The authors report that YouTube transcripts contain more noise and therefore use annotated PGNs exclusively for stable training.Source: §F.1.1 · p. 30
  • ChessGPT’s ChessCLIP can generate move sequences from text prompts by beam-searching legal sequences to maximize similarity.The authors attribute this capability to the relatively low-dimensional space of legal chess moves.Source: §F.1.1 · p. 31

F.1.2 ChessGPT

ChessGPT-Base is fine-tuned on chess corpora, and ChessGPT-Chat is further supervised on chess and general conversations. In black-piece general-policy evaluation, ChessGPT improves over RedPajama and performs on par with LLaMA, while showing limited sensitivity to Elo prompts.

F.1.2 ChessGPT

  • ChessGPT-Base fine-tunes RedPajama-3B on chess corpora with a maximum token length of 1024; supervised conversation fine-tuning produces ChessGPT-Chat.The chat model uses chess instruction/conversation data together with general conversation datasets.Source: §F.1.2 · p. 31
  • ChessGPT improves over RedPajama and performs on par with LLaMA in black-piece general-policy evaluation, but both baselines show limited Elo adaptation.The comparison between ChessGPT models across Elo ratings also shows no noticeable distinction.Source: Table 16 · p. 32, §F.1.2 · p. 32
  • ChessGPT-Base’s attention to WhiteElo and BlackElo is relatively weak, indicating limited incorporation of those prompt values during generation.Figure 3 visualizes attention intensity from black to red, with red marking the highest importance.Source: §F.1.2 · p. 32, Figure 3 · p. 32

F.1.3 Compute resources

  • Experiments use eight A100 80G GPUs; ChessGPT-Base training takes 60 hours and ChessGPT-Chat training takes 18 hours.ChessCLIP training takes five hours on the same GPUs.Source: §F.1.3 · p. 32

G.1 Evaluation task examples and plots

The evaluation examples present task prompts, inputs, and targets for chess-state, notation, and move-prediction tasks. Evaluation data includes high-rated Lichess games and separately constructed states for value and policy tasks; the section also introduces qualitative comparisons among three models.

G.1 Evaluation task examples and plots

  • Evaluation examples separate each task into a task prefix, input, and target, with targets given as exact answers or scores for multiple-choice options.Examples include state tracking, legal-move prediction, FEN generation, and checkmate-in-one prediction.Source: §G.1 · pp. 32–33, Figure 4 · p. 33, Figure 5 · p. 34

G.1.1 Evaluation on chess modeling ability

  • Chess-modeling evaluation uses the first 400,000 classical and rapid games from the Lichess December 2019 dump, requiring both players to have over 2000 Elo.The authors avoid using the two datasets previously mentioned when selecting these games.Source: §G.1.1 · p. 35

G.1.2 Evaluation on task construction

  • State-value and general-policy tasks use 3,000 and 2,000 Lichess 2017-03 game states, respectively; Stockfish-15 at depth 18 scores legal moves for policy evaluation.Legal-move ranks by winning rate determine the choice scores.Source: §G.1.2 · p. 35

H Qualitative Analysis

H.1 Experimental set-up

The appendix reports qualitative chess-language tests, illustrating strengths and shortcomings across model responses, alongside a ChessCLIP visualization and possible research directions.

H.1 Experimental set-up

  • The qualitative experiments sample three responses per question and use model-specific prompt templates; Table 17 lists the experiment hyperparameters.The prompts follow each model’s recommended usage.Source: §H.1 · p. 36, Table 17 · p. 36

H.2 General knowledge

  • Base models answer general chess questions reasonably because the answers are readily available online, although RedPajama sometimes gives less factual answers.The paper points to examples in Tables 18, 19, and 21.Source: §H.2 · p. 37

H.3 Chess modeling

PGN annotation

  • ChessGPT-Base gives an off-task or incomplete response to step-by-step PGN analysis, ChessGPT-Chat begins reasonable analysis but leaves two replies unfinished, and RedPajama mostly repeats the PGN.RedPajama’s remaining response offers an irrelevant follow-up move.Source: PGN annotation · p. 45

I ChessCLIP visualization

  • The ten-opening ChessCLIP similarity matrix has its highest similarities on the diagonal, matching each PGN with its corresponding opening name.The authors report that the visualization matches their expectation.Source: Figure 6 · p. 46, I ChessCLIP visualization · p. 46

J Potential directions

  • Potential directions include dataset augmentation, transfer to chess variants, language-model-based reinforcement learning or planning, and explainable-AI research.The paper also identifies multimodal analysis, chess education, adversarial robustness, and game generation as directions.Source: J Potential directions · pp. 46–47
  • Other proposed directions include multimodal chess analysis, educational tools, adversarial-attack research, and chess-game generation.The paper frames these as possible uses of the dataset and benchmark.Source: J Potential directions · p. 47

K Limitations and Potential Societal Impact

The paper notes that its chess dataset may not generalize across historical periods or cover less-common chess variants, while identifying several possible societal applications.

Limitations

  • Historical Lichess matches from different periods may vary in strategies, openings, and trends, potentially limiting dataset generalizability; the dataset predominantly covers standard chess.Researchers studying niche or less-popular variants may need additional data from specific sources.Source: K Limitations and Potential Societal… · p. 47

Potential Societal Impact

  • The dataset may support more advanced chess-playing agents for analysis, player training, and use as opponents for enthusiasts.The paper presents these as potential applications.Source: K Limitations and Potential Societal… · p. 47
  • The dataset may contribute to chess education through tutorials and interactive learning, and may encourage explainable-AI research that helps users understand model moves.The paper also notes possible AI research applications beyond chess.Source: K Limitations and Potential Societal… · p. 47
Loading 2306.09200v2…