PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognizing Atari and Assessing Sente: Controlled Evaluation Tasks for Language-Based Go Agents

Taiyi Chen, Iskar Deng, Amandalynne Paullada

arXiv:2610.22953v1cs.CL

TL;DR

Whether game-playing language models can accurately connect domain terminology to board relationships and move values remains unclear. This paper constructs controlled Go tests of atari and sente and evaluates a fine-tuned LoGos model. The model struggles with local terminology changes and atari’s strategic value, while atari predictions are associated with move preferences that can contradict reference evaluations.

  • Problem

    It remains unclear whether game-playing language models can accurately relate domain terminology to board positions and candidate-move evaluation as local configurations and strategic conditions change.

    Source: §1 · p. 1

  • Method

    The authors construct controlled tests of atari and its relationship to sente, then evaluate a fine-tuned LoGos model on terminology recognition and move evaluation.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    LoGos does not reliably track local changes in atari relationships or assess atari’s strategic value, and atari predictions are associated with favoring moves even when that preference contradicts reference evaluations.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The controlled tests provide a basis for evaluating whether approaches combining language models with explicit board computations and search improve local concept recognition and strategic value assessment.

    Source: §5 · pp. 4–5

    Only one fine-tuned LoGos model was evaluated, so the specific effect of terminology supervision cannot be determined, and the association between terminology predictions and move preferences was not tested for causality.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Domain terminology provides concise descriptions of game states and strategic relationships, but whether game-playing language models can accurately recognize these concepts and relate them to move evaluation remains unclear. We construct a controlled evaluation set centered on atari and its relationship to sente in Go, examining terminology recognition under local board changes and move evaluation under different strategic conditions. We conduct our tests on LoGos, a Go language model, after fine-tuning it with terminology and commentary supervision. The model struggles to recognize changes in the atari relationship and assess its strategic implications. Its terminology predictions are associated with move preferences even when those preferences contradict the reference evaluation. Our controlled tests provide a framework for examining whether language-based Go agents connect domain terminology with precise board relationships and strategic move values.

1 Introduction

The introduction motivates testing whether language-based Go agents can connect domain terminology to precise board relationships and move values. It presents controlled tests of atari and sente and reports recognition and evaluation difficulties in fine-tuned LoGos.

1 Introduction

  • Domain terminology concisely names recurring board patterns and relationships, but decoding concepts from model representations does not establish that agents apply them appropriately.Prior work found that a Hex agent encoded strategically irrelevant locations yet played moves there.Source: §1 · p. 1
  • The paper uses Go to test whether language models relate terminology to board positions and candidate-move evaluation as local configurations and strategic conditions change.The motivation includes prior reports of appropriate moves paired with inaccurate explanations and misused terminology.Source: §1 · p. 1
  • The tests center on atari and its relationship to sente, examining whether an atari threatens capture and forces a local response that preserves initiative.The evaluation uses a fine-tuned LoGos model developed to recommend moves and generate natural-language analyses.Source: §1 · p. 1
  • LoGos does not reliably adjust terminology judgments to local board changes, and it struggles to assess atari’s strategic value in specific positions.Atari-label predictions are associated with favoring a move even when that preference contradicts the reference evaluation.Source: §1 · pp. 1–2
  • The authors release the evaluation set and construction code to support research on terminology recognition and move evaluation in game-playing language models.The release is described as supporting further research on these capabilities.Source: §1 · p. 2

2 Experimental Design

The experimental design defines atari and sente through local threats and responses, then tests recognition and move evaluation across controlled Go positions. It builds paired and independent cases and selects them using Go records and KataGo estimates.

2.1 Go Concepts

  • An atari leaves an opposing group one liberty from capture, while sente generally describes a move requiring a local response because ignoring it permits a substantial loss.Whether the threat demands a response depends on threatened-stone value and available alternatives; a small threat may be sacrificed for play elsewhere.Source: §2.1 · p. 2
  • Position Z illustrates an atari without sente: White benefits more from sacrificing the threatened stones and playing elsewhere, making B substantially better than A.The example reports an approximately 1% estimated win rate after A versus 50% after B.Source: §2.1 · p. 2, §2.2 · p. 3
  • The X/Y examples isolate a one-stone relocation: A creates a sente atari in X, but not in Y, where A is substantially worse than B.In X, responding lets Black complete a sente exchange before playing B; in Y, A reduces Black’s estimated win rate to approximately 3%, while B remains roughly even.Source: §2.1 · p. 2, §2.2 · p. 3

2.2 Evaluation Tasks

  • The Atari–Sente Contrast Test compares paired X/Y positions with a shared inferior control move, while the Non-Sente Atari Test uses independent Z positions.The reference relationships are V(Y, B) > max{V(Y, A), V(Y, C)} and V(Z, B) > V(Z, A).Source: §2.2 · p. 3
  • Rule-based atari labels and KataGo score-lead and win-rate estimates select 1,000 X/Y pairs and 500 Z positions from public Go game records.Table 1 reports mean reference win rates for each candidate move across evaluation positions.Source: §2.2 · p. 3, Table 1 · p. 3

3 Model and Fine-Tuning

The model section describes LoGos-7B and its adaptation for terminology recognition and candidate-move win-rate prediction. The authors jointly fine-tune it with terminology, win-rate, and commentary supervision.

3 Model and Fine-Tuning

  • LoGos-7B is built on Qwen2.5-7B and combines supervised training on Go expert knowledge with reinforcement learning for move recommendations and natural-language analyses.The model’s base and capabilities are attributed to LoGos and Qwen Team (2024).Source: §3 · p. 3
  • The authors augment Go commentary with rule-based terminology labels and KataGo win-rate estimates, then jointly fine-tune LoGos on terminology, win rates, and human commentary using QLoRA.When multiple terms apply, the training labels prioritize atari.Source: §3 · p. 3
  • Data processing, annotation, input/output formats, and training hyperparameters are provided in Appendix B.Source: §3 · p. 3

4 Results and Analysis

The evaluation tests terminology recognition and move preferences across controlled atari–sente conditions. LoGos recognizes some atari moves but often misses local changes and struggles to assess strategic value; terminology predictions are associated with favoring moves even when that conflicts with reference evaluations.

4.1 Terminology Recognition

  • Only 14.6% of pairs received the correct atari-in-X/non-atari-in-Y labels, while A was most often labeled atari in both positions (58.1%).B was correctly identified as non-atari in both X and Y in 91.8% of pairs; recognition accuracy on B in Z was 87.0%.Source: §4.1 · p. 3, Table 2 · p. 4
  • Atari recognition accuracy was lower for larger groups in the pooled X and Z positions.The analysis pooled positions with a single threatened group; the passage does not provide the group-size-specific accuracy values.Source: §4.1 · pp. 3–4, Table 3 · p. 4

4.2 Move Value

  • When predictions showed a clear preference, the model generally ranked the clearly inferior move C below better alternatives.Predictions on opposite sides of 50% defined coverage; conditional accuracy measured how often those cases followed the reference ordering.Source: §4.2 · p. 4, Table 4 · p. 4
  • A’s conditional accuracy across X/Y comparisons was 88.8%, but the model often incorrectly favored A in Y and favored B in fewer than half of Z comparisons.Z positions contain atari without a sente effect.Source: §4.2 · p. 4
  • Labeling A as atari was associated with greater preference for A across X, Y, and Z, including positions where A lacked atari or the atari lacked sente.The comparison groups move preferences by predicted terminology.Source: §4.2 · p. 4, Table 5 · p. 4
  • Move-ranking results exclude comparisons where both predictions were near 0% or both were near 100%, and these cases were not counted as incorrect.The authors note that such extreme predictions may obscure differences between moves.Source: §4.2 · p. 4

5 Conclusion

The paper introduces controlled atari–sente tests and evaluates a fine-tuned LoGos model, finding unreliable responses to local board changes and difficulty assessing atari’s strategic value. It proposes explicit board computations and search as future directions, with the tests serving as a basis for evaluating whether those approaches improve recognition and value assessment.

5 Conclusion

  • The controlled tests examine whether language models recognize atari terminology and relate it to move evaluation in Go.The evaluation of fine-tuned LoGos finds unreliable responses to local board changes and difficulty judging whether atari makes a move strategically valuable.Source: §5 · p. 4
  • Small local modifications can change tactical situations, requiring precise tracking of board relationships; language-model inductive bias toward low-sensitivity functions may pose challenges for precise local judgments.The cited evidence for this bias includes Bhattamishra et al. (2023) and Hahn and Rofin (2024).Source: §5 · p. 4
  • Future work could combine language models with explicit board computations and search, and use these tests to assess effects on local concept recognition and strategic value assessment.The proposed computations and search could provide accurate local facts and subsequent move sequences.Source: §5 · p. 5

Limitations

The evaluation uses win-rate predictions, but their sensitivity and concentration near extremes constrain how well they assess moves. The study also tests only one fine-tuned LoGos model, so it cannot isolate the effects of terminology supervision or establish causality between terminology predictions and move preferences.

Limitations

  • Win-rate sensitivity and predictions concentrated near extremes limit the move-evaluation approach.The authors suggest score differences, local board relationships, and subsequent responses as future training signals and evaluation criteria.Source: Limitations · p. 5
  • Testing only one LoGos model leaves the effects of its base model, fine-tuning data, and training configuration unresolved.The study does not compare training conditions, so it cannot determine the specific effect of terminology supervision.Source: Limitations · p. 5
  • The observed association between terminology predictions and move preferences was not tested for causality.The authors call for further investigation of the sources of observed errors.Source: Limitations · p. 5

A Evaluation Position Construction

Evaluation positions are constructed from SGF game records by screening structural candidates and verifying move-value relationships with score leads before applying win-rate constraints to select the final positions.

A Evaluation Position Construction

  • SGF records supply the candidate positions, which are screened using score-lead differences before win-rate constraints select the final set.Score leads are used because win rates can saturate in unbalanced positions.Source: A Evaluation Position Construction · p. 6

A.1 KataGo Configuration

The appendix specifies KataGo settings and criteria for constructing two evaluation tests: one contrasting atari with its absence, and one testing atari without a sente effect.

A.1 KataGo Configuration

  • KataGo evaluations use Chinese rules, 7.5 komi, a 19 × 19 board, and 1,000 visits per final candidate query.Each candidate move is evaluated separately by restricting the root move; candidate moves are restricted to the first 150 moves.Source: §A.1 · p. 6

A.2 Atari–Sente Contrast Test

  • The Atari–Sente Contrast Test pairs positions where move A creates atari with a one-stone-location change that removes it, while constraining score leads and win rates.In the changed position, B must exceed A by at least 20 percentage points in win rate; a shared move C is selected to be clearly inferior in both positions.Source: §A.2 · p. 6

A.3 Non-Sente Atari Test

  • The Non-Sente Atari Test selects positions where A puts a group of at least two stones in atari, while the preferred response elsewhere leaves it capturable next turn.The saving move must leave the group with at least two liberties, and the opponent’s preferred response must be at least four points better than the local saving move.Source: §A.3 · p. 6

B.1 Data Annotation

The training data combines Go game histories, board states, candidate moves, and human commentary with rule-based terminology labels and KataGo win-rate labels. The resulting 9,398 examples cover five common terms, while their win-rate labels are often near the extremes.

B.1 Data Annotation

  • Training examples are drawn from Tomlin et al. (2022) Go Teaching Ladder records, pairing move histories and board states with candidate moves and human commentary.Each selected move becomes a separate example, with its comment as the output commentary.Source: §B.1 · p. 6
  • Board states are represented as 19 by 19 matrices, with 1 for Black stones, -1 for White stones, and 0 for empty positions.Move histories use numbered moves with player color and board coordinates.Source: §B.1 · p. 6
  • A rule-based matcher labels candidate moves by matching local board patterns across relative positions, rotations, and inverted stone colors.Atari is checked first so it takes precedence when multiple terms match.Source: §B.1 · pp. 6–7
  • The 9,398 training examples cover Stretch, Hane, Tiger’s Mouth, Atari, and Connect, with each term comprising about 17–23% of examples.Win-rate labels are concentrated near extremes: 45.22% are below 1% or above 99%, and 17.53% fall between 25% and 75%.Source: §B.1 · p. 7

B.2 Fine-Tuning Configuration

The authors fine-tune LoGos-7B for Go commentary using QLoRA and supervised examples pairing game-state prompts with terminology and commentary outputs.

B.2 Fine-Tuning Configuration

  • LoGos-7B is fine-tuned with QLoRA using 4-bit quantization, rank-16 LoRA, scaling 32, and dropout 0.05.Training uses a 5 × 10−5 learning rate, batch size 1 with 8 gradient-accumulation steps, and a 3,072-token maximum sequence length.Source: §B.2 · p. 7
  • Training examples ask the model to identify a Go term and explain a candidate move, with a sample output containing Term, Winrate, and Comment fields.The example includes move history and an abbreviated board state.Source: §B.2 · p. 7, Figure 2 · p. 7
  • Models are trained for two supervised fine-tuning epochs on an NVIDIA RTX 4090 with 24 GB memory, taking approximately four hours per run.Source: §B.2 · p. 7

C Evaluation Pipeline

The evaluation pipeline generates independent predictions for candidate moves, then extracts terminology and win-rate values for analysis.

C Evaluation Pipeline

  • Candidate moves are evaluated independently using the training input format, greedy decoding, and a maximum of 128 new tokens.Source: C Evaluation Pipeline · p. 7
  • For terminology evaluation, predictions other than atari are grouped as non-atari without judging whether those alternative terms are correct.Source: C Evaluation Pipeline · p. 7
  • Win-rate predictions are read from the candidate mover’s perspective, and extracted predictions and reference values are saved in JSON for analysis.Source: C Evaluation Pipeline · p. 7
Loading 2610.22953v1…