ChessQueries: Toward Better Chess Board Recognition

Joël Seytre

Source-linked AI summary

ChessQueries: Toward Better Chess Board Recognition

Joël Seytre

arXiv:2608.30762v1Aug 31, 2026cs.CV

TL;DR

Chess board recognition must recover per-square piece states across synthetic, smartphone, and broadcast imagery, where prior benchmarks and methods provide limited cross-domain evidence. ChessQueries combines a ViT encoder with a DETR-style decoder over learned square queries to predict the board directly. It achieves near-perfect performance on the established benchmarks, generalizes to unseen domains, and introduces a challenging broadcast dataset, while SLCC may be unusually difficult and not fully representative of deployment settings.

Problem
Existing benchmarks are limited to synthetic renders or smartphone photographs of a single physical board, motivating recognition across datasets and real broadcast conditions.
Method
ChessQueries pairs a ViT encoder with a DETR-style decoder that processes 64 learned square queries for direct per-square board prediction.
Results
ChessQueries saturates ChessReD and ChessCog, reaching 99.5% and 98.5% exact-board accuracy respectively, and transfers zero-shot to unseen CVChess imagery.
Takeaways & Limitations
The single architecture supports recognition across synthetic and real-world photographs and introduces SLCC as a difficult broadcast benchmark.
Takeaways & Limitations
SLCC contains 2,174 images and may be too difficult or unrepresentative of real-world use because of heavy occlusions and challenging viewpoints.

Abstract

from arXiv · show

Chess board recognition is the task of mapping the image of a chess board to the information of which piece is on which square. So far this task has two established benchmarks: ChessCog is synthetic, and ChessReD comes from smartphone pictures of a single chess board setup. We introduce ChessQueries, a new method combining a ViT encoder with a DETR-style decoder, which outperforms existing methods. On the ChessReD benchmark, we improve the state of the art from 15.3% to 99.2%, and demonstrate strong capabilities on out-of-distribution datasets. Our method saturates the task on the two datasets, with an average 0.01 wrong squares per board (vs. SotA: 3.4 / 0.15 respectively). We also share a new, harder public dataset, parsed from broadcasted top-level chess tournaments. Code, model weights and the SLCC data will be released.

1. Introduction

Chess board recognition maps board images to per-square states, but existing benchmarks cover limited domains. ChessQueries addresses this gap with a learned-query architecture intended to generalize across datasets and broadcast conditions.

Chess board recognition maps an image to each square’s state, supporting FEN output, chess analysis, broadcasting, and computer-vision research.The task must handle varying camera angles, lighting, shadows, and partial occlusions.
Existing benchmarks are either synthetic ChessCog renders or smartphone photographs of a single physical board in ChessReD.Prior methods generally focused on one dataset rather than a shared architecture across domains.
ChessQueries pairs a ViT encoder with a DETR-style decoder that processes learned square queries.
64 learned square queries correspond to the board’s 64 squares and support full-board prediction in one forward pass.Each decoded query is mapped to one of 13 square classes by a shared linear head.
The paper reports that ChessQueries outperforms existing methods, generalizes to unseen datasets, and introduces a difficult broadcast-derived SLCC dataset.SLCC contains 2,174 images with challenging lighting, partial occlusions, and viewpoints.

2. Related Work

Prior chess-recognition systems either decompose the task into stages or predict the board end to end, but both established approaches show weak cross-domain transfer. ChessQueries is presented as a single architecture spanning synthetic and real-world domains.

Multi-stage pipelines: Multi-stage systems detect the board, localize squares, and classify pieces, with ChessCog using projective fitting plus separate occupancy and piece CNNs.ChessCog uses a synthetic Blender-rendered dataset of 4,888 images and requires the player’s viewing perspective.
End-to-end recognition: ChessReD removed intermediate stages by predicting the full board directly from images using a ResNeXt-101 classifier and 64 square-specific prediction heads.Its benchmark contains 10,800 smartphone photographs from 100 games and three cameras.
Cross-domain generalization: 2% board accuracy was reported for ChessCog adapted to ChessReD, while ChessReD’s model reached 40% when trained on ChessCog.The corresponding in-domain results were 15% and 94%, respectively.
Cross-domain generalization: ChessQueries targets synthetic renders and real-world photographs, including ChessReD and the 352-image CVChess test set.

3. Method

ChessQueries treats recognition as fixed 64-square classification, using learned square queries to extract square-specific information from ViT image features. The decoder produces the complete board in one pass without intermediate board geometry outputs.

Architecture: The model labels each of 64 squares with one of 13 classes representing white pieces, black pieces, or an empty square.A ViT-L/14 encoder with 304M parameters maps a 644×644 image to patch tokens.
Architecture: Sixty-four learned square queries cross-attend to image tokens through a 4-layer DETR-style decoder.Each query encodes a square ID, rank, file, and color.
Architecture: A shared linear head maps each decoded query to its square class, with fixed targets eliminating the need for Hungarian matching.
Interpretability: Square-query attention localizes to the corresponding physical square across domains, viewpoints, lighting, and board styles.Under heavy occlusion, attention can focus on the visible portion of a piece between neighboring pieces.
Inference: The full board is produced in one forward pass without required intermediate board detection or corner estimation.
Inference: The approach accepts images from player perspectives or side views, rather than requiring a single orientation.

4. Experiments

Experiments evaluate ChessQueries across established, out-of-domain, and broadcast datasets, with carefully described data construction and training protocols. The model achieves near-saturated performance on ChessReD and ChessCog while addressing cross-domain generalization.

Datasets: The evaluation uses ChessReD, ChessCog, CVChess, and the newly created SLCC broadcast dataset.SLCC contains 2,174 images from 20 Saint Louis Chess Club and Grand Chess Tour broadcast videos.
Datasets: SLCC positions were verified through Lichess relay matching, model-assisted ranking, and human review.Samples with impossible full occlusion were excluded, while retained images remain challenging because of obstruction, viewpoint, lighting, and low resolution.
Datasets: The SLCC release distributes video identifiers, timestamps, crop coordinates, extraction tooling, and position labels, but not the frames themselves.The release is limited to noncommercial research use under CC BY-NC 4.0.
Training: Training uses per-square 13-way softmax cross-entropy with AdamW, cosine scheduling, warmup, geometric and color augmentation, and validation exact-board checkpoint selection.Reported model results in Table 1 average three independent seeds.
Results: 99.5% / 98.5% perfect board prediction on ChessReD & ChessCog accompanies 0.01 average wrong square per board under joint training.This corresponds to approximately 1 wrong square predicted every ∼6400.
Results: 99.2% exact board accuracy on ChessReD exceeds the published 15.3% baseline under matching training-data conditions.The model also reaches 98.2% on ChessCog versus 93.9% for its corresponding published method.
Results: 40.3% →99.5% on ChessReD against the generalizing ResNeXt recipe isolates a substantial architecture-level difference.The reproduced baseline used the same joint training data and a recipe adapted for generalization.
Cross-domain evaluation: No model trained on ChessReD performs well on ChessCog, or vice versa, underscoring the stark domain difference between the datasets.This contrasts with ChessQueries’ reported performance across the established benchmarks.

5. Analysis

The analysis shows that square queries and a DETR-style decoder improve out-of-domain generalization and localization, while encoder scale and geometric augmentation matter most on difficult domains. Few-shot LoRA adaptation reaches useful SLCC performance with limited parameter updates, but heavy occlusion remains a failure mode.

Ablations: 87.6% to 51.4% zero-shot CVChess accuracy follows from replacing ViT-L with ViT-B, identifying encoder scale as the largest zero-shot factor.Removing geometric augmentation also reduces CVChess accuracy to 41.2% and increases errors to 6.6 wrong squares.
Ablations: ChessReD and ChessCog remain near-saturated across ablations, with less than 3 percentage points of regression, whereas changes mainly affect SLCC and CVChess.The table caption attributes the largest augmentation impact to CVChess’s extreme view angles.
Query decoder vs. linear head: 77.1% zero-shot board accuracy on CVChess versus 38.6% for the linear head shows the query decoder’s advantage under domain shift.On held-out SLCC, the decoder also makes 16.2 wrong squares per board versus 25.3 for the linear head.
Query decoder vs. linear head: 92–94% ChessReD / ChessCog accuracy and 53% SLCC accuracy remain attainable with the encoder frozen, while the linear head stays at 0%.The comparison suggests that the query decoder can compute visual square correspondence without encoder fine-tuning, unlike the linear head.
Failure cases: Heavy occlusion concentrates the model’s worst SLCC errors, with up to seven mistaken squares out of 64, and remains difficult for humans.This is the clearest reported boundary on the hard broadcast domain.
Few-shot adaptation: 69% SLCC board accuracy is reached with the full few-shot training set, below the 87% achieved by full joint training.Ten images raise accuracy from 0% to 24%, while fifty reach 38%.
Few-shot adaptation: 3.1M trainable LoRA parameters match full fine-tuning within seed noise across shot counts, compared with 371.7M for the full model.The resulting fp32 LoRA checkpoint is 12 MB; decoder-only fine-tuning underperforms and catastrophically forgets the source domain.

6. Limitations

The paper identifies limitations in the SLCC dataset, side-agnostic recognition, temporal reasoning, and the relative contributions of decoder architecture versus encoder representation.

SLCC dataset: 2,174 images make SLCC smaller than many image datasets, while expanding it would require additional manual labor.The authors do not expect expansion necessarily to change their findings.
SLCC dataset: ∼87% exact-board accuracy on SLCC contrasts with ∼99% on ChessReD and ChessCog, reflecting the introduced dataset’s greater difficulty.The authors describe SLCC as potentially too hard and not necessarily representative of real-world use cases.
Orientation: ChessQueries does not handle fully side-agnostic boards and can confuse white and black pawn directions in certain CVChess positions.The authors suggest training on images with a clear orientation signal, such as SLCC’s consistent placement of the white player on the left.
Temporal information: The approach does not exploit temporal game structure, where consecutive positions should differ by a legal chess move.This temporal signal is explicitly out of scope for the approach.
Ablation: The query decoder improves out-of-domain representations and square localization, while a naive linear head nearly matches its in-domain performance.This suggests that part of the in-domain gain over prior CNN and multi-stage methods comes from the ViT representation backbone.

7. Conclusion

ChessQueries combines a ViT encoder with a DETR-style decoder over learned square queries for single-pass board recognition. It saturates the established benchmarks, transfers to unseen domains, and introduces the challenging SLCC broadcast dataset, where performance remains lower.

Benchmark performance: 99.5% exact-board accuracy on ChessReD and 98.5% on ChessCog show near-saturated performance on the two established benchmarks.The model reads a full board in one forward pass and runs in 19 ms on a GeForce RTX 4090 GPU.
SLCC dataset: 87.1% exact-board accuracy on SLCC exceeds the ChessReD method’s 26% on the challenging broadcast dataset.SLCC contains occlusions, extreme viewpoints, and low-resolution boards.
Dataset contribution: SLCC provides 2,174 broadcast frames from Saint Louis Chess Club professional tournaments as a harder public benchmark.The annotation pipeline is largely automatic, and the dataset can be expanded using additional chess broadcast videos.

A. Worst samples per dataset

Failure patterns differ by dataset: ChessReD and ChessCog are nearly exact even in difficult images, while CVChess exposes orientation errors on an unusual position and viewpoint sensitivity.

ChessReD: At most 2 / 64 squares are wrong in the worst ChessReD samples, mostly because of occlusion.These failures occur under strong perspective and off-board clutter.
ChessCog: A single square is wrong in each of the four worst ChessCog samples, with the mistaken piece heavily occluded.ChessCog is described as a near-saturated synthetic domain.
CVChess: 2–4 wrong squares occur in the four worst CVChess positions when the ground truth is rotated 180◦, indicating upside-down board readings.The model can also produce impossible outputs such as two white kings.
CVChess: Every board is exact on CVChess’s hard human viewpoints, including alternate views of the worst positions.CVChess provides multiple angles for each position.
Attention: Fine-tuned encoder attention localizes distinct on-board regions corresponding to each square across four domains.Removing the shared global-token hotspot reveals a per-square layout that supports linear readout without a decoder.

B. Attention & training the encoder

Fine-tuning reorganizes encoder attention into a per-square layout, whereas a frozen encoder relies on the query decoder to establish image-to-board correspondence. The supplied adaptation table is described but contains no reported values here.

Fine-tuned encoder: Fine-tuning enables a decoder-free linear head to localize and track each square across four domains.The linear-head model remains less precise than the square-query decoder approach.
Frozen encoder: A frozen encoder still supports square localization through decoder cross-attention, while its linear-head attention is diffuse and largely off-board.This matches majority-class-floor accuracy for the frozen linear-head model.
Few-shot adaptation: Table 5 reports few-shot adaptation to SLCC from a ChessReD + ChessCog base model, with zero-shot SLCC performance listed as 0% board accuracy and 17.8 wrong squares.Runs report mean±SD over three support-set draws, except the full-support run.

C. Few-shot adaptation: full results and protocol

Few-shot adaptation compares fine-tuning modes under matched validation tuning and tracks both target performance and retention on ChessReD and ChessCog. LoRA, encoder-only, and full fine-tuning perform similarly, while decoder-only tuning damages source-domain performance.

Adaptation results: LoRA, encoder-only and full fine-tuning tie at every k, with all pairwise gaps at most 2.1 board points.These results are reported for exact-board accuracy on SLCC across adaptation settings.
Source retention: Worst-case source retention decreases as the number of trained parameters increases: LoRA 96.5/98.2, encoder-only 94.8/97.7, and full fine-tune 87.8/96.5.The paired values are source retention on ChessReD/ChessCog board accuracy, respectively.
Source retention: Decoder-only fine-tuning fails to learn the task appropriately, with retention per-square accuracy falling to 0.73 on ChessReD and 0.64 on ChessCog.The method also collapses on the source domain.
Protocol: Validation learning-rate tuning does not favor LoRA: encoder-only and full fine-tuning both select 5×10−6, while decoder-only tuning has no usable window.Six of seven probed decoder-only learning rates end in near-total source-domain collapse loss; the remaining rate selects the base model.
Protocol: The fine-tuning experiment, including learning-rate sweeps, uses under 8 GPU-hours on two RTX 4090s, and the best LoRA weights occupy 12 MB.The LoRA weights attach to a 1.5 GB checkpoint.

D. SLCC annotation and reconstruction pipeline

The SLCC dataset is reconstructed from YouTube chess broadcasts through template-based frame parsing, OCR, and human review of candidate annotations. The pipeline links broadcast frames to tournament and game information before producing board images and associated positions.

Frame parsing: Broadcast frames are parsed using manually annotated templates that locate the main board, player names, and remaining clock times.Templates contain bounding-box information for these production elements.
Frame parsing: Each frame is mapped to its tournament round through the YouTube video ID, and the pipeline associates frames with the opponents’ game information.The passage describes this as part of reconstructing a proper chess board image and associated chess position.
Metadata linkage: The pipeline uses Lichess as an online source of tournament metadata, moves, remaining player time, and other game information.Lichess provides features including online play, analysis, and information on current and past tournaments.
Broadcast processing: For broadcast layouts, the physical-board view is cropped as the model input, OCR reads player names and clock times, and the analysis board is deliberately ignored.The analysis board may show commentary positions rather than the live game.
Annotation review: An annotation interface reviews the candidate annotation extracted from the broadcast before reconstruction proceeds.The figure caption places this review step after the OCR-to-Lichess relay pipeline.