PaperlayerView exact revision on arXiv

Source-linked AI summary

R-CNN-Based Chess Position Recognition

Paras Govind, Ognjen Arandjelović

arXiv:2610.09191v1cs.CV

TL;DR

Recognizing a three-dimensional chess position from one image requires recovering board geometry, square occupancy, and piece identity. The paper combines independent piece and board detectors, using class weighting, deeper classification, keypoints, and geometric constraints to reconstruct positions. On synthetic test positions, the complete framework recovers most positions exactly or with at most one incorrect square.

  • Problem

    A single image of a three-dimensional board must reveal board position and orientation, square occupancy, and piece type including colour.

    Source: §1 · p. 1

  • Method

    Independent piece and board detectors produce hypotheses that are combined through a homography, with class weighting, deeper classification, and piece constraints refining recognition.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    On the synthetic test set, the complete framework recovers 76.61% of positions exactly and 96.49% with at most one incorrect square.

    Source: §6 · p. 16

  • Takeaways & Limitations

    With ground-truth piece boxes, predicted board geometry assigns all true piece boxes to the correct squares.

    Source: §6 · p. 16

    Small errors in the predicted board bounding box can exclude outer corners, making those keypoints vulnerable to box-edge errors.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Performing chess game position recognition solely from a single image of a three-dimensional board requires predicting the position and orientation of the board relative to the camera, the occupancy of squares and the piece type, which includes its colour. We propose an R-CNN-based framework with independent components for piece recognition and board geometry estimation, whose predictions are combined to reconstruct the position. For piece recognition, we adapt Faster R-CNN using a class-weighted objective and a deeper classification head. The detector operates directly on the input image, retaining alternative piece hypotheses that are subsequently refined using constraints on piece counts and square occupancy. For board detection, we introduce an octagonal arrangement of eight labelled boundary keypoints, predicted using the keypoint head of Mask R-CNN. These provide redundant correspondences for homography estimation and encode board orientation. The estimated homography maps representative points from the piece boxes to an 8x8 grid. On a synthetic dataset, the modifications to piece detection increase mean average precision from 61.59% to 90.14%. Of the predicted board keypoints, 97.11% are within 1% of the image diagonal of their labelled targets. Using ground-truth piece boxes with the predicted homographies gives correct square assignments for every test position. The complete framework recovers 76.61% of test positions exactly and 96.49% with at most one incorrect square.

1 Introduction

Transcribing a physical chessboard into a digital position is laborious and error-prone, while image recognition must jointly address piece identity, square occupancy, and board geometry. The paper decomposes recognition into independent piece and board detection components and combines their predictions geometrically.

1 Introduction

  • Over-the-board players must manually copy physical positions into digital boards, a laborious and error-prone task.Photographs can be saved, but still require manual recreation for engine analysis.Source: §1 · p. 1
  • Single-image recognition must determine board position and orientation, square occupancy, and piece type including colour.A single-network 13-class-per-square formulation must also learn localization, classification, occupancy, and mapping under camera, lighting, occlusion, and colour variation.Source: §1 · p. 1
  • Chessboard-plane points can be mapped to a regular grid by computing a homography, motivating decomposition into smaller tasks.This geometric mapping provides an algorithmic component rather than requiring a neural network to solve every subtask.Source: §1 · p. 2
  • The framework separates piece detection from board detection, with both models operating on the full input image.Faster R-CNN localizes and classifies pieces, while Mask R-CNN keypoints estimate board position and orientation; a homography assigns piece-box points to squares without requiring a preceding board crop.Source: §1 · p. 2

2 Related work

Prior work addresses board localization, piece recognition, and occupancy through several distinct strategies, while single-image recognition of three-dimensional boards remains challenging because of perspective and occlusion. The paper positions its approach as combining independent piece and geometry detectors with constraints during reconstruction.

2 Related work

  • For three-dimensional boards, perspective distortion makes board, piece, and piece-location recognition more challenging than for a camera-facing two-dimensional board.A single image also requires identifying pieces without relying on a known position or move record.Source: §2 · p. 2
  • Czyzewski et al. iteratively crop the most confident board heat-map region, while Xie et al. compare perspective-adjusted piece templates with image pieces.Xie et al. report oriented chamfer performance similar to CNNs, with dataset size limiting performance.Source: §2 · p. 2
  • Wölflein and Arandjelović [11] use a four-stage lattice-and-CNN pipeline with 0.23% per-square error on synthetic data, while Zhu et al. target varied camera angles.The four-stage pipeline is susceptible to earlier errors propagating to later stages; Zhu et al. address low-angle occlusion and obscured board regions.Source: §2 · pp. 2–3
  • Other approaches regress board corners before YOLO detection, study real mobile-device photographs [7], or use vision-transformer square queries.The proposed framework instead uses independent piece and board detectors, then combines visual scores with geometric and piece-count constraints during reconstruction.Source: §2 · p. 3

3 Piece recognition

The piece-recognition component adapts Faster R-CNN so that localization and classification can retain competing hypotheses for later constraint-based selection. Its modifications address unequal piece frequencies and visual similarity, then apply class-count limits and select a deeper classification head.

3 Piece recognition

  • The piece-recognition procedure adapts Faster R-CNN at training and prediction, retaining competing class hypotheses for later piece-count and square-assignment constraints.The detector outputs boxes, classes, and confidence scores; Figure 1 shows unfiltered predictions, including multiple hypotheses for a physical piece.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Baseline model and class weighting

  • Class frequencies are uneven: pawns exceed half the training instances, motivating inverse-frequency weighting of the classification loss.The baseline uses Detectron2 with a ResNeXt-101-32x8d backbone and feature pyramid network; horizontal reflection doubles the counts reported for augmented training data.Source: §3.1 · pp. 3–4, Table 1 · p. 4

3.2 Restricting piece counts

  • Piece-count filtering keeps the highest-confidence predictions that satisfy per-colour class limits, assuming no promotions in the evaluated positions.For positions with promotions, the limits would need to allow additional queens, rooks, bishops, or knights; filtering can remove true positives if their confidence is lower than a false positive’s.Source: §3.2 · p. 5

3.3 Deepening the classification head

  • A four-layer box head was selected by pruning a deeper model until reducing it to three layers caused a substantial validation-performance drop.The five-layer starting model used the custom loss but was evaluated without the class-frequency rule to assess its performance independently.Source: §3.3 · p. 5

4 Board geometry and position reconstruction

Board geometry is needed to place detected pieces on the 8×8 grid, and labelled keypoints provide the correspondences for that mapping. The section develops an octagonal layout and a representative-point heuristic for assigning pieces to squares.

4 Board geometry and position reconstruction

  • Board detection is necessary because identifying pieces alone does not determine which squares they occupy.Board localization supplies the piece-to-board relationship needed to recover a position.Source: §4 · p. 5

4.1 Homography and labelled keypoints

  • The homography maps board-plane points into a regular grid, enabling piece detections to be assigned to squares despite perspective distortion.Labelled correspondences also encode board orientation; the grid’s side near the horizontal axis is fixed as White’s side.Source: §4.1 · p. 6

4.2 From a cross to an octagon

  • The proposed octagonal layout uses eight boundary keypoints, with no three collinear, so every four-point subset is geometrically admissible for homography estimation.Unlike the cross centre, which is collinear with opposite corners and cannot replace a missing corner, the octagonal locations can retain margin inside the board box under oblique views.Source: Figure 3 · p. 7, §4.2 · p. 8
  • RANSAC estimates the homography from four-point subsets, rejects inconsistent keypoint predictions, and refines the transformation using inliers.The final model is refined using the inliers.Source: §4.2 · pp. 8–9

4.3 Combining piece and board predictions

  • The representative point approximates the centre of a piece’s base by shifting the projected bottom box midpoint 0.2 grid units toward the projected top midpoint.This addresses cases where the bottom midpoint falls on an adjacent square even though most of the piece base is on one square.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Each representative point is assigned to a grid square, where the highest-confidence box is kept; piece-count and board-position rules discard incompatible predictions.The resulting visible arrangement gives the piece-placement field of Forsyth–Edwards Notation, but a single image cannot generally reveal turn, castling rights, or the preceding move needed for en passant.Source: §4.3 · pp. 9–10

5 Experimental evaluation

The evaluation measures piece detection and board geometry separately before testing their combined reconstruction on a synthetic dataset. The complete framework recovers most positions exactly or with at most one incorrect square, though Chesscog scores higher on complete-position accuracy.

5.1 Dataset and training

  • The 4,888-image Blender dataset used positions from expert games, with varied illumination and camera angles of 45°–60° but consistent piece appearance and colour scheme.The test set contained 342 images and 7,387 pieces.Source: §5.1 · p. 10

5.2 Piece detection results

  • 90.14% mAP for the deep custom-loss detector, versus 78.64% for custom loss and 61.59% for the baseline.The four-layer model with class frequency filtering reached 99.02% conditional classification accuracy.Source: Figure 6 · p. 11, §5.2 · p. 11
  • Piece-count filtering raised conditional classification accuracy to 97.09%, while leaving 175 pieces missed and 540 unmatched predictions.It improved classification of queens, bishops, and knights; the matched piece total fell to 7,212.Source: §5.2 · p. 11

5.3 Board localization and square assignment

  • 97.11% of predicted keypoints were within 1% of the image diagonal of their labelled locations; mean squared reprojection error was 0.06 grid units squared.The corresponding root mean squared reprojection error was approximately 0.245 square units.Source: §5.3 · p. 13
  • Predicted homographies assigned true piece boxes to the correct squares in every test and validation position.Using true boxes isolates the geometry evaluation from piece detection.Source: §5.3 · p. 13

5.4 Complete positions and comparison with Chesscog

  • 76.61% of test boards were recovered without mistakes, 96.49% with at most one mistake, and the mean was 0.28 incorrect squares per board.Table 3 reports complete-system results.Source: §5.4 · p. 13, Table 3 · p. 15
  • Chesscog reached 93.86% complete-position accuracy, exceeding this framework by 17.25 percentage points; it receives board orientation as input, while this framework predicts it.With one mistake allowed, Chesscog scored 99.71% versus this framework’s 96.49%.Source: Table 4 · p. 15, §5.4 · p. 15

6 Discussion and conclusions

The framework combines refined piece detection with octagonal-keypoint board geometry, achieving strong synthetic-test position recovery while leaving specific detection and orientation challenges. Its discussion identifies representation choices and dataset variation as constraints on further development.

6 Discussion and conclusions

  • Class weighting raised mAP from 61.59% to 78.64%, and a deeper head increased it to 90.14%; piece-count filtering improved conditional classification accuracy but removed some correctly localized pieces.Geometric assignment determines which predictions compete for each square.Source: §6 · p. 15
  • Piece heads are sometimes mistaken for high-confidence pawns, potentially due to occlusion or similar local shapes; unmatched predictions also rose after class weighting.Further evaluation should consider false detections as well as classification accuracy.Source: §6 · pp. 15–16
  • The octagonal keypoints provide redundant homography correspondences, avoid collinearity, preserve board orientation, and give RANSAC alternative four-point subsets when predictions are inaccurate.A keypoint excluded by a predicted box cannot be recovered even if its appearance remains clear.Source: §6 · p. 16
  • Recognition of different piece sets may require transfer learning, while unseen sets remain more demanding; variable piece bases also motivate replacing the fixed 0.2-square offset.Estimating piece bases directly could accommodate different base sizes and viewing angles.Source: §6 · p. 16
  • Board orientation remains ambiguous from the checkerboard pattern alone, and piece distribution is unreliable in endgames; unresolved cases require user confirmation or additional game information.The training evaluation included three orientation errors, while test keypoints recovered the orientation needed to assign all true piece boxes correctly.Source: §6 · p. 16
  • 76.61% of test positions were recovered exactly and 96.49% with at most one incorrect square; predicted geometry assigned all true piece boxes correctly.These are results on the synthetic test set.Source: §6 · p. 16
Loading 2610.09191v1…