PaperlayerView exact revision on arXiv

Source-linked AI summary

PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search

Alexander L. Strehl

arXiv:2608.15146v1cs.LG

TL;DR

It was unclear whether simple reinforcement learning could produce competitive cubeful money-game play without relying on publicly documented training regimes or evaluation-time search. PureTD learns checker play and cube action end-to-end through self-play RL, and its search-free model beats GNU Backgammon and Open Sage with one-ply search.

  • Problem

    It was unclear whether simple reinforcement learning could train competitive cubeful money-game evaluators end-to-end when leading engines’ training regimes are not publicly documented.

  • Method

    PureTD trains checker play and cube action end-to-end from scratch using pure self-play reinforcement learning without look-ahead search.

  • Results

    +19.8 mEq/game versus gnubg at 1-ply and +17.4 versus Open Sage at 1-ply, for searchless PureTD.

  • Takeaways & Limitations

    Pure self-play RL can produce competitive search-free play for backgammon money games.

  • Takeaways & Limitations

    The study is limited to money games and does not evaluate match play, which poses additional challenges for reinforcement learning.

Abstract

from arXiv · show

We revisit Tesauro's TD-Gammon for backgammon money games in the setting of no evaluation-time search. Both checker play and cube action (use of the doubling cube) are learned from scratch via self-play reinforcement learning (RL), with minimal hand-coded logic and no expert features. In this setting, we demonstrate that pure self-play RL suffices to train models that reach near-state-of-the-art playing strength. Specifically, for cubeful money games, our search-free model evaluates faster and is substantially stronger than the open-source engines GNU Backgammon and Open Sage running a one-move (1-ply) look-ahead search.

1 Introduction

PureTD studies search-free backgammon money-game play in which self-play reinforcement learning learns both checker play and cube action end-to-end. The paper introduces this approach as a competitive alternative to conventional systems that combine learned evaluators with search and separately handled cube decisions.

  • Modern backgammon engines combine a learned position evaluator with expectimax search over states reached after future rounds of play.
  • Limitations and motivation: The introduction acknowledges that optimizing a single 0-ply evaluator without regard to inference or search cost leaves a gap between evaluator strength and competitive end-to-end performance.The authors nevertheless motivate the searchless setting as a useful measure of the ceiling reachable by self-play RL.
  • Cube decisions as an action-space extension: PureTD extends the self-play RL action space with doubling decisions, learning cubeful checker play and cube action jointly for money games.Doubling decisions are selected by querying the network on resulting states, with each decision producing a normal TD state transition.
  • Cube decisions as an action-space extension: The paper claims its model is the first competitive money-game system to learn both cube action and cubeful checker play end-to-end with RL.The formulation follows Lin (2020), while correcting a conceptual flaw and scaling to larger networks evaluated against strong baselines.
  • Open-source pipeline and large-scale evaluation: The study evaluates search-free models using offline cubeful-money error metrics and head-to-head play against gnubg and Open Sage at 0-ply and 1-ply.Offline metrics include gnubg’s 4-ply mEMG and eXtreme Gammon’s XG++ Performance Rating, with lower values better for both.

2 Background and Related Work

This section situates the work in TD self-play for backgammon, including pseudo-state value learning and exact Bellman targets. It also contrasts existing engines and cube-learning approaches, highlighting reliance on engineered features, rollout refinement, search, or analytic cube formulas.

  • Temporal-difference learning: TD learning models a position’s value in an MDP and selects actions using the current value estimate, with backgammon’s dice providing natural variation during self-play.The maximizer acts greedily, while the opponent minimizes the value; the model is trained incrementally toward Bellman consistency.
  • TD-Gammon and Bellman targets: Pseudo-state training avoids encoding dice, while 0-ply targets sample one roll and Exact Bellman updates average all 21 distinct rolls.Exact Bellman updates remove dice variance but require enumerating every roll.
  • Existing engines: GNU Backgammon uses three specialized single-hidden-layer networks with ∼30k weights each, hand-engineered features, and a precomputed bearoff database.Its networks correspond to contact, race, and crashed game phases.
  • Existing engines: Open Sage uses 16 phase-specialized neural networks and relies on gnubg’s rollout databases, making it dependent on an existing strong engine rather than pure self-play.Its networks range from roughly 100k parameters in common contact positions to 20k in pure races.
  • RL versus supervised refinement: Existing engines refine TD-trained evaluators on selected positions using targets averaged over repeated rollouts, a variance-reduced RL procedure beyond TD self-play alone.This technique is confirmed for Open Sage and believed to be used by gnubg.
  • Cube decisions: Prior cube approaches either integrate cube actions into the action space or extend cubeless models with Janowski’s formulas, while Lin’s work used a roughly 48k-weight single-hidden-layer network.Janowski’s formulas approximate cube decisions from cubeless equity and gammon rates rather than providing a closed-form optimum.

3 Method

PureTD represents backgammon positions from the side-to-play perspective and extends the board encoding with cube-state features for cubeful money play. It trains progressively larger ReLU MLPs through self-play, using sampled backups initially and exact 1-ply backups later, while treating cube offers as learned actions.

  • Representation: The board encoding uses 196 features, including four per-point checker features per player and a fixed side-to-play perspective.The fourth per-point feature represents surplus checkers beyond three, halved; the perspective encoding also captures directional symmetry.
  • Representation: Cubeful money play adds four cube-related inputs, producing 200 features that encode cube ownership and whether the state is a cube-decision position.The added inputs are three one-hot ownership features—cube_centered, cube_own, and cube_opp_own—and the binary is_cube_action flag.
  • Architecture: The cubeful model is a 562k-parameter PyTorch MLP with 200 inputs, hidden layers [512, 512, 256, 256], and ReLU activations.Its linear output is interpreted as equity per unit cube value, with realized equity equal to output × current_cube_value.
  • Training: Training alternates self-play data collection and one minibatch-Adam epoch per round, playing N = 1000 games with the current network as both players.Models are expanded through stages from [80] to [512, 512, 256, 256], transferring parameters with function-preserving Net2Net transformations.
  • Cube decisions: PureTD treats cube offers as ordinary learned actions and resolves take/pass greedily from the learned evaluator rather than using a separate analytic cube module.During 1-ply training, checker plays use the expectation over all 21 dice rolls, while cube offers use the cheaper update.
  • Cube decisions: The is_cube_action feature distinguishes the cube-decision value from the post-no-double checker-play value, correcting a representational flaw in Lin’s formulation.The cube-decision value can be exactly 1 when the opponent passes, whereas continued play is typically worth much less than 1.

4 Experiments

Experiments compare PureTD with GNU Backgammon and Open Sage using head-to-head play and offline cubeful analyses. PureTD is substantially stronger at matched 0-ply and remains stronger against opponents granted 1-ply search, while offering a favorable strength–cost tradeoff.

  • Experimental setup: The experiments compare PureTD against GNU Backgammon and Open Sage, with head-to-head play as the primary metric and XG++ PR and gnubg mEMG for offline cubeful analysis.Both offline error metrics are lower-is-better, with 0 representing perfect play.
  • Cubeless money: At matched 0-ply, the cubeless prob5 model is substantially stronger than both baselines, and PureTD’s advantage persists when the baselines receive an extra search ply.The prob5 model uses five outcome-probability outputs trained with the same TD-learning procedure as the cubeful model.
  • Throughput: At 0-ply, PureTD’s code is 5–12× slower than gnubg and 2.7–4.6× slower than Open Sage’s efficient path.The gap reflects larger networks and PureTD’s general-purpose PyTorch evaluation path versus gnubg’s optimized hand-written network-evaluation code.
  • Throughput: PureTD’s 0-ply network is stronger than both baselines with 1-ply search and evaluates at least 2.5× faster than that search.Together, the results indicate a favorable cost-versus-strength tradeoff for PureTD.

5 Limitations and Future Work

The paper identifies three future-work directions: extending evaluation beyond money games to match play, systematically studying models with evaluation-time search, and addressing an additional unfinished direction that is truncated in the supplied passage.

  • Future work: The study is limited to cubeless and cubeful money games, not match play, where score-aware cube strategy and longer horizons challenge RL.Match play introduces score-aware cube strategy and a much longer horizon to a terminal state.
  • Future work: A systematic study of the models with evaluation-time search is identified as an important follow-up.The paper’s focus was on neural-network performance and capacity without evaluation-time search.
  • Future work: The passage signals a third future-work direction, but its description is truncated in the supplied text.The supplied paragraph ends after “we do not y,” so the direction cannot be specified further.

6 Conclusion

PureTD is an open-source backgammon money-game engine trained with a pure reinforcement-learning formulation. Without look-ahead search, it plays more strongly than Open Sage and gnubg using one-step look-ahead, while differing from prior work through larger networks, batch training, and direct cube-action learning.

  • Contribution: PureTD is an open-source backgammon money-game engine trained using a pure reinforcement-learning formulation.Its formulation is similar to Tesauro’s original TD-Gammon.
  • Playing strength: Without look-ahead search, PureTD achieves stronger play than Open Sage and gnubg with one-step look-ahead search.
  • Methodological differences: Relative to prior work, PureTD uses substantially larger networks, batch rather than online training, and direct learning of cube action.

A Reproducibility Details

The experiments specify exact engine versions, evaluation configurations, pruning policies, and repository commands for reproducibility. Benchmarks use controlled timing procedures and show PureTD 0-ply achieves at least 2.5× the combined decision throughput of any 1-ply method.

  • Engine versions: Open Sage v1.3.20260723 and GNU Backgammon 1.08.003 were used, with exact experiment commands recorded in the repository.GNU Backgammon mEMG error rates were computed through its command-line interface at 4-ply.
  • Open Sage configuration: Open Sage runs use BgBotAnalyzer for cubeful play and best_move_index for cubeless play, with reported N-ply corresponding to its “(N+1)P” setting.All other Open Sage parameters use shipped defaults, and its root move filter is inert at 0-ply.
  • 1-ply move pruning: Head-to-head, mEMG, and XG++ PR analyses disable move pruning, while cost-versus-strength tables use each engine’s developer-default pruning.At 1-ply without pruning, every engine expands every legal move; pruning has only a small effect on the reported results.
  • Computational benchmark setup: Fixed-position benchmarks replay identical positions and cube states, timing production per-decision calls on shared pools of 8,000 cubeless and 10,000 checker plus 6,000 cube-offer positions.Cube-offer positions include about half redouble decisions, matching the natural self-play mix.
  • Computational benchmark setup: 2.5×: PureTD 0-ply’s combined decision throughput is at least 2.5× that of any 1-ply method.The closest case is GNU Backgammon 1-ply on Zen 3, at 2.60×.

B Complete Benchmark Tables

Tables 9 and 10 provide single-thread throughput results for cubeless and cubeful money play across self-play and fixed-position benchmarks on both machines. Cubeful fixed-position testing separates checker play from cube action, whereas self-play reports a combined rate.

  • Benchmark scope: Tables 9 and 10 cover throughput under self-play and fixed-position benchmarks for both game types on both machines.The measurements are single-thread throughput results.
  • Cubeless money: Cubeless-money throughput reports single-thread checker-decision throughput in decisions/s.Each figure is the mean of five independent runs with bracketed 95% t-intervals over those runs.
  • Cubeful money: Cubeful-money self-play reports one combined rate over checker and cube decisions, while fixed-position testing reports the two decision types separately.Each figure is the mean of five independent runs with bracketed 95% t-intervals over those runs.
  • Cubeful money: Open Sage’s cubeful rows run through BgBotAnalyzer, which is far slower than its efficient cubeless path for both decision types.This condition applies to both cubeful decision types in Table 10.

C Offline Analysis on Mixed-Play Games

An offline XG++ PR robustness analysis used 1,000 head-to-head games between the cubeful money model at 0-ply and gnubg at 1-ply. The resulting estimates agreed with self-play values within the confidence intervals.

  • Mixed-play robustness analysis: 1,000 head-to-head games compared the cubeful money model at 0-ply with gnubg at 1-ply, unpruned.Each engine’s decisions were scored separately.
  • Mixed-play robustness analysis: The mixed-play offline XG++ PR estimates agreed with the self-play values of Table 3 within the confidence intervals.The analysis used 95% confidence intervals.

D Retraining the Released Models

The released models were fully retrained from a shared DMP-to-cubeless-equity base, then specialized into prob5 and cubeful-money models. The cubeful retraining reached parity with the production model at 190.5M episodes and ended at +1.5 after 232M episodes.

  • Shared base: Steps 1–15 train the DMP network, while steps 16–19 use it to warm-start a cubeless equity model.This shared base supports both retrained models, with prob5 branching after step 17 and cubeful continuing through steps 18–19.
  • prob5 retraining: The prob5 model starts from step 17 of the shared base, copying ≈495k of its 528k parameters and freshly initializing the final hidden layer and five-output probability head.The copied parameters are the base’s first three hidden layers.
  • Cubeful retraining: The cubeful-money model starts from step 19, widening the input layer from 196 to 200 features while copying the hidden layers and equity head unchanged.The four new features are three cube-ownership indicators and is_cube_action, initialized fresh.
  • Cubeful retraining: 190.5M episodes mark parity with the production cubeful model, and the run ends at +1.5 [−0.3, 3.2] after 232M episodes.The reported evaluation precision at these checkpoints is 107 games (±1.8).
Loading 2608.15146v1…