PaperlayerView exact revision on arXiv

Source-linked AI summary

PokerBench: Training Large Language Models to become Professional Poker Players

Richard Zhuang, Akshat Gupta, Richard Yang, Aniket Rahane, Zhengyu Li, Gopala Anumanchipalli

arXiv:2501.08328v2cs.CLcs.AIcs.GT

TL;DR

Poker is a demanding incomplete-information game, yet reliable evaluation of LLM poker skill is difficult because meaningful play assessment requires large hand samples. The paper introduces POKERBENCH, an 11,000-spot benchmark and training dataset, and finds that current models underperform optimal poker while fine-tuning improves performance. Benchmark scores also correspond to stronger results in game simulations, though dataset evaluation and gameplay measure different things.

  • Problem

    Poker combines hidden opponent holdings with strategic planning, while reliable assessment requires statistically significant samples of hands, making quick evaluation of LLM poker ability a challenge.

    Source: Introduction · pp. 1–2

  • Method

    The authors build POKERBENCH, an 11,000-spot benchmark spanning pre-flop and post-flop play, and use it to evaluate and fine-tune language models.

    Source: Introduction · p. 2

  • Results

    Current LLMs underperform optimal poker, with GPT-4 at 53.55% accuracy; fine-tuning improves models, and higher POKERBENCH scores correspond to superior play in 50k-hand simulations.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    POKERBENCH offers a quick measure of LLM optimal-poker ability and a benchmark for tracking progress in this domain.

    Source: Conclusion · p. 7

    Dataset evaluation measures performance against GTO play, whereas gameplay measures performance against imperfect strategies, so the two evaluations need not align for a non-perfect GTO model.

    Source: Fine-Tuning LLMs into Better Poker… · p. 7

Abstract

from arXiv · show

We introduce PokerBench - a benchmark for evaluating the poker-playing abilities of large language models (LLMs). As LLMs excel in traditional NLP tasks, their application to complex, strategic games like poker poses a new challenge. Poker, an incomplete information game, demands a multitude of skills such as mathematics, reasoning, planning, strategy, and a deep understanding of game theory and human psychology. This makes Poker the ideal next frontier for large language models. PokerBench consists of a comprehensive compilation of 11,000 most important scenarios, split between pre-flop and post-flop play, developed in collaboration with trained poker players. We evaluate prominent models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models, finding that all state-of-the-art LLMs underperform in playing optimal poker. However, after fine-tuning, these models show marked improvements. We validate PokerBench by having models with different scores compete with each other, demonstrating that higher scores on PokerBench lead to higher win rates in actual poker games. Through gameplay between our fine-tuned model and GPT-4, we also identify limitations of simple supervised fine-tuning for learning optimal playing strategy, suggesting the need for more advanced methodologies for effectively training language models to excel in games. PokerBench thus presents a unique benchmark for a quick and reliable evaluation of the poker-playing ability of LLMs as well as a comprehensive benchmark to study the progress of LLMs in complex game-playing scenarios.

Introduction

Poker requires reasoning under incomplete information, planning, and modeling opponent behavior, making it a demanding game-playing test for LLMs. The paper introduces POKERBENCH as a quick evaluation dataset because reliable poker assessment otherwise requires large samples of hands.

Introduction

  • Poker tests LLMs beyond math and reasoning because players must estimate hidden opponent holdings from actions and plan around possible future outcomes.The paper frames game-playing as a next evaluation setting as LLMs improve on traditional NLP tasks.Source: Introduction · p. 1
  • Poker solvers can be slow, cover only discrete scenarios, miss opportunities to exploit imperfect opponents, and provide strategies that are difficult to interpret.The authors describe LLMs as potentially offering instant solutions across spots and explainable strategies, while noting those potential uses are not explored in this paper.Source: Introduction · pp. 1–2
  • POKERBENCH provides 11,000 poker decision-making spots, including 1,000 pre-flop and 10,000 post-flop scenarios, as a quick, accurate indicator of LLM poker-playing skills, with an accompanying training dataset.The dataset is intended to avoid running simulations for tens of thousands of hands after every fine-tuning iteration.Source: Introduction · p. 2
  • GPT-4 leads the evaluated state-of-the-art models at 53.55% accuracy, while fine-tuning improves the performance of Llama-3-8B, Llama-2-7B, and Gemma-2B.The paper also reports that models with higher POKERBENCH scores beat lower-scoring models over a large sample of hands.Source: Introduction · p. 2

Related Work

Prior work spans superhuman poker solvers and early LLM game-playing studies, but earlier LLM poker evaluations and training workflows had important scope or evaluation constraints.

Related Work

  • Poker solvers achieved superhuman performance through algorithmic advances, but Gupta (2023) limited LLM poker evaluation to raise-first-in spots, and Huang et al. (2024) evaluated models only after fine-tuning.The paper presents POKERBENCH as a universal benchmark for ongoing development and evaluation of LLM poker agents.Source: Related Work · p. 2
  • Deep Blue beat Kasparov in one game in 1997 but lost their match 4-2, while AlphaGo defeated a Go world champion 4-1 in 2016.Poker differs from chess and Go because players cannot see opponents' holdings.Source: Related Work · p. 2
  • Ruoss et al. (2024) explored non-search-based LLM chess systems, while Gupta (2023) was the first cited work examining LLM poker play.Gupta concluded LLMs were not good poker players, based on the restricted raise-first-in evaluation.Source: Related Work · p. 2

Poker Preliminaries

The paper studies Texas No-Limit Hold’em, where players act with private hole cards and shared community cards across four betting rounds. It introduces the units, actions, and game-theoretic strategy concepts used to describe play.

Poker Preliminaries

  • Texas No-Limit Hold’em gives each player two private hole cards and reveals five shared community cards over three rounds, with four betting rounds.A player makes a winning combination from the best five of their seven available cards.Source: Poker Preliminaries · pp. 2–3
  • A hand is one iteration of dealing hole and community cards and playing through the betting rounds to determine who wins or loses the pot..Source: Poker Preliminaries · p. 3
  • Poker actions include checking without wagering when no bet is pending, betting first, calling a wager, raising it, or folding and relinquishing the pot.Source: Actions in Poker · p. 3
  • The big blind (BB) is the normalized minimum bet unit, and win rate is measured as big blinds won per hundred hands.Poker stakes are commonly classified by the cost of a 100-BB buy-in.Source: The Unit of Measurement in Poker · p. 3
  • A game-theory-optimal (GTO) strategy is balanced so opponents cannot exploit it by reliably linking a player's actions to their holdings.The paper illustrates this with varying whether pocket aces are played all-in so opponents cannot infer that holding from the action.Source: Game Theory Optimal Poker · p. 3

The POKERBENCH Benchmark

POKERBENCH is designed as a quick, reliable evaluation of optimal poker play, balancing broad scenario coverage against manageable inference time. It curates pre-flop and post-flop scenarios through GTO-based filtering and sampling.

The POKERBENCH Benchmark

  • POKERBENCH aims to evaluate optimal poker play quickly and reliably by covering diverse scenarios while keeping inference time reasonable.A spot combines a player's hole cards, the board, and actions across betting rounds.Source: The POKERBENCH Benchmark · p. 3
  • Pre-flop evaluation limits scenarios to at most two raises, covering folds, single raises, and 3-bet pots.The authors state that most pots do not go beyond a single raise.Source: Pre-flop Action Selection · p. 4
  • From roughly 311 million possible boards, the benchmark groups common flop situations into 11 textures and samples flops evenly across them.Turn cards are selected to cover new features, such as completing or not completing a flush draw.Source: Board Selection · p. 4
  • Hole cards are filtered to cases where one action is dominant with greater than 50% probability, excluding more mixed-strategy hands.This selection rule is used because GTO play can require nondeterministic action choices.Source: Selecting Hole Cards · p. 4
  • The benchmark contains 1,000 pre-flop and 10,000 post-flop evaluation spots, alongside training sets of 60k pre-flop and 500k post-flop spots.GTO strategies come from GTOWizard for pre-flop and WASM-Postflop for post-flop play.Source: Dataset Summary · p. 4, The POKERBENCH Benchmark · p. 4
  • Evaluation uses balanced shares of fold, call, check, and bet/raise labels, while post-flop training resamples actions to reduce fold dominance.The benchmark down-samples folds because a fold-all strategy could otherwise achieve close to 90% accuracy.Source: Dataset Summary · pp. 4–5

Experiments

Experiments find substantial GTO poker gaps in pretrained LLMs and gains from fine-tuning on PokerBench. Scores align with head-to-head results among fine-tuned checkpoints, but the GPT-4 match exposes a mismatch between GTO evaluation and play against imperfect strategies.

Are Modern LLMs Good at Playing Optimal Poker?

  • The few-shot evaluation used five in-context examples and measured both optimal action choice and exact wager amounts.Action Accuracy checks the chosen action; Exact Match Accuracy also checks whether the bet or raise amount is optimal.Source: Are Modern LLMs Good at Playing… · p. 5, Experiments · p. 5
  • GPT-4 outperformed every evaluated model in pre-flop and post-flop play, while all modern LLMs substantially lacked GTO poker ability.ChatGPT 3.5 performed comparably to the much smaller Llama-3-8B; Llama-3 had higher exact-match scores than Llama-2 but similar action accuracy.Source: Are Modern LLMs Good at Playing… · p. 5, Table 2 · p. 6

Fine-Tuning LLMs into Better Poker Players

  • After fine-tuning, Llama-3-8B improved considerably and outperformed GPT-4 on PokerBench; Llama-2-7B was similar, while Gemma-2B lagged.Training ran for 5,000 optimization steps; a higher learning rate produced unstable fine-tuning, and Figure 2 plots Llama-3-8B loss and checkpoint accuracy.Source: Fine-Tuning LLMs into Better Poker… · p. 6, Table 2 · p. 6, Figure 2 · p. 6

Heads-Up Games between Fine-tuned Checkpoints

Fine-tuned Model vs. GPT-4

  • In the 1,000-hand match, GPT-4 outplayed Player 5000 despite Llama’s stronger GTO-style pre-flop play; GPT-4 won every hand in which it used donking.The benchmark assumes GTO play, leaving the fine-tuned model unfamiliar with donking; the limited sample makes absolute win rate less meaningful.Source: Fine-Tuning LLMs into Better Poker… · p. 7, Table 5 · p. 7

Discussion

  • PokerBench evaluates agreement with GTO play, whereas gameplay tests performance against imperfect opponents; the relationship is complex for an imperfectly learned GTO strategy.The authors identify this as a limitation of simple supervised fine-tuning for poker.Source: Fine-Tuning LLMs into Better Poker… · p. 7

Conclusion

POKERBENCH evaluates LLMs’ optimal poker-playing ability and supports comparisons of their strategic performance. The authors report that fine-tuning can produce models that outperform much larger models, and that benchmark scores correspond to poker-playing performance in 50,000-hand simulations.

Conclusion

  • POKERBENCH scores translate to superior poker-playing skills in game simulations over 50k hands.The authors present the benchmark as a quick and reliable measure of optimal poker-playing ability.Source: Conclusion · p. 7
  • Fine-tuned Llama-3-8B and other models can outperform much larger models.The paper reports that current state-of-the-art LLMs fail significantly at optimal poker before fine-tuning.Source: Conclusion · p. 7
  • POKERBENCH is presented as a comprehensive benchmark for evaluating LLMs’ optimal poker-playing ability.The authors also position it as a way to study LLM progress in complex game-playing scenarios.Source: Conclusion · p. 7
  • The optimal rock-paper-scissors strategy plays each option 33% of the time.Source: Conclusion · p. 7

Appendix

The appendix includes analysis of GPT-4 versus a fine-tuned model across poker stages, alongside board-texture descriptions, training-dynamics figures, and evaluation-prompt examples.

Explanation of Board Textures

  • The appendix describes flop board textures including wet, dynamic, mid-, low-, and monotone boards, and Figure 3 shows the textures used to filter post-flop search space.A monotone board has all cards of the same suit.Source: Explanation of Board Textures · p. 9, Figure 3 · p. 10

Pre-flop

  • In 752 hands ending pre-flop, Llama open-raised 27.3% of the time and GPT-4 15.3%, with GPT-4 described as playing too tightly.Llama’s multiple-raise strategy included bluffing hands, while GPT-4 reraised only with premium hands.Source: GPT-4 vs. Fine-tuned Model Gameplay… · p. 9

Flop/Turn

  • GPT-4’s donking was underrepresented in the GTO-based dataset, and it won all 15 hands in which it donked as the pre-flop non-aggressor.The analysis describes donking as betting out of position on the flop or turn; the pre-flop aggressor is the player who raised last before the flop and generally holds a stronger hand.Source: GPT-4 vs. Fine-tuned Model Gameplay… · p. 9

GPT-4 vs. Fine-tuned Model Gameplay Analysis

  • GPT-4 bet or raised 0.823 times per street on average, versus 0.312 for Llama; despite being farther from GTO, GPT-4 still won against Llama.The authors attribute this matchup result partly to GPT-4’s post-flop aggression and style suppression.Source: GPT-4 vs. Fine-tuned Model Gameplay… · p. 9

River

Training dynamics

Prompt example

  • A sample prompt asks a six-handed Texas Hold’em specialist to choose an optimal action, illustrated with an HJ holding pocket kings facing raises and an all-in answer.The example gives a 27-chip pot before asking for the decision.Source: GPT-4 vs. Fine-tuned Model Gameplay… · p. 11

Few-shot evaluation

  • The few-shot evaluation presents multiple poker spots before the query spot.Table 7 is identified as an example prompt for few-shot evaluations of pretrained LLMs.Source: Table 7

Related papers

Loading 2501.08328v2…