PaperlayerView exact revision on arXiv

Source-linked AI summary

PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Model

Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, Yanru Zhang

arXiv:2401.06781v1cs.AIcs.CL

TL;DR

CFR-based Poker methods are computationally costly and difficult to extend to multiple players as game trees grow exponentially. PokerGPT fine-tunes a lightweight LLM on processed real-game records, and the authors report competitive win rate with advantages in training cost and inference speed.

  • Problem

    CFR is computationally expensive and difficult to extend to multi-player Poker because the game tree grows exponentially with player count.

    Source: §I · p. 1

  • Method

    PokerGPT processes real-game data into prompts and fine-tunes a lightweight pre-trained LLM using RLHF and data selection based on player win rates.

    Source: §I · p. 2, C. Training Process · p. 7

  • Results

    PokerGPT achieves a competitive win rate while showing advantages in training cost and inference speed among previous works.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The authors conclude that PokerGPT demonstrates the potential of LLMs for solving imperfect-information games.

    Source: §VI · p. 12

Abstract

from arXiv · show

Poker, also known as Texas Hold'em, has always been a typical research target within imperfect information games (IIGs). IIGs have long served as a measure of artificial intelligence (AI) development. Representative prior works, such as DeepStack and Libratus heavily rely on counterfactual regret minimization (CFR) to tackle heads-up no-limit Poker. However, it is challenging for subsequent researchers to learn CFR from previous models and apply it to other real-world applications due to the expensive computational cost of CFR iterations. Additionally, CFR is difficult to apply to multi-player games due to the exponential growth of the game tree size. In this work, we introduce PokerGPT, an end-to-end solver for playing Texas Hold'em with arbitrary number of players and gaining high win rates, established on a lightweight large language model (LLM). PokerGPT only requires simple textual information of Poker games for generating decision-making advice, thus guaranteeing the convenient interaction between AI and humans. We mainly transform a set of textual records acquired from real games into prompts, and use them to fine-tune a lightweight pre-trained LLM using reinforcement learning human feedback technique. To improve fine-tuning performance, we conduct prompt engineering on raw data, including filtering useful information, selecting behaviors of players with high win rates, and further processing them into textual instruction using multiple prompt engineering techniques. Through the experiments, we demonstrate that PokerGPT outperforms previous approaches in terms of win rate, model size, training time, and response speed, indicating the great potential of LLMs in solving IIGs.

I. INTRODUCTION

Poker is a challenging imperfect-information game, while CFR-based approaches face high resource costs and difficulty scaling to multiple players. PokerGPT applies a lightweight LLM to real-game text and reports advantages across several performance and efficiency measures.

I. INTRODUCTION

  • CFR-based Poker solvers require substantial computation and storage, and multi-player game trees grow exponentially with player count.The introduction also identifies information loss from game-tree compression and reliance on human expert knowledge as concerns.Source: §I · pp. 1–2
  • PokerGPT uses real-game records transformed into textual prompts, labeled with observed actions, to fine-tune a lightweight LLM with RLHF.The selected model is OPT-1.3B [8].Source: §I · p. 2
  • The proposed approach is presented as end-to-end, cost-effective to train, convenient for human interaction, and able to handle arbitrary player counts.The paper also claims fewer parameters and shorter inference time than conventional methods such as CFR, and less training time than newer deep-learning works.Source: §I · p. 2
  • PokerGPT’s real-game data processing combines cleansing and prompt engineering, which the authors say significantly improves fine-tuning.The introduction also reports selecting high-win-rate player behavior as part of the data-processing approach.Source: §I · p. 2
  • Experiments report advantages over existing models in win rate, training speed, and response time, and introduce action score and average investment to study strategy.The supplied introduction gives no specific numerical comparisons for these outcomes.Source: §I · p. 2

II. RELATED WORK

Prior Poker research developed CFR-based solvers and neural approaches, while LLM studies explored language models in games and other tasks. The paper positions PokerGPT as an LLM-based approach to Poker.

A. Solutions for Poker

  • CFR and its variants supported strong Poker results, including Libratus in heads-up no-limit Poker and Pluribus in six-player Poker, but remained computationally and memory intensive.Earlier work also used subgame solving, leaf-value substitution, regret discounting, and reweighting to improve CFR methods.Source: A. Solutions for Poker · p. 3
  • Neural methods approximated CFR values, learned strategies, or combined self-play reinforcement learning with CFR to improve exploration and performance.The cited work also describes end-to-end deep reinforcement learning approaches and acceleration techniques.Source: A. Solutions for Poker · p. 3
  • The review identifies continued reliance on computation, prior knowledge, and domain-specific heuristics, alongside limited human practicality, as open concerns for Poker methods.It concludes that more efficient algorithms for Poker and other complex imperfect-information games remain an open problem.Source: A. Solutions for Poker · p. 3

B. LLM

  • Prior LLM game studies covered human-like responses, game-quest generation, dialogue suggestions, and repeated games, with performance varying by task.The reviewed work reports suboptimal coordination but stronger performance in individual-competition games; PokerGPT is proposed as an LLM framework for Poker.Source: B. LLM · p. 3

III. PREREQUISITES

Poker proceeds through private cards, community-card reveals, betting, and a showdown or folds; the section also introduces LLM training stages relevant to PokerGPT. The accompanying figures illustrate a two-player game and card strengths.

A. Poker Rules

  • Poker players use two private and five community cards to form a best five-card hand, betting or folding until a showdown or a winner remains.The rules describe PREFLOP, FLOP, TURN, RIVER, and SHOWDOWN stages.Source: A. Poker Rules · p. 3

B. LLM

  • LLMs generate natural-language text from inputs and are trained through data preparation, pre-training, and fine-tuning.Source: B. LLM · pp. 3–4

A. Poker Rules

1) Pre-training:

  • Pre-training uses self-supervised learning on data itself, with objectives including masked, next-sentence, permutation, and causal language modeling.Source: 1) Pre-training · p. 4

2) Fine-tuning:

  • Fine-tuning uses labeled or partially labeled data to adapt an LLM to a task before inference.Source: 2) Fine-tuning · p. 4

3) RLHF:

PokerGPT combines real-game logs, text prompts, supervised fine-tuning, reward modeling, and RLHF. The logs capture game setup and staged play, while fixed instructions are intended to support simple human interaction.

3) RLHF:

  • RLHF trains a reward model from human feedback and uses reinforcement learning to optimize an agent policy, especially when rewards are sparse or noisy.Source: 3) RLHF · p. 4

IV. POKERGPT

  • PokerGPT collects real-game data, turns it into prompts, selects high-quality prompts, and uses supervised fine-tuning, reward modeling, and RLHF.The framework fixes information as instructions to keep human interaction simple.Source: §IV · p. 4

1) Data Description:

  • PokerStars logs include blinds, player seating and chip counts, dealer position, game stages, and a summary of the pool, community cards, and player earnings.Source: 1) Data Description · p. 4
  • A hand log follows PREFLOP, FLOP, TURN, and RIVER play through SHOWDOWN, when private cards are revealed and the winner is determined.Source: 1) Data Description · p. 4

2) Information Filtering:

The information-filtering discussion identifies game-state and opponent-behavior details that PokerGPT uses to represent a hand. It emphasizes cards, action history, player context, and betting amounts as prompt information.

2) Information Filtering:

  • Opponent actions and bet sizes are treated as clues to card strength and playing style, supporting decisions about bluffing and reading bluffs.Source: 2) Information Filtering · pp. 5–6
  • Prompts include private and public cards, card characteristics, and the hand rank so the model can assess combinations and their strength.Source: 2) Information Filtering · p. 6
  • The representation records sequential player actions, blind amounts, and positions relative to the dealer as context for interpreting decisions.Source: 2) Information Filtering · p. 6
  • Player count, starting and invested balances, folds, pot value, and betting stage are also tracked to convey game structure and decision context.The passages associate pot value with odds and note that an action can serve different purposes at different stages.Source: 2) Information Filtering · p. 6

B. Prompt Engineering

PokerGPT converts verbose game records into prompts intended to be readable by people and the model, then filters examples for training. Selection uses revealed outcomes and player performance, with low-performing players supplying negative examples.

B. Prompt Engineering

  • Prompts turn verbose raw game data into a human- and model-understandable form, allowing suggestions from simple inputs during real-time play.Source: B. Prompt Engineering · p. 6

1) Prompt Selection:

  • Because game logs contain diverse information, the authors construct information-rich prompts and select high-quality examples for training.Source: 1) Prompt Selection · p. 6
  • Selection first retains SHOWDOWN logs with revealed private cards and excludes cases without a specific winning or losing hand.The stated reason is that missing hand-strength information makes optimal play harder to learn in observable situations.Source: 1) Prompt Selection · p. 6
  • Players are ranked by revenue converted to milli-bigblinds per hand, while those below a minimum game-count threshold are excluded.Source: 1) Prompt Selection · p. 6
  • Top-ranked players supply high-quality training data; their losing actions can still help the model learn how to lose less.Source: 1) Prompt Selection · p. 6
  • Data from the worst players provides negative examples whose actions and ideas the model can learn to avoid through reward modeling.Source: 1) Prompt Selection · p. 6

2) Prompt Construction:

PokerGPT uses concise, zero-shot prompts that combine game information with fixed instructions, then fine-tunes OPT-1.3B through supervised training, reward modeling, and RLHF. Prompt engineering reduces input demands and converts betting amounts into discrete choices.

2) Prompt Construction:

  • Zero-shot prompts use fixed instructions rather than few-shot demonstrations, which the authors set aside because of decision-time and token constraints.They note players generally have about 15 seconds to decide and that fine-tuning makes few-shot examples unnecessary.Source: 2) Prompt Construction · p. 7
  • Prompts separate constant information, fixed at the start of a game, from dynamic information updated each decision round.Source: 2) Prompt Construction · p. 7
  • Prompt engineering assigns the model a role and discretizes bet and raise amounts into big-blind multiples, turning value prediction into classification.When an opponent reveals private cards, the prompt can replace the hidden-card marker with those cards.Source: 2) Prompt Construction · p. 7

C. Training Process

  • Training uses OPT-1.3B with supervised fine-tuning, a reward model, and RLHF implemented with PPO.Win rates label the reward model, which scores prompt-action pairs to guide PPO updates.Source: C. Training Process · p. 7
  • The authors motivate supervised fine-tuning as a way to adapt a pre-trained model without the substantial resources required to retrain it.They state that domain-specific refinement can use tens to hundreds of high-quality examples.Source: C. Training Process · p. 7
  • The iterative RLHF process generates answers, ranks them with the reward model, and updates the model to favor higher-reward answers.The authors say this is intended to improve robustness to human-like input and generalizability across game scales and situations.Source: C. Training Process · pp. 7–8

V. EXPERIMENT

The experiments describe a Slumbot comparison and ablations of data filtering, player-win-rate selection, and prompt engineering. The section also defines the evaluation measures used for action classification, value prediction, and language-model prediction.

A. Fundamental Setup

  • PokerGPT is fine-tuned on OPT-1.3B using about 120,000 processed games from a pool of over 1 million, with a 90/10 train-test split.The experiments run on one NVIDIA GeForce 3090, with mini-batch size 4.Source: A. Fundamental Setup · p. 8

1) Performance Comparison:

2) Ablation Studies:

  • Ablations vary raw versus filtered data and player win-rate thresholds; action prediction uses macro F1, while bet and raise values use frequency comparisons and MSE.Perplexity is also measured; lower perplexity generally indicates better next-word prediction.Source: 2) Ablation Studies · p. 8

3) Multi-Player Evaluation:

The multi-player evaluation examines how PokerGPT’s action patterns and investment change as player count changes. It uses action score and average investment to describe those strategy variations.

3) Multi-Player Evaluation:

  • The multi-player evaluation uses an open-source Poker simulation and holds the equity-calculation method constant across bots.Source: 3) Multi-Player Evaluation · p. 8
  • Action score measures an action’s frequency per game stage, then averages that score across games.The normalization accounts for actions such as checks that can occur repeatedly in a game.Source: 3) Multi-Player Evaluation · pp. 8–9
  • Average investment is the mean money used by the model per game; together with action scores, it characterizes strategy across player counts.The authors note that greater money use can mean higher revenue as well as higher losses.Source: 3) Multi-Player Evaluation · p. 9

4) Analysis on Interaction:

PokerGPT’s Slumbot comparison reports a higher win rate than two recent models, alongside shorter training time and support for arbitrary player counts. Ablations find the strongest effect from selecting training prompts by player win rate, with lower-quality data associated with less accurate and more aggressive play.

4) Analysis on Interaction:

C. Comparison with Previous Works

  • 158+49 mbb/h is PokerGPT’s win rate against Slumbot, versus 45+5 for Rebel and 111+16 for Alphaholdem.The authors report greater standard deviation for PokerGPT, indicating more fluctuation in its performance.Source: C. Comparison with Previous Works · p. 9
  • 9.5 GPU hours is PokerGPT’s training time, compared with 580 GPU hours for Alphaholdem.PokerGPT supports arbitrary player counts, while the two comparison models are limited to one-on-one play.Source: C. Comparison with Previous Works · p. 9
  • PokerGPT’s revenue distribution is reported as more stable than human players’ in later stages, where humans are more likely to take risks.The figure compares revenue distributions over 1,000 games, using big blinds as the unit.Source: C. Comparison with Previous Works · p. 9, Figure 7 · p. 9

D. Ablation Results

  • Dataset III performs best and Dataset VI worst; poorer datasets yield more incorrect action choices and less rational investment, with win-rate-based prompt selection having the strongest effect.The authors also report that Dataset VI has perplexity similar to Datasets III–V despite performing worse.Source: D. Ablation Results · pp. 9–10, Table III · p. 9
  • On the confusion-matrix test, Dataset III predicts check, call, fold, and bet better than Dataset VI, while Dataset VI predicts raise better.Both models tend to fold more than other actions; the authors suggest this may reflect avoiding investment when the choice is unclear.Source: D. Ablation Results · p. 10, Figure 8 · p. 10
  • As data quality decreases, predicted investment means rise and distributions diversify; Dataset VI’s mean investment reaches about 10 big blinds.The authors interpret higher-quality players’ strategies as more conservative, while all models concentrate on relatively small investments.Source: D. Ablation Results · p. 10, Figure 9 · p. 10
  • Dataset III’s model is more likely to fold across stages, while Dataset VI’s is more likely to raise and bet; both tend to call in later stages.The authors characterize Dataset III’s pattern as conservative and Dataset VI’s as aggressive.Source: D. Ablation Results · p. 10, Figure 10 · p. 10

E. Multi-Player Evaluation

Across player counts, PokerGPT’s win rate declines as games grow, while response time rises roughly linearly; its action scores also shift toward more cautious play against larger tables.

E. Multi-Player Evaluation

  • Across 1,000 games per player count, win rate falls sharply from 2–5 players, then more slowly at 6–9 and fluctuates slightly at 10–15, while response time rises roughly linearly.Fig. 11(a) presents win-rate and response-speed trends, with dashed lines showing standard deviations.Source: E. Multi-Player Evaluation · p. 11, Figure 11 · p. 11
  • Call scores increase with player count, while bet and raise scores decrease, indicating more conservative choices against larger tables.Source: E. Multi-Player Evaluation · p. 11
  • Check scores rise through six-player games and then settle into a range; the authors say bait tactics become less effective above ten players.Source: E. Multi-Player Evaluation · p. 11
  • Fold scores increase substantially above six players, which the authors attribute to harder hand-strength forecasting and the possibility of shortening games.Source: E. Multi-Player Evaluation · p. 11
  • All-in scores are very low and tend toward zero with more opponents, consistent with the model’s more cautious play as table size grows.The paper describes all-in as a special case of bet or raise.Source: E. Multi-Player Evaluation · p. 11

F. Analysis of Interaction

PokerGPT’s interaction example shows responses adjusted to user requests and game information, but also difficulty with card guessing and winning probabilities. The conclusion presents it as a competitive, efficient solver for arbitrary player counts and identifies explainability as future work.

F. Analysis of Interaction

  • PokerGPT changes its recommendation from call to raise when asked to be more aggressive, interprets card strengths and opponent actions, but struggles with card guessing and winning probabilities.The authors attribute probability difficulty to omitting probabilities from prompts and LLMs’ limited mathematical proficiency.Source: F. Analysis of Interaction · p. 11

VI. CONCLUSION AND FUTURE WORK

  • PokerGPT combines prompt engineering on real-game data with fine-tuning a lightweight pretrained LLM using RLHF.Source: §VI · p. 11
  • The authors report an end-to-end solver for arbitrary player counts with convenient interaction, competitive win rate, and advantages in training cost and inference speed.Source: §VI · p. 12
  • Future work would prepare human-written action explanations, CFR-produced action probabilities, and multiple candidate decisions to improve explainability.Source: §VI · p. 12
Loading 2401.06781v1…