PaperlayerView exact revision on arXiv

Source-linked AI summary

Code World Models for General Game Playing

Wolfgang Lehrach, Daniel Hennes, Miguel Lazaro-Gredilla, Xinghua Lou, Carter Wendelken, Zun Li, Antoine Dedieu, Jordi Grau-Moya, Marc Lanctot, Atil Iscen, John Schultz, Marcus Chiam, Ian Gemp, Piotr Zielinski, Satinder Singh, Kevin P. Murphy

arXiv:2510.04542v1cs.AI

TL;DR

Direct LLM move generation is a common approach to game playing, but generalist models often lack deep tactical foresight and may not work well on novel games. This paper synthesizes executable game models from rules and trajectories, then uses planning; it reports outperforming a thinking LLM across various two-player games, including novel ones.

  • Problem

    Direct play by generalist LLMs often lacks deep tactical foresight and may not work well on games outside their training set.

    Source: §1 · p. 1

  • Method

    The paper uses an LLM to synthesize a Python Code World Model from game rules and trajectories, then applies search with optional value functions and hidden-state inference.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    The approach outperforms a state-of-the-art thinking LLM across various two-player games, including novel games created to avoid training-set contamination.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework extends code-world-model planning to partially observed and stochastic games, including closed-deck state estimation.

    Source: §1 · p. 2, §4.4 · p. 5, §6 · p. 9

    The method struggles to learn Gin rummy's intricate rules, especially in the challenging closed-deck setting.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) reasoning abilities are increasingly being applied to classical board and card games, but the dominant approach -- involving prompting for direct move generation -- has significant drawbacks. It relies on the model's implicit fragile pattern-matching capabilities, leading to frequent illegal moves and strategically shallow play. Here we introduce an alternative approach: We use the LLM to translate natural language rules and game trajectories into a formal, executable world model represented as Python code. This generated model -- comprising functions for state transition, legal move enumeration, and termination checks -- serves as a verifiable simulation engine for high-performance planning algorithms like Monte Carlo tree search (MCTS). In addition, we prompt the LLM to generate heuristic value functions (to make MCTS more efficient), and inference functions (to estimate hidden states in imperfect information games). Our method offers three distinct advantages compared to directly using the LLM as a policy: (1) Verifiability: The generated CWM serves as a formal specification of the game's rules, allowing planners to algorithmically enumerate valid actions and avoid illegal moves, contingent on the correctness of the synthesized model; (2) Strategic Depth: We combine LLM semantic understanding with the deep search power of classical planners; and (3) Generalization: We direct the LLM to focus on the meta-task of data-to-code translation, enabling it to adapt to new games more easily. We evaluate our agent on 10 different games, of which 4 are novel and created for this paper. 5 of the games are fully observed (perfect information), and 5 are partially observed (imperfect information). We find that our method outperforms or matches Gemini 2.5 Pro in 9 out of the 10 considered games.

1 INTRODUCTION

The paper replaces direct LLM move selection with code-based world-model induction and classical planning. It reports stronger performance than a state-of-the-art thinking LLM across two-player games, including novel games.

1 INTRODUCTION

  • The authors use LLMs as induction engines, converting textual rules and a small amount of game-trajectory data into plausible Python world models through iterative refinement.For partially observable games, they also synthesize inference functions that map observations to plausible latent histories, with the game rules and API acting as structural regularizers.Source: §1 · p. 2
  • The approach outperforms a state-of-the-art thinking LLM across various two-player games, including novel games created to avoid training-set contamination.The paper also addresses partially observed and stochastic worlds, unlike prior work that generally assumes fully observed deterministic environments or post-hoc observability.Source: §1 · p. 2
  • The method additionally synthesizes heuristic value functions, which the authors report significantly improve search-based policies such as MCTS and Information Set MCTS.The paper's three stated differences are support for partially observed and stochastic worlds, value-function synthesis, and evaluation against a thinking LLM on novel games.Source: §1 · p. 2

2 BACKGROUND

The background frames multiplayer games as histories of actions with player or chance decisions and rewards. The agent learns an executable game model from rules and example play, then uses search to select moves.

2 BACKGROUND

  • An extensive-form game represents play through histories of actions, with legal actions and a player or chance node at each history, ending in terminal histories.Chance follows a fixed stochastic policy, while each player seeks to maximize cumulative reward in a setting where outcomes also depend on other agents' choices.Source: §2 · p. 2
  • The agent does not know the environment model; it learns a Code World Model from a textual game description and example gameplay data.The game environments provide ground-truth rules for legal actions, transitions, rewards, and observations, but those rules are not given to the agent.Source: §2 · p. 2
  • The agent selects moves with MCTS in perfect-information games and Information Set MCTS in imperfect-information games, optionally adding a value function and hidden-state estimator.The paper also tried learning a policy with PPO on the partially observed model.Source: §2 · p. 2

3 RELATED WORK

Related work applies LLMs directly as game policies, synthesizes models for planning, or constructs game-theoretic representations. This paper positions its contribution around strategic multiagent play, value functions, and inference for imperfect information.

3 RELATED WORK

  • Recent LLM game-agent evaluations include Kaggle Game Arena and assess generality using novel games that are out-of-distribution for the LLM.Unlike direct-policy approaches, this paper uses the LLM to generate a Code World Model and then applies standard solvers such as (IS)MCTS or PPO.Source: §3 · p. 3
  • WorldCoder uses LLM code synthesis and Thompson sampling to refine model hypotheses, while GIF-MCTS uses MCTS for decision-making; this work adds strategic multiagent settings, value functions, and inference functions.WorldCoder uses ReAct-style decision-making after learning its model.Source: §3 · p. 3
  • POMDP Coder learns a partially observed model but assumes hidden states become observable at trajectory end; this paper also studies closed-deck games where they are never observed.POMDP Coder uses a determinized belief-space planner, whereas this work uses ISMCTS or PPO.Source: §3 · p. 3
  • Other work builds game-theoretic representations from scenarios, dialogues, or natural-language rules; one approach constructs extensive-form trees from rules alone and evaluates games with at most 25 decision nodes.The paper contrasts this with using observed trajectories and notes that code-world models can encode repeated transitions more efficiently in some cases.Source: §3 · p. 3

4 METHODS

The method synthesizes executable Code World Models from game rules and offline trajectories, then uses planning and learned functions to support play, including in imperfect-information games. For closed-deck training, it combines model and inference components and refines them using observable evidence.

4 METHODS

  • The agent gathers random-play trajectories, synthesizes a CWM from game rules and trajectories, then uses MCTS or ISMCTS to choose moves.The CWM is learned up front from offline trajectories for efficiency.Source: §4 · p. 3

4.1 SYNTHESIZING THE CODE WORLD MODEL

  • A CWM provides deterministic game logic for transitions, legal actions, observations, chance outcomes, and rewards; trajectory-based unit tests guide iterative refinement.Tests compare predicted states, observations, rewards, and action legality against the offline trajectories and check for execution errors.Source: §4.1 · p. 4

4.2 SYNTHESIZING INFERENCE FUNCTIONS FOR IIGS

  • For imperfect-information play, synthesized inference functions approximate hidden-state beliefs from the player’s actions and observations so ISMCTS can sample states.Hidden-history inference can yield valid CWM states and posterior-support samples after passing tests, but novel observations can reduce test accuracy; direct hidden-state inference cannot guarantee posterior support or valid states.Source: §4.2 · pp. 4–5

4.3 SYNTHESIZING VALUE FUNCTIONS

  • The LLM generates deterministic value functions for leaf states; multiple candidates are selected through a tournament because no ground truth is available for refinement.These functions may speed up and improve value estimation relative to random rollouts.Source: §4.3 · p. 5

4.4 OPEN DECK VS CLOSED DECK DURING TRAINING

  • Open-deck synthesis assumes offline trajectories include hidden states, an assumption that may be unrealistic in some settings.The paper identifies cooperative training, game design, and fully annotated demonstrations as scenarios where such access may be available.Source: §4.4 · p. 5
  • Closed-deck synthesis uses a hidden-history inference function and CWM as encoder and decoder, refining them with observable checks and execution-error tests.Game rules and the OpenSpiel API act as regularizers; valid inferred histories provide a lower bound on CWM likelihood.Source: §4.4 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate CWM-(IS)MCTS on five perfect-information and five imperfect-information games, including four out-of-distribution games created for the study. Synthesis accuracy and arena play are assessed using models learned from offline trajectories.

5 EXPERIMENTS

  • The evaluation covers five perfect- and five imperfect-information games, including four out-of-distribution games created for these experiments.The agent synthesizes models from open- or closed-deck trajectories, then uses MCTS or ISMCTS; comparisons include Random, ground-truth (IS)MCTS, and Gemini 2.5Pro.Source: §5 · p. 6

5.1.1 PERFECT INFORMATION GAMES

  • Perfect-information games yield correct CWMs with high test accuracy, and both conversation and tree-search refinement work well.Tree search is used thereafter because its ability to backtrack is described as more resilient in harder settings.Source: §5.1.1 · p. 7

5.1.2 IMPERFECT INFORMATION GAMES, OPEN DECK

  • 84% training and 79% test transition accuracy for Gin rummy accompany about 52% hidden-history inference accuracy; three of five games have good inference results.The authors hypothesize that Gin rummy’s multi-stage scoring contributes to its difficulty, and report that hidden-history inference is harder than learning transitions from fully observed trajectories.Source: §5.1.2 · p. 7, Table 1 · p. 7, Figure 1 · p. 7

5.1.3 IMPERFECT INFORMATION GAMES, CLOSED DECK

  • Closed-deck synthesis quality degrades relative to open deck, although subsequent gameplay performance does not degrade significantly.Closed deck supplies no hidden information, even post hoc, in the offline trajectories.Source: §5.1.3 · p. 7, Table 2 · p. 8

5.2 ARENA: GAME PLAY PERFORMANCE

  • Arena results average 100 matches after five stochastic syntheses, with bad samples automatically rejected and a random CWM selected for each match.The agents receive the same game rules and five offline trajectories; (IS)MCTS runs 1,000 simulations per move.Source: §5 · p. 6, §5.2 · p. 8

5.2.1 PERFECT INFORMATION GAMES

  • CWM-MCTS beats Gemini 2.5Pro in every perfect-information game, while performing similarly to ground-truth MCTS without a clear winner between them.The comparison uses win/loss/draw rates; failures to provide a valid action within the time limit count as forfeits.Source: Figure 2 · p. 8, §5.2.1 · p. 8

5.2.2 IMPERFECT INFORMATION GAMES, OPEN DECK

  • Except for Hand of war, open-deck CWM-ISMCTS beats or matches Gemini 2.5Pro; its average Leduc poker performance is higher but highly variable.Gemini 2.5Pro is described as a very weak Gin rummy player, and the Bargaining value function significantly improves play as player 1.Source: §5.2.2 · p. 8, Figure 3 · p. 9

5.2.3 IMPERFECT INFORMATION GAMES, CLOSED DECK

  • Closed-deck CWM-ISMCTS beats or matches Gemini 2.5Pro despite worse results than open deck, with high variance in Leduc poker.The setting provides no hidden-state information or other players’ actions in offline trajectories; the proposed explanation for Hand of war’s improvement is simpler synthesized state spaces.Source: Figure 4 · p. 9, §5.2.3 · p. 9

6 DISCUSSION

The discussion summarizes extensions to the CWM framework and reports its broad comparison with LLM policies. It also identifies difficulty learning Gin rummy rules, particularly in the closed-deck setting.

6 DISCUSSION

  • The framework adds two-player games, synthesized value functions, inference as code, and code-based autoencoder learning for closed-deck games.The authors report superiority to LLM policies across multiple perfect- and imperfect-information games, including newly created games.Source: §6 · p. 9
  • The method struggles to learn Gin rummy rules, especially in the challenging closed-deck setting.The authors identify active and online world-model learning and support for open-world games with free-form text or visual interfaces as future directions.Source: §6 · p. 9

A INFORMATION ON THE GAMES

  • Table 3 summarizes game characteristics including novelty, observability, payoff type, action count, and observation and information-set dimensions.The information-set dimension corresponds to the game's hidden state.Source: Table 3 · p. 13, A INFORMATION ON THE GAMES · p. 13

B INFORMATION SET MONTE CARLO TREE SEARCH

ISMCTS plans in imperfect-information games by sampling possible ground-truth states and aggregating statistics across information states available to the current player. Its tree search represents indistinguishable histories together.

B INFORMATION SET MONTE CARLO TREE SEARCH

  • ISMCTS starts simulations from a distribution of possible ground-truth states and aggregates statistics across information states for the current player.These statistics include state-action value estimates and simulation counts.Source: B INFORMATION SET MONTE CARLO TREE… · p. 14
  • In the three-card poker example, the player holding the King faces two possible opponent cards, the Queen or Jack.An iteration samples a ground-truth state, expands the tree until a new node, then uses a random rollout policy to reach a terminal state.Source: B INFORMATION SET MONTE CARLO TREE… · p. 14
  • The search tree branches over ground-truth histories, while each information set groups histories that appear identical to the player.Statistics are aggregated at the information-set level.Source: Figure 5 · p. 14

C ADDITIONAL EXPERIMENTAL RESULTS

This section introduces additional experimental results.

C ADDITIONAL EXPERIMENTAL RESULTS

C.1.1 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS

The additional results compare tree-search and conversational refinement for perfect-information games, and also include a table on imperfect-information games with hidden-state inference.

C.1.1 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS

  • Tree-search refinement has higher accuracy and uses fewer LLM calls than conversational refinement.The comparison is between Tables 4 and 5.Source: §C.1.1 · p. 15, Table 4 · p. 15, Table 5 · p. 15
  • Table 4 reports perfect-information games refined via tree search.Source: Table 4 · p. 15
  • Table 5 reports perfect-information games refined via conversation.Source: Table 5 · p. 15
  • Table 6 covers imperfect-information games with tree-search refinement and hidden-state inference.Source: Table 6 · p. 15

C.1.2 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS VS NUMBER OF LLM CALLS

Figures 6–8 track learned transition or inference accuracy as LLM calls increase, across perfect-information and imperfect-information settings.

C.1.2 ACCURACY OF LEARNED TRANSITION AND INFERENCE FUNCTIONS VS NUMBER OF LLM CALLS

  • Figure 6 tracks the best generated CWM’s transition accuracy against the number of LLM calls for perfect-information games with tree-search refinement.Source: Figure 6 · p. 16
  • Figure 7 tracks transition and inference accuracy against LLM calls for imperfect-information games using tree-search refinement and hidden-state inference.Source: Figure 7 · p. 16
  • Figure 8 tracks inference accuracy against LLM calls for imperfect-information games using tree-search refinement with a closed deck.Source: Figure 8 · p. 16

C.1.3 TREE SEARCH SETTINGS

The tree-search settings specify heuristic weighting, retry count, and thresholds for including tests and expanding nodes.

C.1.3 TREE SEARCH SETTINGS

  • The heuristic weight is 5.0, with the heuristic value setting the Beta-prior parameters as α = 1 + C × h and β = 1 + (1 − C) × h.Here, h is the average pass rate of unit tests; higher heuristic weight means more exploitation.Source: §C.1.3 · p. 17
  • Tree search uses 500 retries, while initial synthesis includes five tests of each type and refinement includes one failed test of each type.Source: §C.1.3 · p. 17
  • Nodes are considered for expansion at initialization at a minimum heuristic value of 0.01, and later at a minimum heuristic-value gain of 0.01.Source: §C.1.3 · p. 17

C.2 DETAILED PER-GAME ARENA RESULTS

The detailed arena results report outcomes or payoffs by game and player order, with separate settings for hidden-information inference and forfeit rates.

C.2 DETAILED PER-GAME ARENA RESULTS

  • For win/loss/draw games, results separate the three outcomes; arbitrary-payoff games report each player’s payoff, and rows distinguish whether the agent plays first or second.Source: §C.2 · p. 18

C.2.1 PERFECT INFORMATION GAMES

  • Table 7 reports win rates for perfect-information games against multiple opponents, with rows for the agent playing first or second.Source: Table 7 · p. 18

C.2.2 HIDDEN HISTORY INFERENCE

  • Tables 8 and 9 report payoffs and win rates, respectively, against multiple opponents using tree-search CWM refinement with hidden-history inference.Source: Table 8 · p. 19, Table 9 · p. 19

C.2.3 HIDDEN STATE INFERENCE

  • Tables 10 and 11 report payoffs and win rates, respectively, using tree-search refinement with hidden-state inference against multiple opponents.Source: Table 10 · p. 20, Table 11 · p. 20

C.2.4 HIDDEN HISTORY INFERENCE WITH CLOSED DECK LEARNING

  • Tables 12 and 13 report payoffs and win rates, respectively, using tree-search refinement with closed-deck learning against multiple opponents.Source: Table 12 · p. 21, Table 13 · p. 21

C.3 FORFEIT RATES FOR NON-TERNARY-OUTCOME GAMES

C.4 VALUE FUNCTION ABLATIONS

The value-function ablation compares planning with and without synthesized values and finds improvements only in Gen. tic-tac-toe and Bargaining.

C.4 VALUE FUNCTION ABLATIONS

  • The ablation evaluates agents with different value functions, or no value function, by having them compete on the synthesized CWM.Source: §C.4 · p. 23
  • Synthesized value functions improved results only in Gen. tic-tac-toe and Bargaining, so the agent used them only in those games.The authors say the choice can be assessed before online play by local play with and without a value function on the synthetic CWM.Source: §C.4 · p. 23
  • Figure 9 compares planning without a value function in Gen. tic-tac-toe and Bargaining; its right and left columns show use versus non-use of synthesized values.Source: Figure 9 · p. 23, §C.4 · p. 23

D.1 TRAINING A PPO AGENT ON TOP OF A CWM

Because online planning with a CWM can be slow, the authors train a reactive PPO policy inside the learned CWM. The section describes its observation encoding, network architecture, training, and evaluation-agent selection.

D.1 TRAINING A PPO AGENT ON TOP OF A CWM

  • Online planning with the learned CWM can be slow, motivating a reactive policy trained with PPO.The PPO-CWM agent is trained inside the learned CWM against an opponent that uniformly selects a legal action.Source: §D.1 · p. 24
  • JSON observations are mapped to fixed-size 1D tensors using a programmatically generated mapping prompted from the LLM.The requested observation-to-tensor and tensor-to-observation functions are described as a bijective pair.Source: §D.1 · p. 24
  • For perfect-information games, the actor and critic share two 256-unit tanh layers, with legal-action masking on actor logits and a scalar critic output.The text notes transformers as an alternative but says they would be slower to train.Source: §D.1 · p. 24
  • For imperfect-information games, a recurrent network processes the observation history before actor and critic heads.A 256-unit tanh layer transforms each observation, and the RNN output is concatenated with the original observation before a final 256-unit tanh layer.Source: §D.1 · p. 24
  • PPO-CWM trains for 10M agent steps in the learned CWM, using filtered single-agent observation, action, and reward sequences to compute advantages and the PPO loss.The filtered sequences are extracted from two-player trajectories.Source: §D.1 · p. 24
  • For each game, five seeded agents are trained, and the one with the highest win rate against Random is selected for Arena evaluation and comparison with CWM MCTS.The selected agent is evaluated using the Arena procedure described in Sec. 5.2.Source: §D.1 · p. 24

D.2 RESULTS

PPO-CWM outperforms or matches Gemini 2.5 Pro across the reported games, and beats Random in every game. Against CWM-MCTS, its relative performance depends on the game and information setting.

D.2 RESULTS

  • PPO-CWM outperforms the Random agent in all games.Source: §D.2 · p. 25
  • PPO-CWM outperforms or matches Gemini in all games: it wins in three perfect-information games, has mixed results in two, wins in Bargaining and Quadranto, and ties in Hand of War and Leduc Poker.The imperfect-information outcomes apply to both open-deck and closed-deck settings.Source: §D.2 · p. 25
  • Against CWM-MCTS, PPO-CWM loses in perfect-information games except Generalized Tic-Tac-Toe as Player 0; in imperfect-information games, it wins two and loses two.The imperfect-information wins are in Hand of War and Bargaining, and the losses are in Leduc Poker and Quadranto.Source: §D.2 · p. 25

D.2.1 GAMES WITH PERFECT INFORMATION

  • Table 18 reports PPO-CWM win rates for perfect-information games against multiple opponents, with rows distinguishing whether the agent goes first or second.Source: Table 18 · p. 25

D.2.2 HIDDEN HISTORY INFERENCE

  • Tables 19 and 20 report win rates and payoffs, respectively, for hidden-history inference against multiple opponents, with first- and second-player rows.The imperfect-game CWMs in this results section were trained on 100 offline trajectories.Source: §D.2 · p. 25, Table 19 · p. 26, Table 20 · p. 26

D.2.3 HIDDEN STATE INFERENCE

  • Tables 21 and 22 report win rates and payoffs, respectively, for hidden-state inference against multiple opponents, with first- and second-player rows.Source: Table 21 · p. 26, Table 22 · p. 26

D.2.4 HIDDEN HISTORY INFERENCE WITH CLOSED DECK LEARNING

  • Tables 23 and 24 report closed-deck win rates and payoffs against multiple opponents, with rows distinguishing whether PPO-CWM goes first or second.Source: Table 23 · p. 27, Table 24 · p. 27

E AUTOMATIC REJECTION OF BAD CWM SAMPLES

This section describes procedures for generating and refining CWMs, including competition-based rejection of weak models for imperfect-information games, alongside game rules, action constraints, and related checks and functions.

E AUTOMATIC REJECTION OF BAD CWM SAMPLES

  • Five CWM-ISMCTS agents compete, and samples scoring over 10% below the best agent’s observed utility range are rejected for imperfect-information games.The competitions use CWMs as stand-in hosts because ground-truth games are unavailable: 2 hosts × 5 Player 0 agents × 5 Player 1 agents yields 50 matches, each repeated 50 times; execution failures or illegal actions make both players lose.Source: E AUTOMATIC REJECTION OF BAD CWM SAMPLES · p. 28

F SKETCH OF INFORMATION FLOW OF EACH AGENT

  • Perfect-information agents use a generated or ground-truth CWM and value function with MCTS, while imperfect-information agents additionally use an inference function with ISMCTS.The direct LLM policy sketch instead prompts for the next action using the rules, example trajectories, and action-observation history.Source: F SKETCH OF INFORMATION FLOW OF EACH… · p. 28

G.1 TREE SEARCH

  • The tree-search prompt asks the LLM to implement game functions from a signature and unit tests, refine prior code on later iterations, and generate multiple versions.The signature and tests vary with information setting, deck style, and inference method.Source: §G.1 · p. 29

G.2 PERFECT INFORMATION GAMES

  • For perfect-information games, the function API covers transitions, players, rewards, legal actions, and observations, while unit tests check API behavior against example transitions.When a test fails, the LLM receives the Python error and a detailed assertion difference.Source: §G.2 · p. 30

G.3 HIDDEN HISTORY INFERENCE FUNCTION SYNTHESIS, OPEN DECK

  • Open-deck hidden-history inference samples possible full action histories from one player’s observation-action sequence and must reproduce that player’s observations.Tests replay sampled actions from the initial state and check the observations and actions against the supplied history.Source: §G.3 · pp. 30–31

G.4 HIDDEN STATE INFERENCE FUNCTION SYNTHESIS

  • Hidden-state inference samples a reachable state consistent with the player’s observation-action history and tests that the resulting state reproduces the latest observation.Source: §G.4 · p. 31

G.5 HIDDEN HISTORY INFERENCE FUNCTION SYNTHESIS, CLOSED DECK

  • Closed-deck hidden-history inference adds a terminal-state flag so tests can cover the full game and terminal reward; random-play tests check playability because state-transition tests are unavailable.The authors assume rewards are terminal for simplicity, though they note this can be relaxed.Source: §G.5 · pp. 31–32
  • The generated-game test runs random valid moves to check termination, assuming a static, deterministic initial state before chance nodes occur.The initial state could also be synthesized by the LLM.Source: §G.5 · p. 32

G.6 RESAMPLING THE STATE AT GAME PLAYING TIME FOR IMPERFECT INFORMATION GAMES

  • At game time, the system allows up to 10 resampling tries to match observations and falls back to a uniformly sampled legal action if ISMCTS fails.The stated failure example is poor understanding of the CWM’s termination criteria.Source: §G.6 · p. 33

G.7 VALUE FUNCTION SYNTHESIS

  • The synthesized value function returns true rewards at terminal states and float estimates of a player’s winning potential during ongoing play.The prompt asks for informative heuristics rather than zeros outside terminal states and includes tests for float outputs and terminal reward matching.Source: §G.7 · pp. 33–34

H.1 BACKGAMMON

  • Backgammon combines dice-driven checker movement with mandatory legal moves, bar re-entry, hitting and blocking, and constrained bearing off; the first player to bear off all 15 checkers wins.Bearing off is allowed only after all checkers are in the home board and none remain on the bar; higher rolls have a specified bearing-off rule.Source: §H.1 · pp. 35–37

H.2 CONNECT FOUR

  • Connect Four uses a 6-by-7 grid where players drop marks into columns, aiming to form four in a row horizontally, vertically, or diagonally.The game ends on a win or when all 42 squares are filled without a winner.Source: §H.2 · pp. 37–38

H.3 TIC-TAC-TOE

  • Tic-Tac-Toe uses a 3-by-3 grid, with alternating turns to place marks in empty squares; three in a row wins, and a full grid without a win draws.Source: §H.3 · p. 38

H.4 GEN. TIC-TAC-TOE

  • Generalized Tic-Tac-Toe is specified as a two-player 6-by-6 variant in which the first player to form an unbroken line of four marks wins.A full board without a winning line is a draw.Source: §H.4 · pp. 38–39

H.5 GEN. CHESS

  • Generalized chess is a two-player game where players win by capturing a target piece, using piece-specific moves that differ from standard chess.Moves must stay on the board and cannot land on the moving player’s own pieces; landing on an opponent’s piece captures it.Source: §H.5 · p. 39
  • Generalized chess variant army5x5a uses a 5x5 board, and capturing the general wins the game.Moves use row-column coordinates such as A2_to_C2, and PASS denotes passing a turn.Source: §H.5 · p. 39

H.6 BARGAINING

  • Bargaining is an iterative exchange of offers shaped by information asymmetry, perceived value, and expected compromise, with norms varying by context and power dynamics.Common strategies include setting limits, anchoring opening offers, making concessions incrementally, and listening actively.Source: §H.6 · pp. 40–41

H.7 LEDUC POKER

  • In Leduc Poker, raises are fixed at 2 units in round 1 and 4 in round 2, with at most two raises per round; Player 1 acts first in both rounds.The game has two players and uses a six-card deck of two Jacks, two Queens, and two Kings.Source: §H.7 · p. 41

H.8 GIN RUMMY

  • In Gin Rummy, players form melds to reduce deadwood, and a player may knock when deadwood is at or below the round's threshold; gin means no deadwood.A draw-phase turn permits drawing from the stock or taking the upcard; a discard-phase turn permits discarding a card or knocking, and discarding requires a prior draw.Source: §H.8 · pp. 42–43
  • On Gin Rummy's first turn, the non-dealer may take the upcard or pass; if both players pass, the non-dealer must draw from the stock.After the standard draw/discard phases, knocking ends the round and begins hand declaration, opponent layoff, and deadwood scoring.Source: §H.8 · pp. 43–44
  • In Gin Rummy, going gin scores the opponent’s full deadwood count plus a typically 25-point bonus, and the opponent cannot lay off cards.The example gives a 42-point deadwood count plus a 25-point bonus for 67 points.Source: §H.8 · p. 44
  • In Gin Rummy, if the stock is down to its last two cards after the third-to-last draw is discarded without a knock, the hand is a draw with no points awarded.The round is terminal, and the player must pass to acknowledge the result and await the next hand.Source: §H.8 · p. 45

H.9 QUADRANTO

  • In this pursuit game, each player observes their own location and the opponent’s quadrant, not exact location; landing on the opponent’s location catches them and ends the game.Players can move in four cardinal directions or stay. If they make 20 total moves without a catch, the game ends in a draw and both receive 0 points.Source: §H.9 · p. 45

H.10 HAND OF WAR

  • In Hand of War, players choose cards simultaneously; the higher card wins both, while ties trigger a repeated showdown with face-down cards and a selected battle card.Players replenish their hands to three cards after battles and showdowns.Source: §H.10 · p. 46
  • Hand of War ends when one player holds all 16 cards or a required draw-pile action cannot be completed; the larger win pile wins, and equal piles draw.The win pile is not reshuffled into the draw pile during play.Source: §H.10 · p. 46

I SAMPLE SYNTHESIZED CWMS AND INFERENCE FUNCTIONS

This appendix presents sample synthesized game models after tree-search refinement, including tic-tac-toe and open- and closed-deck bargaining implementations. The bargaining code covers hidden-action history reconstruction or resampling, player-value estimation and heuristics, and game logic for chance scenarios, agreements, turn limits, and rewards.

I SAMPLE SYNTHESIZED CWMS AND INFERENCE FUNCTIONS

  • The appendix’s samples include refined CWMs for perfect- and imperfect-information games, with open- and closed-deck inference for the latter where available.Inference and value functions are included when they exist.Source: I SAMPLE SYNTHESIZED CWMS AND… · p. 47

I.1 TIC-TAC-TOE (PERFECT INFORMATION)

  • The tic-tac-toe model implements winner and draw checks, state transitions, legal moves, terminal rewards, and identical full-state observations for both players.Legal actions correspond to empty board squares, and the game ends on a win or a full board.Source: §I.1 · pp. 47–48

I.2 BARGAINING (IMPERFECT INFORMATION, OPEN DECK)

  • The open-deck bargaining model initializes pool and player values through chance, tracks offers and agreements, and forces an agreement at the 10-turn limit.The turn-limit agreement receives zero reward for both players.Source: §I.2 · pp. 49, 51
  • Open-deck inference samples the opponent’s private item values and reconstructs a possible action sequence from offers in the player’s observation and action history.The code gathers known offers, orders them chronologically, and combines them with a reconstructed chance action.Source: §I.2 · pp. 52–53
  • The open-deck heuristic starts from half the player’s total pool value, compares an opponent’s offer with that baseline, and discounts value as turns run out.At the turn limit it returns zero; otherwise, it applies a square-root time-pressure factor.Source: §I.2 · pp. 53–54

I.3 BARGAINING (IMPERFECT INFORMATION, CLOSED DECK)

  • The closed-deck model creates 20 reproducible chance scenarios with item pools and private valuations for both players.A random generator seeded with 0 selects pool quantities and player values from predefined ranges.Source: §I.3 · p. 55
  • Closed-deck transitions make agreement terminal, switch players after offers, and return zero rewards when play reaches the turn limit.On agreement, the offerer receives the proposed bundle and the accepter receives the remainder of the pool.Source: §I.3 · pp. 55–56
  • The inference function reconstructs interleaved actions from a player's observations and samples histories using fixed opponent values for reproducibility.At terminal states, it infers whether the opponent accepted the player's last offer or made a counter-offer that the player implicitly accepted.Source: §I.3 · pp. 56–57
  • For terminal states, the value function returns exact rewards, with zero rewards when play ends at the turn limit or without agreement.For nonterminal states with no offers, it estimates value as half the player's total value for the pool.Source: §I.3 · p. 57
  • When it is the opponent's turn, the heuristic values the player's last offer as their current potential and aspiration level.If the player has not made an offer, aspiration defaults to the initial 50/50 baseline.Source: §I.3 · p. 58
  • When it is the player's turn, the heuristic averages the value of the opponent's offer to the player and the player's own aspiration.The offer's value is calculated from the items left in the pool after the opponent's bundle is allocated.Source: §I.3 · p. 58
Loading 2510.04542v1…