Source-linked AI summary
Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
Lucia Cipolina-Kun, Marianna Nezhurina, Jenia Jitsev
TL;DR
Evaluating LLM reasoning and planning beyond standard language benchmarks calls for controlled strategic settings. Game Reasoning Arena provides a modular framework for comparing models across games and backend conditions, and its analyses report game-dependent reasoning patterns.
Problem
The paper addresses how to evaluate LLM reasoning and planning beyond standard language benchmarks using strategic games as a controlled testbed.
Source: §1 · p. 1Method
Game Reasoning Arena wraps diverse strategic games and supports cross-provider evaluation, while categorizing model justifications and analyzing reasoning patterns over play.
Source: §1 · p. 1, §3 · p. 3, §5.3 · p. 7Results
Reasoning patterns varied by game: opponent-focused behavior appeared in tic tac toe and prisoners dilemma, positional emphasis in connect four, and rule-based regularity in matching pennies.
Source: §5.3.1 · p. 8Takeaways & Limitations
The framework offers researchers a structured platform to examine strategic capabilities by logging reasoning strings and illegal moves for studying failure modes and prompt designs.
Source: §3 · p. 4
Abstract
from arXiv · showhide
The Game Reasoning Arena library provides a framework for evaluating the decision making abilities of large language models (LLMs) through strategic board games implemented in Google OpenSpiel library. The framework enables systematic comparisons between LLM based agents and other agents (random, heuristic, reinforcement learning agents, etc.) in various game scenarios by wrapping multiple board and matrix games and supporting different agent types. It integrates API access to models via liteLLM, local model deployment via vLLM, and offers distributed execution through Ray. This paper summarises the library structure, key characteristics, and motivation of the repository, highlighting how it contributes to the empirical evaluation of the reasoning of LLM and game theoretic behaviour.
1 Introduction
Game Reasoning Arena provides a configurable framework for evaluating LLM decision-making in strategic games using OpenSpiel. Its modular organization supports adding games and agents through shared interfaces and registration.
1 Introduction
- The framework uses OpenSpiel to support configurable games, agents, and evaluation settings for strategic decision-making.The described games include Tic–Tac–Toe, Connect Four, Kuhn Poker, and matrix games.Source: §1 · p. 1
- Games are dynamically loaded through a registry that associates game names with OpenSpiel loaders and custom environment classes.Tic–Tac–Toe and Connect Four are examples of games with dedicated environment classes.Source: §1 · p. 2
- Agents include random, human, and LLM types, while simulation utilities run turns, validate moves, update rewards, and log outcomes.The simulation supports both turn-based and simultaneous games, with optional parallel execution.Source: §1 · p. 2
- Researchers can extend the library by implementing and registering new environment or agent classes.The architecture connects shared OpenSpiel wrappers, agent classes, model backends, and simulation utilities.Source: §1 · p. 2
2 Framework Design
The framework wraps OpenSpiel games in a common interface and connects environments, agents, and model-inference backends. This design supports evaluation across game types and model deployment settings.
2 Framework Design
- The framework adopts a multi-agent reinforcement-learning paradigm and provides a common interface for evaluating LLM reasoning in strategic games.Its design covers the game loop, environment, agent interface, and language-model inference backends.Source: §2 · p. 2
2.1 Game loop and environment
- OpenSpiel mediates game state, action validity, and transitions, while text prompts provide observations and legal actions for turn-based or simultaneous games.Some prompts also summarize previous moves.Source: §2.1 · p. 2
2.2 Agent interface and observation structure
- Agents return both a selected action and its reasoning, and the environment supplies player-indexed reward signals.The prompt requests reasoning before the action to avoid ex-post contamination.Source: §2.2 · pp. 2–3
2.3 Language model inference backends
- LiteLLM supports hosted providers, while vLLM runs models locally; a shared backend interface allows researchers to compare models and inference infrastructure.The backends offer different conditions: LiteLLM includes batching and caching, while vLLM supports privacy and offline evaluation but requires local weights and sufficient GPU memory.Source: §2.3 · p. 3
3 Motivation
The framework is motivated by evaluating LLM planning and reasoning in strategic settings, with varied games and agent matchups. It also supports cross-provider experiments, extensibility, and large-scale execution.
3 Motivation
- Strategic games provide settings with sequential decisions, imperfect information, and adversarial dynamics for testing planning and reasoning.The motivation frames these capabilities as relevant to reinforcement learning and game theory.Source: §3 · p. 3
- The library supports LLMs against random agents or humans, LLM-versus-LLM play, and self-play for comparative analyses.Source: §3 · p. 3
- The game suite spans perfect-information board games, hidden-information Kuhn Poker, and matrix games, covering varied strategic-reasoning settings.Source: §3 · p. 3
- Researchers can compare hosted and local models across providers and assess how inference speed, token limits, and temperature influence performance.Ray integration supports parallel simulation across CPUs, GPUs, or SLURM clusters for large-scale experiments.Source: §3 · p. 3
- The modular library is designed to accommodate new games and agents, including reinforcement-learning agents for comparison.Its Gymnasium- and RLlib-oriented design also makes it suitable for training agents in the provided environment.Source: §3 · pp. 3–4
- Logging model reasoning and illegal moves lets researchers study failure modes and prompt designs across strategic tasks.The framework is presented as a structured platform for examining LLM strategic capabilities.Source: §3 · p. 4
4 Prompting System Architecture
The prompting architecture turns game states into structured observations and supports game-specific customization, consistent output parsing, and backend formatting. Its examples and extension hooks illustrate how prompts are used across environments.
4 Prompting System Architecture
- The prompting system is designed for consistent, analyzable interactions with diverse board-game environments and is modular and extensible.Source: §4 · p. 4
4.1 Observation and Decision Cycle
- Each environment supplies a structured observation, and the backend response is parsed into an action and accompanying reasoning.The observation includes a state string, legal actions, and a constructed prompt.Source: §4.1 · p. 4
4.2 Hierarchical Prompt Construction
- Hierarchical prompt construction reuses base prompts and adds private observations, game history, and interpretable action labels for specialized games.The generic prompt includes the game name, player role, move number, board visualization, and legal actions.Source: §4.2 · pp. 4–5
4.3 Standardized Prompt Augmentation
- Standardized prompt augmentation adds a reasoning directive and JSON output schema to support consistent response parsing.Source: §4.3 · p. 5
Example: Prompt in Kuhn Poker
- The Kuhn Poker example gives the player’s private card, move number, betting history, pot, contribution, and available actions before requesting a reasoned choice in JSON.Source: Example: Prompt in Kuhn Poker · p. 5
4.4 Backend Integration and Parsing
- Backend integration formats prompts for raw-text or chat models and parses outputs while preserving generated reasoning for analysis.Source: §4.4 · p. 6
4.5 Extensibility for New Environments
- To add a game, developers can override the environment’s prompt-generation method while the framework handles subsequent formatting and parsing.Source: §4.5 · p. 6
5 Evaluation and Experimental Design
The evaluation measures LLM strategic competence and reasoning quality through game trajectories, while repeated simulations and statistical comparisons assess performance differences.
5 Evaluation and Experimental Design
- Evaluation metrics cover rewards, optimal decisions, rationale quality, and illegal or suboptimal moves.They include average and maximum cumulative reward, the proportion of moves matching an equilibrium or optimal policy, heuristic measures of reasoning length and coherence, and error rates.Source: §5 · p. 6
- Multiple simulations per configuration are summarized with means and standard errors.Source: §5 · p. 6
- Paired-sample tests and bootstrap confidence intervals assess differences across models, games, and back-ends.Source: §5 · p. 6
5.1 Experimental design
The experimental design supports controlled, reproducible studies across configurations and scales simulations through distributed execution.
5.1 Experimental design
- The framework supports varied game configurations, large-scale simulations, hyperparameter sweeps, and ablation studies.Source: §5.1 · p. 6
- Experiments use consistent random seeds to support reproducibility.Source: §5.1 · p. 6
- Ray distributes simulation workloads across hardware, while SLURM schedules experiments on high-performance computing clusters.Source: §5.1 · p. 6
5.2 Analysis tools
Analysis tools pair agent actions with LLM reasoning strings and use post-game processing to inspect reasoning patterns and flag problematic behavior.
5.2 Analysis tools
- Logged reasoning strings accompany agent actions, enabling qualitative analysis of decision-making and failure modes.Source: §5.2 · p. 6
- Post-game scripts summarize common reasoning categories and identify references to game-specific concepts.Source: §5.2 · p. 7
- The tools flag hallucinations and rule violations to inform prompt engineering and model selection.Source: §5.2 · p. 7
5.3 Results
The results use a reasoning-category framework to analyze gameplay logs across games and models. Across these views, LLM reasoning varies with game structure, turn, model size, and agent diversity.
5.3 Results
- The analysis classifies justifications as positional, opponent modeling, blocking, winning logic, heuristic-based, rule-based, or random/unjustified, then applies the framework to gameplay logs from multiple LLMs and games.The categories are associated with lexical cues to support structured analysis of decision making.Source: §5.3 · p. 7, Table 1 · p. 7
5.3.1 Temporal Adaptation Across Games (Radar Overview)
- Opponent-focused reasoning appears in tic tac toe and Prisoner’s Dilemma, positional emphasis in Connect Four, rule-based regularity in Matching Pennies, and more uncategorized reasoning in matrix PD.Figure 1 plots normalized reasoning-type frequencies across seven games for groq llama3 70b 8192.Source: Figure 1 · p. 8, §5.3.1 · p. 8
5.3.2 Aggregate Per-Game Profiles (Stacked Bars and By-Game Views)
- The 70B model shows richer variety, including opponent modeling in tic tac toe and Matching Pennies, while the 8B model relies on fewer categories and has more uncategorized segments.Kimi-K2-Instruct and GPT-4 show game-specific adaptation, whereas GPT-3.5-Turbo shows weaker specialization and more uncategorized reasoning.Source: §5.3.2 · pp. 8–9, Figure 2 · p. 9, Figure 3 · p. 9
5.3.3 Within-Game Evolution (Same Game, Different Turns)
- In tic tac toe, both models shift toward Blocking in the endgame, but their openings differ: one begins with Winning Logic then Opponent Modeling, and the other with Opponent Modeling.Turns are binned to calculate reasoning proportions; heatmaps show concentration and switching, while stacked bars show the mixture per bin.Source: Figure 7 · p. 11, §5.3.3 · p. 11, Figure 8 · p. 12
5.3.4 Per-Game Pie Charts (Detailed Breakdown Examples)
- The instant 8B model’s tic tac toe rationales comprise Blocking (50%), Winning Logic (25%), and Opponent Modeling (25%); the other pies show further model- and game-specific mixes.The 8B 8192 variant diversifies further, while the 70B model emphasizes Opponent Modeling in Kuhn Poker, with smaller rule-based and uncategorized shares.Source: Figure 11 · p. 13, §5.3.4 · p. 13, Figure 12 · p. 14
5.3.5 Entropy Across Agents Within a Game
- In tic tac toe, groq llama3 8b 8192 shows a midgame peak in cross-agent reasoning entropy followed by near-zero late-turn entropy; other agents remain near zero throughout.Higher entropy indicates a broader spread across reasoning types; zero indicates unanimity.Source: §5.3.5 · p. 14, Figure 14 · p. 15
5.3.6 Synthesis
- Across games and turns, reasoning aligns with game structure; larger models tend toward richer profiles, while diversity peaks midgame and collapses as terminal constraints dominate.The synthesis also reports that openings show planning or anticipation, midgames emphasize board control, and endgames converge to defensive blocking.Source: §5.3.6 · p. 15
6 Conclusion and Outlook
Game Reasoning Arena unifies discrete games and LLM inference backends in a benchmark that records actions and reasoning strings for systematic comparisons.
6 Conclusion and Outlook
- The framework unifies diverse discrete games, multiple LLM inference backends, and action-and-reasoning records for systematic comparisons across models, games, and inference modalities.These comparisons are intended to provide insights into how language models plan, adapt, and cooperate in strategic settings.Source: §6 · p. 16
- The experiments outline a broad suite of evaluation metrics and show that distributed execution facilitates large-scale benchmarking.The modular design is described as readily extensible to other game genres and agent types.Source: §6 · p. 16
- Future work will explore richer reasoning analyses, adaptive prompting strategies, and integration with reinforcement-learning training loops.The authors hope the benchmark stimulates research into LLM game-theoretic capabilities and human-AI interaction.Source: §6 · p. 16
7 Related Research
Related benchmarks evaluate LLMs in text-based, live, board, card, social-deduction, and video-game settings. Collectively, they point to strategic gameplay as a continuing challenge and motivate dynamic, interactive evaluation.
7 Related Research
- TextArena offers more than fifty text-based games, multiplayer scenarios, and an online platform with a real-time TrueSkill leaderboard.It is designed to train and evaluate agentic behavior in LLMs.Source: §7 · p. 17
- GameArena tests deductive and inductive reasoning through live computer games against human opponents, analyzing over 2,000 sessions across three games.Source: §7 · p. 17
- Topsakal et al. (2024) use an extensible simulator for board and grid games, while GameBench evaluates nine strategic games across imperfect information, stochasticity, and communication.GameBench evaluations report that even the best tested models often underperform random baselines and fail to approach human-level performance.Source: §7 · p. 17
- Board Game Bench pits LLMs head-to-head in board games and ranks them with a Bayesian Elo-like rating; lmgame-Bench builds unified benchmarks from real video games.Reinforcement-learning experiments on lmgame-Bench show in-domain and cross-domain generalization gains.Source: §7 · p. 17
- Together, these benchmarks indicate that strategic gameplay remains challenging for LLMs and highlight dynamic, multi-agent interactive environments for evaluating reasoning, planning, and social skills.Source: §7 · p. 17
A Appendix
The appendix describes a modular matrix-game architecture and traces how agents interact with the environment and how their behavior is logged.
A Appendix
- The episode logs LLM reasoning, action choices, and outcome metrics until the game reaches a terminal state.The described logging uses structured logs and real-time visualization tools.Source: A Appendix · p. 19
- The matrix-game architecture routes configuration from runner.py to simulation logic, which connects the game registry, environment classes, and policy manager.Structured logging records game outcomes and agent behavior for later analysis.Source: A Appendix · p. 19, Figure 15 · p. 20
- A single episode initializes the environment, creates player observations, and enters an action-computation loop before executing joint actions and updating rewards.The sequence diagram depicts this episode flow.Source: A Appendix · p. 19, Figure 16 · p. 21