LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents

1 INTRODUCTION

• Balancing difficulty is important because poorly calibrated challenges can cause frustration and disengagement, while a smooth difficulty curve supports player flow.The paper illustrates this risk with Elden Ring: Shadow of the Erdtree, whose excessive initial difficulty prompted a later reduction patch.Source: §1 · p. 1
• Human testing can map difficulty across levels, but it is tedious, complex, resource-intensive, and costly to repeat during iterative development.Existing AI testing also raises concerns about human fidelity, computational cost, and game-specific training.Source: §1 · p. 1
• The target is an off-the-shelf, no-fine-tuning framework whose difficulty assessments align closely with human judgments.The motivation follows LLMs’ potential as broadly deployable game-playing agents across different games.Source: §1 · p. 1
• The paper proposes a general LLM game-testing framework and evaluates it in Wordle and Slay the Spire.Its contribution is to assess whether game challenges match developers’ intended difficulty and to derive guidelines for using LLM agents in testing.Source: §1 · pp. 1–2

1.1 Measurement of Difficulty

• For game development, relative difficulty scaling across challenges matters more than an absolute difficulty value for any single challenge.Difficulty can be estimated from context-dependent statistics such as win rate, score, or completion time; no single metric is universally correct.Source: §1.1 · p. 2

2.1 AI for Game Testing

• Research has increasingly examined AI’s role in game design and player experience, including interaction patterns, player mental models, collaboration, and design-space exploration.These studies broaden AI’s role beyond building stronger game-playing agents.Source: §2.1 · p. 2
• Rule-based and deep-learning game AI established automated gameplay testing, but modern systems can be costly, game-specific, and difficult to interpret.These limitations leave uncertainty about how AI can increase game-development productivity.Source: §2.1 · p. 2

2.2 LLM Agent for Gaming

• LLM agents have been applied across conversational, board, card, and complex video games, with prompting and system structures sometimes producing human-like or heuristic-beating gameplay.Most of this work still targets optimal gameplay or adversarial play rather than other development tasks.Source: §2.2 · pp. 2–3
• LLMs have also been explored as NPCs, commentators, Game Masters, and procedural-content generators in game development.These applications show a broader design role, including text-based level generation and assistance for tabletop game sessions.Source: §2.2 · p. 3

2.3 LLM as Human-like Simulator

• Using LLM agents to assess game difficulty remains underexplored, motivating evaluation of how closely they can simulate human difficulty judgments.Related human-simulation research in economics, politics, healthcare, and social science provides context for testing this capability in games.Source: §2.2 · p. 3, §2.3 · p. 3

3 FRAMEWORK OVERVIEW AND GAME CHOICE

• The framework measures game difficulty from LLM gameplay performance using game I/O and instruction components.Game I/O converts environment states into summaries and LLM responses into executable actions.Source: §3 · p. 3
• The instruction component combines game rules, external game strategies, and prompting techniques such as Chain-of-Thought.These elements explain how the game works, guide tactics, and determine how knowledge is used during decisions.Source: §3 · p. 3

3.1 Choice of Game

• The framework selects games whose necessary information can be represented as text, excluding games that rely heavily on visual information.The study evaluates such games with models including GPT-3.5 and GPT-4.Source: §3.1 · p. 3
• The study prioritizes original, widely played games because their polished designs and real-world scale may make findings more applicable to game development.This criterion favors representative full-scale games rather than simplified experimental environments.Source: §3.1 · p. 3
• Human gameplay data is required so LLM difficulty assessments can be compared with human performance under shared conditions.The authors therefore select Wordle and Slay the Spire, which differ substantially in rule complexity and action space.Source: §3.1 · p. 3

4.2 Implementation

• The framework converts game state and rules into natural-language prompts, receives an LLM-generated action, and executes it through the game interface until success or failure.For Wordle, the local engine returns letter-position feedback, guess history, and the number of guesses to the agent.Source: Figure 1 · p. 4, §4.2 · p. 4

4.1 Background

• Wordle supports puzzle-level difficulty analysis because its independent text-described puzzles provide feedback and extensive human performance data.WordleBot records over 600,000 real-user plays, enabling comparison through average attempts or six-guess win rate.Source: §4.1 · p. 4

4.2 Implementation

• Zero-shot, CoT, and CoT+ prompting were evaluated with GPT-3.5 Turbo and GPT-4, with CoT+ adding expert Wordle strategies without fine-tuning.CoT requests step-by-step reasoning, while CoT+ combines that reasoning with externally sourced game strategies.Source: §4.2 · pp. 4–5
• The Wordle Solver achieved near-optimal gameplay at 3.55 guesses and a 100% win rate, but its optimized strategy correlated weakly and nonsignificantly with human difficulty.The solver minimizes information-theoretic uncertainty, unlike typical human play.Source: §4.2 · p. 5, §4.3 · p. 5

4.3 Experiment

• Wordle difficulty can be evaluated by comparing each agent’s average guesses across 529 puzzles with human average guesses using Pearson correlation.Agents attempted every puzzle, with 20 trials per puzzle for stochastic LLMs; the guess cap was increased from six to 12.Source: §4.3 · p. 5
• GPT-4 CoT/CoT+ agents had moderate-to-strong, statistically significant correlations with human puzzle difficulty despite requiring more guesses than humans.All reported correlations were significant at p<0.001, while LLM agents averaged 5.12–9.35 guesses versus the human average.Source: §4.3 · p. 5

5.1 Background

• Slay the Spire provides a challenging testbed because its text-representable mechanics require continuous state tracking, strategic planning, and card-synergy reasoning.Card order matters: applying vulnerability with Bash can increase subsequent Strike damage from 6 to 9, and the game contains more than 100 card types.Source: §5.1 · p. 7

5.2 Implementation

• The framework connects an LLM to the original game through BasicMod, CommunicationMod, and a Python interface that converts game states into text and actions into executable inputs.The setup treats the environment as a black box and supports repeated interaction until victory or player death.Source: §5.2 · p. 7
• The study evaluates Zero-Shot and Chain-of-Thought LLM agents against random and expert rule-based baselines in Slay the Spire, using game rules and real-time game information.The CoT agent analyzes the in-game situation, determines an optimal strategy, and evaluates each playable card before selecting one.Source: §5.2 · pp. 7–8

5.3 Experiment

• Using an average-strength deck, LLM agents won less than 5% of boss battles, while a stronger deck raised their average win rate to 65%.Experiments fixed the character as Ironclad and tested all Act 1 and Act 2 bosses.Source: §5.3 · p. 8
• GPT-4 with CoT reached a correlation of up to 0.871 with human-indicated enemy difficulty, showing that human-difficult enemies were also difficult for the agent.Difficulty was derived from human boss win rates, while agent difficulty used remaining HP after battles.Source: §5.3 · p. 8
• GPT-4 CoT produced the strongest correlations, followed by the expert AI, GPT-4 Zero-Shot, and GPT-3.5 CoT; GPT-3.5 Zero-Shot performed worse than a random agent in Act 2.The results indicate that complex strategic games require stronger models and advanced prompting for difficulty measurement.Source: §5.3 · p. 8
• Although the expert rule-based AI performed nearly as well as GPT-4 CoT in remaining HP, GPT-4 CoT correlated much better with human win rates.This suggests that human-like behavior, rather than gameplay capability alone, better approximates human-indicated difficulty in Slay the Spire.Source: §5.3 · p. 10
• Expert-level qualitative behavior emerged through threat prioritization, card-synergy reasoning, future-state prediction, and independently generated strategies.One example calculated 59 damage from Heavy Blade after using Spot Weakness to increase Strength, without supplying human-crafted strategic information.Source: §5.3 · p. 10

Guidelines

• Structured natural-language representations can substantially improve LLM gameplay performance, such as formatting Wordle words as character lists rather than plain text.The authors attribute this to tokenization limits and recommend explaining game-specific mechanics in accessible language.Source: §6 · p. 10
• LLM performance may require case-specific compensation, but overcompensation must not erase challenge variation or depart substantially from human gameplay.The study increased Wordle’s guessing limit and supplied Slay the Spire with a stronger deck to improve performance variation and boss completion.Source: §6 · pp. 10–11
• LLMs are better suited to comparing relative difficulty and shaping difficulty curves than to estimating human win rates for isolated challenges.After calibrating human performance on one or more challenges, LLMs can assess new challenges relative to those references.Source: §6 · p. 11
• Larger models, chain-of-thought prompting, and typical gameplay strategies can improve correlation with human gameplay, whereas exploitative strategies may bias difficulty estimates.The discussion specifically contrasts GPT-4 with GPT-3.5 and recommends more capable models when available.Source: §6 · p. 11
• Small-scale human pilot data can help select difficulty indicators and calibrate compensation despite the framework being otherwise LLM-based.Candidate indicators include scores, time taken, and win rates.Source: §6 · p. 11

Limitations and future work

• The framework currently isolates challenges, treats LLM abilities as fixed, and uses text-only inputs, limiting cumulative-sequence, learning, and visually intensive game evaluation.Future work proposes gameplay history, repeated trials, varied play styles, and vision-language models.Source: §6 · pp. 11–12
• Future applications could extend LLM game testing beyond difficulty to bugs, gameplay imbalances, and user-experience factors.The authors present these as open areas for further evaluation and exploration.Source: §6 · pp. 11–12

• The framework measures game difficulty with LLM agents and was tested on Wordle and Slay the Spire.It is designed to adapt to a wide range of games.Source: §7 · p. 12
• LLMs show high correlation with human players' difficulty assessments under simple and generic prompting.Source: §7 · p. 12
• Compared with rule-based and machine learning-based agents, LLMs can play varied games with minimal adjustment and without extensive development or training resources.Source: §7 · p. 12
• The findings motivate using LLMs as game-testing agents and developing broader applications and guidelines for gaming-industry testing.Source: §7 · p. 12