PaperlayerView exact revision on arXiv

Source-linked AI summary

CriticaLLM: Using LLMs as Autonomous Game Master for Tabletop Role-Playing Games

Gaëtan Berlaimont, Justin Vanwichelen, Benoît Ronval, Hélène Verhaeghe

arXiv:2610.21525v1cs.AI

TL;DR

TTRPGs require a Game Master to create coherent stories, follow rules, and adapt to players, while long adventures can exceed an LLM’s context window. CriticaLLM automates game mastering with a Judge LLM and story summaries; the paper reports that it meets key requirements most of the time, with limitations in bias and fantasy-rule handling.

  • Problem

    LLM Game Masters must create original, coherent stories, adapt to player actions, and follow rules, while TTRPG adventures can exceed model context windows.

    Source: §I · pp. 1–2

  • Method

    CriticaLLM is a fully automated Game Master that uses a Judge LLM to validate player input and summarizes generated story text to retain key game information.

    Source: §I · p. 2

  • Results

    The framework meets the four identified requirements most of the time, and player feedback was positive about enemy behavior, theme adherence, and story coherence.

    Source: C. Users’ Feedback · p. 4, §VI · p. 6

  • Takeaways & Limitations

    The paper concludes that LLMs can run entire TTRPG games, from story creation through the final-boss fight.

    Source: §VI · p. 6

    The Judge LLM sometimes rejects actions based on sexist stereotypes and has difficulty applying fantasy-world rules, such as allowing an arrow to be shot at a ghost.

    Source: §V · p. 6

Abstract

from arXiv · show

Large Language Models (LLMs) have shown impressive capabilities across many tasks, including processing large amounts of text, following instructions, and generating ideas. This allows the use of such models in various contexts, including games and Tabletop Role-Playing Games (TTRPGs), such as Dungeons & Dragons. However, these games include many complex components, from crafting interesting stories to adhering to rules, while allowing some freedom in players' actions. In this work, we first highlight key components that LLMs should exhibit in the context of TTRPGs. Then, we introduce our framework, CriticaLLM, a fully automated Game Master that can run entire TTRPG games, from creating diverse and interesting stories to managing exploration and combat phases. We evaluate our framework with actual TTRPG players and examine the diversity of generated plots and antagonists across four LLMs of varying sizes. Overall, our framework appears to be a promising option for running entire TTRPG games.

I. INTRODUCTION

TTRPG game mastering combines story creation, rule enforcement, and adaptation to player choices, while LLMs face robustness and context-window constraints. CriticaLLM addresses these demands with an automated GM, a Judge LLM, and a story-summarization mechanism.

I. INTRODUCTION

  • LLMs can hallucinate, align with users over rules, and be attacked into ignoring instructions; these issues motivate checking their behavior as Game Masters.The paper also identifies fairness issues as a concern for LLMs.Source: §I · p. 1
  • The proposed GM role requires originality, coherence, adaptability, and instruction following: generating engaging stories, preserving the theme and plot, adapting to player actions, and rejecting rule-breaking actions.These requirements are framed as challenges for LLMs serving as fully automated Game Masters.Source: §I · pp. 1–2
  • The story, rules, and player actions must fit within the LLM’s context window, so the system needs a way to keep relevant game information accessible.The passage notes that TTRPG adventures often require more text than the model context can hold.Source: §I · p. 2
  • CriticaLLM is a fully automated LLM-powered Game Master that creates themed stories and runs games under Dungeons & Dragons rules.Its contributions include a Judge LLM for validating player actions, a summarization mechanism for retaining essential details, and robustness tests against 11 attack types.Source: §I · p. 2

A. LLMs for TTRPGs

Prior work ranges from tools that assist human Game Masters to systems that automate the role. The paper distinguishes these approaches and notes reported issues in prior systems.

A. LLMs for TTRPGs

  • Prior TTRPG work distinguishes LLM tools for human Game Masters from systems that implement fully automated Game Masters.The review identifies Dungeons & Dragons as the primary focus of this work.Source: A. LLMs for TTRPGs · p. 2

1) LLMs as a Tool for TTRPGs:

  • Shoelace supports NPC dialogue, while Calypso provides conversations and game-state information but is reported to have hallucinations and repetitions..Source: 1) LLMs as a Tool for TTRPGs · p. 2

2) LLMs as Automated GM:

  • Automated-GM examples include text-only AI Dungeon, human-evaluated D&D systems, Fable Reborn’s added tools, and models studied for predicting player intents.Prior human evaluations reported quality inconsistencies for Fable Reborn, and its input checker had not undergone robustness evaluation.Source: 2) LLMs as Automated GM · p. 2

B. LLM-as-a-Judge

LLM-as-a-judge systems evaluate text, and prompt injection can let users bypass a GM LLM’s rules. CriticaLLM separates story and game running from player-input validation within a three-room adventure.

B. LLM-as-a-Judge

  • LLM-as-a-judge uses an LLM to evaluate text, typically text generated by another LLM.The review notes its use in evaluating and critiquing generated data and in LLM fine-tuning.Source: B. LLM-as-a-Judge · p. 2

C. Attacks against LLMs

  • Prompt injection can circumvent an LLM’s generation rules, allowing players to break game rules and disrupt coherence and other players’ experience..Source: C. Attacks against LLMs · p. 2

A. Overall Structure

  • CriticaLLM separates the GM LLM, which runs the story and game, from the Judge LLM, which validates player input in a three-room adventure ending with a boss or character death.The generated map example depicts one group of four players.Source: A. Overall Structure · p. 2, Figure 1 · p. 3

B. Game Map Structures

CriticaLLM generates room maps and groups nearby entities for interaction, then supplies local game state to the GM during exploration and combat. Combat uses turn order, validated actions, dice outcomes, and reasoning-based enemy choices.

B. Game Map Structures

  • Rooms use four layouts—hall, corridor, big hall, and big corridor—generated by random walks, with chests and enemies placed to avoid overpopulation or scarcity.The maps are divided into squares used to represent distances.Source: B. Game Map Structures · p. 3
  • A Union-Find grouping rule treats entities within three squares as close, including transitively connected entities, to determine which can interact.The example map’s four players form one group.Source: B. Game Map Structures · p. 3

1) Map Exploration:

  • During exploration, players move a fixed distance and submit an action after Judge LLM validation; the GM receives the theme, plot summary, room and group details, enemy presence, and the last three phases..Source: 1) Map Exploration · p. 3

2) Battle Management:

  • Combat starts when a player can see an enemy; dice and character statistics determine turn order, and validated player actions include attacks and healing.A 20-sided die determines action outcomes.Source: 2) Battle Management · p. 3
  • Enemies have four size categories and health, armor, and accuracy values that scale with character count and, for bosses, room number; the GM reasons over current combat status to choose strategic actions and targets.The design aims to make enemy behavior less repetitive and battles more varied.Source: 2) Battle Management · p. 3

D. Judge LLM

The Judge LLM checks player text against the game’s story, rules, state, and theme before passing it to the GM or rejecting it. Memory management condenses generated text while preserving recent continuity and location-specific events.

D. Judge LLM

  • The Judge LLM checks player text against the current game state, story, rules, and theme, then validates it for the GM or rejects it with an explanation.It receives the text and descriptions of the current state and room, then classifies the player’s text as valid or invalid.Source: D. Judge LLM · p. 3

E. Input Text Memory Management

  • The GM produces short summaries of generated descriptions and uses them with the last three gameplay phases to reduce prompt length while retaining immediate continuity and a global story overview.The summaries focus subsequent prompts on important story events and help keep text within the context window.Source: E. Input Text Memory Management · pp. 3–4
  • Map squares store location-specific event text, which is loaded into the prompt when a player is nearby.This preserves events such as finding a chest or breaking an object without keeping all map information in the prompt.Source: E. Input Text Memory Management · p. 4

IV. EXPERIMENTS AND RESULTS

The evaluation combines volunteer expert play, questionnaires, and recorded game logs, using four LLMs for the GM and Judge roles. Players reported strengths in coherence and rule compliance, alongside weaker long-term adaptability.

A. Selected LLMs

  • The GM and Judge roles were evaluated with Llama 3.3 (70B parameters), Gemma 2 (9B parameters), Mistral Small, and GPT-4o-mini.The model selection was intended to examine originality and robustness by size and origin.Source: A. Selected LLMs · p. 4

B. Collecting the Data

  • Nine volunteer human experts played games with the framework, completed a questionnaire, and had their prompts and logs recorded for analysis.Source: B. Collecting the Data · p. 4

C. Users’ Feedback

  • Players generally reported coherent themes and stories and positive enemy behavior; all encountered Judge rejections, most legitimate, supporting rule-following.The authors interpret the feedback as meeting the coherence and instruction-following requirements.Source: C. Users’ Feedback · p. 4
  • Most players said their choices had only slight impact, and the GM often missed their intentions, limiting long-term story adaptation.The authors relate this difficulty to prompting with recent actions to keep prompts and response times short, and to tension between coherence and adaptability.Source: C. Users’ Feedback · p. 4

D. Story Plot Diversity

Judge robustness varies by model: Gemma 2 resists the tested attacks better than Llama 3.3. Plot similarity remains high across models, while antagonist generations are less repetitive than plots.

D. Story Plot Diversity

  • All models’ generated plots have average cosine similarity above 0.7, with Gemma 2 less repetitive than Mistral Small and Llama 3.3.Plots were compared across 200 generations per model.Source: D. Story Plot Diversity · p. 5, Figure 3 · p. 5
  • Generated antagonists appear less repetitive than plots, but Llama 3.3 performs worse than the other models on antagonist diversity.The comparison uses 200 antagonist generations per model.Source: D. Story Plot Diversity · p. 5, Figure 4 · p. 5

E. Attacks against the Judge

F. Price Analysis

Gemma 2 is presented as a strong quality-cost choice for running games as GM and Judge, with local execution also possible.

F. Price Analysis

  • With a $10 budget, Gemma 2 is presented as one of the best quality-cost choices for the GM and Judge and can also run locally.Source: F. Price Analysis · p. 6

V. LIMITATIONS

The Judge has observed limitations involving sexist stereotypes and difficulty applying rules within a fantasy world.

V. LIMITATIONS

  • The Judge can reject actions based on sexist stereotypes and struggle to apply rules in fantasy settings, such as refusing an arrow attack on a ghost.The authors suggest bias-aware mitigation and balancing logical validation with creative freedom as possible improvements.Source: §V · p. 6

VI. CONCLUSION

CriticaLLM is presented as a fully automated TTRPG framework using an LLM judge and attention to prompt size. The authors report that it can run entire games, while noting room to improve adaptability and address judge flaws.

VI. CONCLUSION

  • CriticaLLM can run entire TTRPG games from story creation through the final boss fight.The conclusion describes the framework as entirely LLM-run.Source: §VI · p. 6
  • The approach meets the requirements of originality, coherence, adaptability, and instruction-following most of the time, though adaptability may need improvement.Source: §VI · p. 6
  • The authors demonstrate the Judge LLM's robustness while also highlighting potential flaws.Source: §VI · p. 6
  • Future work could reduce biases in the Judge LLM and improve adaptation to player actions without significantly changing the main plot.Source: §VI · p. 6

Related papers

Loading 2610.21525v1…