Source-linked AI summary
Large Language Models and Games: A Survey and Roadmap
Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, Georgios N. Yannakakis
TL;DR
Games are important AI benchmarks and application areas, while developers and researchers are still exploring how to use LLM advances effectively. This paper surveys LLM roles and applications in games, synthesizes their potential and challenges, and proposes future research directions; it concludes that their impact on Game AI research and the game industry is unlikely to be ignored.
Problem
Games are important AI benchmarks and application areas, but developers and researchers are still exploring how to use recent AI advances effectively.
Method
The paper surveys applications and roles of LLMs in games, including player-facing systems, design assistance, and systems that generate action-producing programs.
Results
The survey identifies opportunities for LLMs to improve player experiences and enhance designers’ ability to bring ideas to life, alongside technical, ethical, and legal challenges.
Takeaways & Limitations
LLM research is likely to affect Game AI research and the game industry, motivating technical innovation and further applications of LLMs to games.
Takeaways & Limitations
LLM applications in games remain constrained by hallucinations, difficulty capturing user intent, high simulation costs, copyright concerns, limited explainability, and inconsistent behavior from closed-source models.
Abstract
from arXiv · showhide
Recent years have seen an explosive increase in research on large language models (LLMs), and accompanying public engagement on the topic. While starting as a niche area within natural language processing, LLMs have shown remarkable potential across a broad range of applications and domains, including games. This paper surveys the current state of the art across the various applications of LLMs in and for games, and identifies the different roles LLMs can take within a game. Importantly, we discuss underexplored areas and promising directions for future uses of LLMs in games and we reconcile the potential and limitations of LLMs within the games domain. As the first comprehensive survey and roadmap at the intersection of LLMs and games, we are hopeful that this paper will serve as the basis for groundbreaking research and innovation in this exciting new field.
1 INTRODUCTION
The paper examines how LLMs can be used in games while balancing their potential against technical, theoretical, and ethical shortcomings. It surveys existing academic and creator work, proposes roles for LLMs, and identifies future directions.
- Motivation: LLMs have moved from a niche NLP topic to a highly active research field with broad potential and widely varying expectations.Research addresses both capability improvements and reductions in compute and memory requirements.
- Motivation: Games are both important AI benchmarks and application areas, making current efforts to use recent AI advances exploratory.The paper frames developers and researchers as still determining how best to use these advances.
- Scope and approach: The survey charts LLMs’ past and likely near- to mid-term impact on games and games research.It covers work from academia and mostly independent game creators.
- Scope and approach: Rather than surveying LLM architectures or training algorithms, the paper focuses on applications that leverage LLMs in games and the broader games ecosystem.The authors note that rapid technical progress could make a technology-focused survey obsolete quickly.
- Scope and approach: The paper’s typology is developed top-down from the authors’ expertise and supported by a comprehensive manual review of recent major AI-and-games conference proceedings.The authors distinguish this from a bottom-up keyword search through general repositories.
2 A NOTE ON TERMINOLOGY
The survey defines LLMs around large-scale, predominantly transformer-based text models with broad task versatility, while including text-capable multimodal models. It excludes machine-learning methods without textual input and output.
- Scope: The survey concerns the intersection of games and LLMs, including board games, video games, and other games.
- Definition: LLMs are treated as large text-trained models that reproduce text in response to text, distinguishing them from earlier small language models by scale.The authors use GPT-2’s size as a soft cutoff, reflecting the term’s association with GPT-class models.
- Definition: The survey relies primarily on transformer-based LLMs, because that architecture dominates the current LLM landscape.Transformers were introduced in 2017 and were perceived as producing a major quality improvement.
- Definition: A defining feature is versatility across many tasks with minimal or no fine-tuning, reflecting a shift toward general-purpose conversational models.
- Scope: The survey includes text-capable large multimodal models and excludes AI or machine-learning techniques that do not use textual input and output.Examples of included multimodal models include GPT-4V and Llava; open-source models such as Mistral and Llama may run locally or inside game runtimes.
3 ROLES OF LLMS IN GAMES
The survey organizes LLM applications across roles inside and outside games, including players, NPCs, assistants, Game Masters, mechanics, designers, analysts, commentators, and retellers. Player applications require mappings between game states, actions, and language-model outputs.
- Roles of LLMs in games: LLMs can act as players, NPCs, player assistants, Game Masters, game-mechanic controllers, designers, gameplay analysts, commentators, or retellers.
- Player: LLM players need game-state information and transformations from token sequences into game inputs, with three suitable classes of games.These classes use compact abstract tokens, natural-language interaction, or external APIs for controlling actions.
- Player: Tokenized board-game states and moves map action selection to autoregressive next-move prediction, supporting applications in Chess, Go, and Othello.
- Player: Text-game systems can query LLMs directly for actions, but ChatGPT’s Zork performance approached existing algorithms only with a human interlocutor assisting the model.Generalization to entirely novel or unseen text games remains largely unexplored.
- Player: CICERO combines a pretrained and Diplomacy-finetuned LLM with dialogue-conditioned intent and strategic reasoning to select actions in negotiation-based play.
- Player: VOYAGER uses GPT-4 to generate API-calling programs that convert high-level Minecraft skills into low-level inputs, and it completed varied in-game challenges.Its success benefits substantially from Minecraft’s robust API and extensive online discussions; generalization to less popular or unseen games remains uncertain.
3.2 Non-Player Characters
LLM-based NPCs can provide dynamic dialogue and believable, rule-constrained behavior across narrative roles, while player assistants can offer conversational guidance. These applications remain constrained by memory, hallucination, volatility, and limited research on assistance.
- Non-Player Characters: NPCs enrich game worlds as pets, allies, enemies, merchants, quest givers, or bystanders, with agency distinct from AI-controlled players.
- Non-Player Characters: LLM dialogue systems can generate dynamic, contextually appropriate NPC responses that make interactions more engaging and reduce repetitive discourse.Foreground NPCs require attention to narrative scope, player actions, game context, and events across the playthrough.
- Non-Player Characters: Foreground, background, and narrator NPCs serve different functions, from advancing narratives to making environments believable and acting independently of players.Narrator LLMs are treated as commentators elsewhere in the survey.
- Non-Player Characters: Multiple LLM-based agents can follow game rules and exhibit model-specific aptitudes and weaknesses in different roles.Unconstrained multi-character generation can instead behave like theatrical improvisation and produce scenario-inconsistent hallucinations.
- Non-Player Characters: Carefully crafted personas can constrain LLM-based NPCs to a game narrative and prescribed personality while supporting free player conversation.Ubisoft’s Neo NPC demo illustrates this approach.
- Player Assistants: LLM player assistants can provide context-specific tips, suggested actions, and natural-language explanations, potentially accompanied by sentiment and embodied expression.The survey describes this role as relatively unexplored in current research, despite earlier work on avatar inner voices.
3.4 Commentator/Reteller
LLMs can act as commentators or retellers that narrate game events for players or spectators, but research remains limited and unconstrained outputs often feel generic.
- Commentator LLMs narrate sequences of in-game events for human players or spectators, using game context and sometimes information about the player.
- Earlier machine-learning commentary systems produced repetitive or context-irrelevant text when generating scripts from game inputs.
- A League of Legends pipeline detected key events, prompted ChatGPT for character-styled commentary, and converted the output to speech.
- Research on LLM game commentators remains limited, and generic prompting tends to yield stories that feel generic without specified style or substance.
3.5 Analyst
LLMs have been explored as analysts of player behavior and as assistants for game analysis, while broader analyst use and automated game-mastering remain bounded by practical reliability concerns.
- LLM-based analysis focuses primarily on player experience and behavior, an area that has received relatively little attention.
- Player2vec used transformer representations of JSON game events to identify eight player-behavior clusters useful for market research.
- Current work positions LLMs more naturally as analyst assistants than as independent analysts.
- LLM representations of gameplay logs can support action-similarity analysis and alignment between gameplay footage and recorded actions.
- Automated game masters can support interactive storytelling, while GM assistants can generate encounters, brainstorm ideas, or converse as fictional characters.
- Automated and assisted GM systems can receive positive player responses, but hallucinated or biased details may also produce incorrect or constrained content.
3.7 Game Mechanic
LLMs can serve as the central mechanic of games by enabling conversational social interaction, language synthesis, and open-ended combinations whose outcomes are model-generated.
- LLM-powered conversational NPCs can populate sandbox worlds with agents that communicate, form relationships, and display social behavior.
- Natural-language games can turn attempts to manipulate or trick an LLM into the central player challenge.
- In Infinite Craft, players combine core elements while an LLM generates the resulting entities, including unexpected fictional concepts.
3.8 Automated Designer
LLMs can automate game design by generating levels, adapting content from instructions, producing game assets, and helping create complete games, but these systems depend on suitable data and constraints.
- Procedural content generation targets digital-game content such as levels and visuals, which must satisfy constraints including playability and aesthetic quality.
- PCGML with LLMs depends on high-quality machine-readable datasets, but game content is often unavailable or protected by intellectual-property laws.
- Fine-tuned GPT models have generated Sokoban and Super Mario Bros levels, with evolutionary procedures helping address limited training data.
- Natural-language instructions can condition level generation, while Cardistry transforms personal narratives into playing cards and prompts for DALL-E artwork.
- A second LLM can refine fuzzy user requests with domain knowledge before generating game-interpretable structure descriptions.
- LLMs can generate natural-language puzzles, game programs, or design blueprints that guide additional models producing narrative, textures, and animations.
3.9 Design Assistant
LLM design assistance spans conceptual suggestions, production support, and procedural editing, with the human designer retaining control over generated artifacts. Existing systems show useful conversational and constraint-aware workflows, but capability and autonomy vary substantially.
- Design-assistance roles: Design assistants can reduce development time and effort, support collaboration, and stimulate creativity, while production assistance keeps the designer responsible for refining or rejecting artifacts.Production assistance operates in a constrained space because it must respect finalized game mechanics and designer goals.
- Conceptual assistance: Conceptual assistance generates generic game-feature suggestions that may require substantial design work, yet can provide useful alternative perspectives and inspiration.Players found specific LLM-generated features less compelling than human-designed ones, while designers still benefited from the suggestions as creative starting points.
- Production assistance: Production-assistance systems generate levels from natural-language specifications, after which humans curate and edit the results to ensure playability.One workflow uses the curated levels for further fine-tuning, potentially supporting later automation.
- Production assistance: Future Worlds parses user requests into level parameters and generates multiple candidate levels matching requested difficulty, size, sustainability problem, and simulation goals.
- Procedural assistance: LLMaker lets designers inspect and modify dungeon layouts and entities through natural-language conversation, translating requested changes into valid function calls that preserve domain constraints.Unspecified parameters can be filled by the LLM using the existing level and user preferences, including related descriptive changes.
4 A ROADMAP FOR FUTURE APPLICATIONS OF LLMS IN GAMES
The roadmap identifies heavily studied LLM roles alongside substantial gaps in design assistance, player support, commentary, and player-experience modeling. It calls for more procedural, conversational, and research-oriented applications while emphasizing unresolved consistency, explainability, privacy, and intellectual-property concerns.
- Research landscape: Player and automated-designer roles have received the most attention, while the survey proposes broader research across the other roles identified in its typology.The authors expect community events, benchmarks, and competitions to help develop LLM-based approaches in games.
- Design assistance: Design assistance remains underused because current systems either demand extensive human interpretation and development or reduce the designer to a content curator.The proposed opportunity is procedural assistance in which the LLM produces artifacts, reasons about them, and supports iterative refinement.
- Player and Game Master assistance: Player assistants could provide tutorials, hints, or help with trivial tasks, while Game Master assistants could track visited locations, met NPCs, and rules.Hallucination and consistency problems remain central requirements for these applications.
- Commentator and reteller roles: Streamer assistance could summarize audience interactions and engagement so human streamers can follow chat topics without reading every comment.The survey frames this as assistance rather than replacement, with commentary explainability remaining important.
- Player modeling and user research: LLMs could extend game-user research from clustering gameplay logs to explaining play personas and modeling affective state transitions for adaptive environments.Such systems would still face explainability, privacy, multimodal-input, and quality-assurance challenges.
- Broader directions: The survey concludes that LLMs are promising conversational assistants and may support open-ended content such as narratives or game-design documents, but implementation and intellectual-property issues remain unresolved.Games also offer rich multimodal, complex, constrained settings for LLM research.
- Constraints and evaluation: Game applications must account for hard constraints such as level completion and soft constraints such as balance and pacing, which may be difficult for LLMs to represent.Function calling can handle some hard constraints, but more complex domains and nuanced soft constraints remain challenging.
5 LIMITATIONS OF LLMS IN GAMES
LLMs offer useful capabilities for games but remain constrained by hallucination, intent, memory, responsiveness, compliance, and cost problems. These limitations restrict reliable long-term, real-time, and large-scale deployment, although external databases and retrieval systems offer possible mitigations.
- Reliability: Hallucinations produce plausible but false statements because LLM outputs lack grounding in real-world constraints.
- Intent understanding: LLMs often struggle to understand sarcasm and user requests, making repeated clarification frustrating in conversational game applications.This limitation is especially relevant to design assistants, player assistants, and other direct user interactions.
- Memory and continuity: Context-size limits make long conversations increasingly likely to lose early events, weakening continuity in long-term game interactions.This is especially relevant to retellers and Game Masters, whose roles require sustained engagement.
- Possible mitigations: Larger context windows, external databases, and retrieval-augmented generation could help LLMs consult stored game events and maintain consistent narrative progression.The survey presents these approaches as potential mitigations rather than established solutions for complex game states.
- Game Master control: An overly compliant LLM Game Master may accommodate bizarre requests that disrupt predetermined game events, unlike human GMs who can curb them.
- Deployment cost: Running AAA games and LLMs together on consumer hardware is infeasible, forcing developers toward hosted models or APIs with recurring and difficult-to-estimate costs.Costs can grow with player scale and repeated sessions.
- Deployment cost: “Thousands of dollars in token credits” were spent on several days of simulations involving multiple LLM-based NPCs.Cost-reduction techniques exist but are not yet widespread and require additional engineering.
- Responsiveness: Real-time LLM application remains implausible for digital games because player responsiveness requires rapid responses.Research on faster and more efficient architectures is ongoing.
6 ETHICAL ISSUES WITH LLMS IN GAMES
LLMs introduce ethical concerns in games involving sustainability, bias, copyright, explainability, privacy, and reproducibility. These challenges affect how developers and researchers can responsibly deploy LLM-based game systems.
- Scope of ethical issues: Ethical issues in games extend beyond technical performance to sustainability, copyright, explainability, privacy, and biases.The paper frames each issue as having serious implications for the games field.
- Sustainability: LLM training and inference raise sustainability concerns because frequent querying over a model’s lifespan can increase its environmental impact.Renewable and local energy, better architectures, and more meaningful training data may mitigate this impact.
- Copyright: Copyright issues affect training inputs, generated outputs, and model licenses, creating distinct risks for game companies and creators.Automatically produced content may not be copyrightable in the United States, while substantial human transformation may support copyright for assistive-tool outputs, though the authors retain a caveat.
- Explainability: LLMs’ opaque generation processes make explainability difficult, especially for iteratively refined products such as design assistants.Chain-of-thought explanations can appear reasoned while still reaching correct answers through incorrect steps or disregarding the stated reasoning.
- Replicability: Closed-source LLMs can produce substantially different outputs for the same request over time, undermining application replicability.Developers may therefore consider locally running open-source models for greater control, while compacting model size and carbon footprint remains important.
- Bias: Biases from predominantly Western-focused internet data can reproduce social stereotypes and exclusionary norms in games.The paper distinguishes biases that may be targeted and alleviated from exclusionary norms that pose greater challenges.
7 CONCLUSIONS
The survey identifies diverse roles for LLMs in games and game development, while recognizing technical, ethical, and legal challenges. It concludes that LLMs’ likely impact on Game AI research and the game industry warrants further technical innovation and exploration of promising applications.
- 7 CONCLUSIONS: LLMs can improve players’ digital-game experiences and enhance game designers’ ability to bring ideas to life.The conclusion presents these roles alongside challenges intrinsic to LLMs and their surrounding ecosystem.
- 7 CONCLUSIONS: Despite technical, ethical, and legal challenges, the paper argues that ignoring LLMs’ likely impact on Game AI research and the game industry is unrealistic.
- 7 CONCLUSIONS: The paper proposes promising directions for future applications of LLMs to games in anticipation of new technical innovations from researchers and corporations.