Source-linked AI summary
Leveraging LLM Agents for Automated Video Game Testing
Chengjia Wang, Lanling Tang, Ming Yuan, Jiongchi Yu, Xiaofei Xie, Jiajun Bu
TL;DR
MMORPG testing remains costly and incomplete because conventional automation struggles with open-ended state-action spaces and existing LLM agents have limited long-horizon reasoning. TITAN combines structured state abstraction, action optimization, reflective reasoning, and diagnostic oracles; across two commercial MMORPGs, it achieves 95% task completion and detects four previously unknown bugs. Its component ablations and real-world deployments support its practical relevance, although evaluation scope and model choice limit generalizability.
Problem
MMORPG testing is costly and incomplete because traditional automation struggles with rich open-ended environments and existing LLM agents have limited reasoning over complex states and long tasks.
Method
TITAN is an LLM-driven MMORPG testing framework combining state abstraction, action optimization, reflective reasoning, and diagnostic oracles.
Results
95% task completion and bug detection performance of 15 exceeded automated approaches’ best results of 82% and 9, while TITAN found four previously unknown bugs.
Takeaways & Limitations
TITAN provides a training-free framework deployed in eight real-world QA pipelines for complex MMORPG testing.
Takeaways & Limitations
Findings are limited to two evaluated games, one foundation model, and MMORPGs or similar genres where LLM reasoning aligns with game mechanics.
Abstract
from arXiv · showhide
Testing MMORPGs (Massively Multiplayer Online Role-Playing Games) is a critical yet labor-intensive task in game development due to their complexity and frequent updating nature. Traditional automated game testing approaches struggle to achieve high state coverage and efficiency in these rich, open-ended environments, while existing LLM-based game-playing approaches are limited to shallow reasoning ability in understanding complex game state-action spaces and long-complex tasks. To address these challenges, we propose TITAN, an effective LLM-driven agent framework for intelligent MMORPG testing. TITAN incorporates four key components to: (1) perceive and abstract high-dimensional game states, (2) proactively optimize and prioritize available actions, (3) enable long-horizon reasoning with action trace memory and reflective self-correction, and (4) employ LLM-based oracles to detect potential functional and logic bugs with diagnostic reports. We implement the prototype of TITAN and evaluate it on two large-scale commercial MMORPGs spanning both PC and mobile platforms. In our experiments, TITAN achieves significantly higher task completion rates (95%) and bug detection performance compared to existing automated game testing approaches. An ablation study further demonstrates that each core component of TITAN contributes substantially to its overall performance. Notably, TITAN detects four previously unknown bugs that prior testing approaches fail to identify. We provide an in-depth discussion of these results, which offer guidance for new avenues of advancing intelligent, general-purpose testing systems. Moreover, TITAN has been deployed in eight real-world game QA pipelines, underscoring its practical impact as an LLM-driven game testing framework.
1 Introduction
MMORPG testing is difficult because expansive, evolving game worlds overwhelm conventional automation and shallow LLM agents. TITAN addresses this gap with structured perception, action optimization, reflective reasoning, and diagnostic oracles, achieving strong results across commercial games.
- Evaluation: 95% task completion and 15 detected bugs surpassed automated approaches’ best results of 82% and 9, respectively.The evaluation covered two large-scale commercial MMORPGs on PC and mobile platforms.
- Findings: TITAN detected four previously unknown bugs, including Model Logic Bug and Hang Interaction Bug, that prior approaches missed.These findings demonstrate coverage beyond identifiable bugs from earlier testing approaches.
- Practical impact: TITAN has been adopted in eight real-world game QA pipelines, while the authors provide guidance for future MMORPG and LLM-agent testing.The reported deployments connect the prototype evaluation to practical game QA use.
- Approach: TITAN combines state abstraction, action optimization, reflective thinking, and novel oracles to support complex MMORPG testing.The framework is designed to perceive complex states and maintain long-horizon plans without task-specific training.
- Study design: The study evaluates 20 tasks of varying difficulty across two commercial MMORPGs and asks about task completion, bug detection, and component contributions.The research questions explicitly examine effectiveness, bug detection against existing methods, and ablation effects.
2 Background and Related Works
MMORPGs combine persistent worlds, nonlinear tasks, complex mechanics, and high player interactivity, making conventional testing insufficient. Existing scripted, reinforcement-learning, and LLM-agent approaches remain limited by brittleness, training demands, scalability, or constrained generalization.
- MMORPG testing challenges: Persistent worlds, nonlinear tasks, NPC behavior, combat, and cooperative gameplay create multifaceted MMORPG testing challenges.These challenges span functionality, performance, balance, regression, and emergent gameplay patterns.
- Traditional methods: GUI and traditional testing techniques often fail to handle game tasks requiring player intelligence and complex interaction.Surveys describe current game-testing approaches as broad but still immature.
- Automated testing: Scripted automation is brittle under updates, while DRL approaches require extensive training, reward design, and computational resources.These constraints make both approaches difficult to apply to dynamic MMORPGs.
- LLM-based agents: LLM agents provide promising zero-shot generalization, but MMORPGs’ vast partially observable states can exceed their perception and memory limits.Existing game agents also often depend on structured APIs or predefined action spaces that constrain cross-game generalization.
3 Design
TITAN uses a closed loop that abstracts multimodal game states, filters and prioritizes actions, reasons over traces and coverage, and detects issues through assembled oracles. These mechanisms reduce state and action complexity while supporting adaptive exploration and long-horizon testing.
- 3.1 Overview: TITAN’s loop perceives game states, selects and executes actions, maintains coverage maps, reflects when progress stalls, and diagnoses issues.A foundation model supplies high-level reasoning while supporting modules provide perception, scaffolding, memory, and oracles.
- 3.2 Perception Abstraction Module: State abstraction selects expert-relevant features, discretizes continuous values, filters irrelevant data, and encodes concise representations for LLM decisions.Examples include location, objectives, vitals, nearby interactables, status effects, and High/Medium/Low health buckets; game-agnostic templates support portability.
- 3.3 Action Optimization Module: Action optimization narrows MMORPGs’ vast action spaces using high-level templates, relevance rules, retrieved guidance, LLM heuristics, and validation.The module recommends roughly five contextually plausible actions while preserving flexibility and syntactic validity checks.
- Reflective reasoning: Reflective reasoning uses state information, recommendations, action traces, and a coverage map to select subsequent actions and self-correct during long tasks.Task-centric memory balances completing the current task with exploring under-tested regions.
- Oracle detection: TITAN’s assembled oracles detect diverse issues with a 30% overall false alarm rate, most commonly from imperfect environment adaptation during state abstraction.Exploration and detection mechanisms jointly form the reported autonomous testing framework.
4 Experiment Setup
TITAN is evaluated on two contrasting commercial MMORPGs using a 20-task benchmark spanning simple to highly complex scenarios. The study compares TITAN with DRL, ReAct, and human-testing baselines under a pipeline that abstracts states, optimizes actions, reflects during execution, and reports anomalies.
- 4.1 Target Games: Evaluation spans a PC 3D MMORPG and a mobile MMORPG, differing in themes, controls, perspectives, and player scale.Game A has real-time combat and large open-world areas, while Game B uses tap-based controls and auto-pathfinding.
- 4.2 Evaluated Tasks: The benchmark contains 20 tasks, evenly sampled across Simple, Normal, and Hard categories defined by state-action complexity.Tasks contain approximately 10, around 20, or over 20 state-action pairs, respectively, and include multi-step scenarios such as reaching difficult locations or triggering optional branches.
- TITAN Framework: TITAN’s algorithm takes game APIs, state information, screenshots, expert knowledge, feasibility rules, thresholds, and a test objective, producing potential issues and diagnosis reports.Its pipeline initializes coverage and action history, reads and abstracts state, updates coverage, recommends feasible actions, monitors execution, reflects, and records anomalies or crashes.
- 4.3 Baseline Methods: The comparison includes Wuji’s DRL game-testing method, ReAct with the same foundation models as TITAN, and professional human testers.For each task, the study measures task completion, test coverage, and bug detection; human results serve as a proxy for manual test cases.
- 4.3 Baseline Methods: Purely scripted testing is excluded because diverse states and flexible complex scenarios make manually scripting unique solutions impractical.TITAN and automated baselines use GPT-4o with temperature 0, while automated results are averaged over five runs and human results across repeated testing.
5 Evaluation Result
TITAN outperforms automated baselines in task completion, state coverage, bug detection, and execution efficiency across two MMORPGs, while ablations show that its core components work synergistically.
- RQ1: Task Completion Performance: 95% task completion exceeded Wuji (82%) and ReAct (83%), while approaching professional human testers (100%).TITAN solved all simple and normal tasks and failed only twice on complex tasks with very long action chains.
- RQ1: Task Completion Performance: 73.26% average state coverage exceeded Wuji (54.54%) and ReAct (59.98%), indicating stronger exploration of unique abstract states.TITAN balances completing objectives with strategically exploring untested scenarios.
- RQ2: Bug Detection Performance: 82% bug detection across nine bugs exceeded Wuji and ReAct (45.5% each) and human testers (18%), with higher execution efficiency.TITAN detected hard-to-reach crash, hang, and logic bugs through reflection, progress monitoring, and category-aware oracles.
- RQ2: Bug Detection Performance: TITAN’s reflective reasoning helped expose logic inconsistencies, while varied interaction strategies improved detection of hang and crash conditions.The agent tried multiple combat and interaction pathways instead of repeatedly following a single habitual action.
- RQ3: Ablation Study: Removing any core component reduced performance by up to 24% in task completion and 27% in bug detection, confirming component synergy.The full system outperformed ablated variants, and removing all three components produced results comparable to ReAct.
6 Discussion
TITAN uncovered four previously unknown bugs and offers a holistic, adaptable testing approach, but its evidence is constrained by model, genre, and evaluation scope.
- Newly Detected Bugs: TITAN uncovered four previously unknown bugs that conventional testing methods failed to identify.Representative cases include world-geometry clipping, an interaction hang, and a quest step-counting error.
- Newly Detected Bugs: TITAN’s unconventional exploration exposed failures missed by agents and testers following expected execution routes.Its interaction and planning mechanisms enabled deviations that surfaced invisible-boundary, hang, and off-by-one quest defects.
- Implications: TITAN combines task completion, coverage-guided exploration, and oracle-based bug detection within one testing framework.This addresses functional correctness, testing thoroughness, and diagnosis simultaneously.
- Implications: Its game-agnostic abstractions and retrieval-augmented knowledge enabled configuration for two distinct MMORPGs within minutes.This adaptability targets rapidly changing development cycles without requiring extensive retraining.
- Threats to Validity: Generalizability is limited by evaluation on two games, reliance on GPT-4o, and possible reduced benefits in reflex-oriented or minimally narrative genres.Performance also depends partly on alignment between game mechanics and the LLM’s prior knowledge of conventional game logic.
7 Conclusion
TITAN is an LLM-driven MMORPG testing framework that outperforms prior approaches and human testers while detecting bugs they miss. Its training-free design supports rapid adaptation and integration into real-world QA pipelines.
- TITAN outperforms state-of-the-art baselines and human testers in task completion and bug detection, identifying four previously unknown logic and performance bugs.Ablation studies confirm that each component contributes substantially to overall effectiveness.
- TITAN offers a practical, training-free solution that adapts quickly to game changes and is suitable for integration into real-world QA pipelines.
- The framework demonstrates the promise of structured LLM-based agents for complex game testing and motivates further development of intelligent, general-purpose testing systems.