PaperlayerView exact revision on arXiv

Source-linked AI summary

GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers

Shufan Jiang, Chios Chen, Zhiyang Chen

arXiv:2604.02648v1cs.SEcs.AI

TL;DR

Autonomous bug discovery remains underexplored because runtime testing requires agents to perceive anomalies and localize causes. GBQA evaluates this ability in interactive games, finding that even the best-performing model identifies less than half of verified bugs.

  • Problem

    LLM testing and bug discovery remain underexplored because agents must perceive runtime anomalies and localize their causes rather than receive precise task descriptions.

  • Method

    GBQA evaluates autonomous bug discovery across 30 diverse games containing 124 human-verified bugs, using an interactive agent with multi-round exploration and memory.

  • Results

    Less than half of the verified bugs are identified by the best-performing model, Claude-4.6-Opus in thinking mode.

  • Takeaways & Limitations

    State-of-the-art LLMs remain substantially limited in discovering bugs, particularly long-horizon and state-dependent errors, leaving a significant gap with quality-assurance demands.

  • Takeaways & Limitations

    The evaluation prioritizes recall because undetected defects are treated as more costly than false-positive reports that human reviewers can filter.

Abstract

from arXiv · show

The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.

1 INTRODUCTION

GBQA targets the largely unexplored problem of autonomous bug discovery, which requires proactive anomaly detection, systematic exploration, and reasoning about runtime behavior without explicit issue descriptions. It benchmarks this capability in interactive games using human-verified bugs, scalable bug injection, and a long-horizon ReAct agent, finding that even the best model detects less than half of the bugs.

  • Motivation: Autonomous bug discovery remains difficult because agents must recognize unspecified anomalies, explore large behavioral state spaces systematically, and compare expected with actual runtime behavior.Unlike generation or fixing, the task lacks a precise target or bug report, so anomaly perception and cause localization cannot be delegated to humans.
  • Motivation: Games provide a representative testbed because they combine internal state, user input, rendering, long-term dynamic interaction, and clearly defined action spaces and state transitions.These properties make games self-contained while supporting structured agent interaction in complex runtime environments.
  • Benchmark: GBQA evaluates autonomous bug discovery across 30 diverse games containing 124 human-verified bugs spanning three difficulty levels, with automated evaluation supported by a critic agent.Agents explore games, identify potential bugs, and report descriptions with reproducible steps.
  • Benchmark Construction: A scalable multi-agent builder generates games and injects bugs with controllable complexity, while human experts verify annotation correctness.The benchmark organizes implanted bugs into Easy, Medium, and Hard categories.
  • Results: Even Claude-4.6-Opus in thinking mode, the best-performing model, identifies less than half of the bugs using the provided multi-round ReAct agent with memory for long-horizon exploration.The experiments reveal substantial room for improvement in autonomous bug discovery and characterize current model limitations and failure modes.

2 RELATED WORK

Prior software-engineering, interactive-agent, and game-based benchmarks largely evaluate task completion or skill acquisition in functioning environments. GBQA instead evaluates agents by treating the environment as the object of evaluation and measuring flaw discovery.

  • Software Engineering and Agent Benchmarks: Unlike WebArena (Zhou et al., 2024), AgentBench (Liu et al., 2024), and SMART (Mu et al., 2025), GBQA treats the environment itself as the evaluation target and adds flaw discovery rate to task completion.The comparison distinguishes environments treated as ground truth from environments whose flaws agents must discover.
  • Game-Based Agents and Automated Game Testing: Voyager (Wang et al., 2023), MineDojo (Fan et al., 2022), CRADLE, and Generative Agents (Park et al., 2023) study goal achievement and skill acquisition in correctly functioning game environments.TITAN (Wang et al., 2025) and Orak (Park et al., 2025) are closer to GBQA because they explore LLM-assisted game testing.

3 GBQA

GBQA defines autonomous game-bug discovery as interactive exploration that produces reproducible bug reports covering ground-truth defects, with optional documentation enabling player-oriented or specification-driven testing. It comprises 30 game environments and 124 human-verified bugs across six genres, organized by discovery difficulty and evaluated primarily by recall.

  • Task formulation: The agent explores an environment by observing states, selecting actions, and generating natural-language reports with reproduction steps to maximize coverage of ground-truth bugs.Each environment is modeled as E = (S, A, T, s0), and the objective is to detect and describe every bug sufficiently for engineers to reproduce and fix it.
  • Task formulation: With no documentation, agents use Player Exploring Mode; with design documents or source code, they use Quality Assurance Mode for informed, specification-driven testing.Both modes are evaluated in Section 5.
  • Benchmark construction: A hierarchical multi-agent game-development system supports scalable construction, while iterative complexity scaling adds features, interactions, or narrative branches when preliminary bug discovery falls below a threshold.The system simulates a professional game studio through producer and specialized team agents, then increases structural complexity until the detected bug count meets or exceeds τ.
  • Benchmark construction: GBQA contains 30 diverse game environments and 124 human-verified bugs spanning Action, Adventure, Role-Playing, Strategy, Simulation, and Puzzle.The benchmark’s ground-truth bugs are curated for evaluation across multiple core gameplay genres.
  • Discovery difficulty: The benchmark classifies bugs as easy, medium, or hard according to increasing demands for perception, short-horizon rule reasoning, and long-horizon temporal consistency tracking.Its difficulty structure is balanced around medium bugs while retaining meaningful proportions of surface-level and long-horizon defects.
  • Evaluation: Recall is the primary metric because the evaluation matches generated reports to ground-truth bugs and prioritizes defect coverage over avoiding false positives.A critic agent determines semantic correspondence between reports and annotated bugs; false negatives are considered more costly in practical QA workflows.

4 BASELINE AGENT

The baseline agent combines ReAct-based interactive exploration with reflection, verification, feedback grounding, and hierarchical memory to support autonomous bug discovery over extended gameplay sessions. Its memory separates short-term trajectory tracking from persistent cross-session experience to enable long-horizon testing under context constraints.

  • Architecture: The baseline architecture equips LLMs with dynamic exploration, reflective reasoning, feedback grounding, and memory management for extended autonomous bug-discovery sessions.
  • Interactive exploration: At each ReAct step, the agent reasons about the current observation, selects an available tool action, and transitions to the next observation.
  • Reflection and verification: After each transition, step-level reflection checks whether the observed outcome matches the agent’s expected correct game behavior.
  • Reflection and verification: When discrepancies arise, the agent records triggering action, observed and expected behavior, and violation type, then reproduces candidates and promotes only sufficiently confident bugs.Confidence depends on reproducibility and deviation magnitude.
  • Hierarchical memory: The hierarchical memory separates short-term trajectory tracking from long-term experiential accumulation, addressing context-window limits in long-horizon bug discovery.
  • Hierarchical memory: Cross-session summaries preserve explored regions, confirmed bugs, unresolved hypotheses, unexplored branches, and priority targets, guiding testing across multiple restarts.Within sessions, recent interactions remain detailed while older trajectory segments are compressed into summaries preserving semantically critical state information and causal structure.

5 EXPERIMENTS

Experiments show that autonomous bug discovery remains difficult even for frontier LLMs, with performance shaped by reasoning capability, testing mode, exploration budget, and memory. The benchmark and evaluation pipeline also receive strong annotation and critic-agent validation.

  • Main Results: 48.39% is the best reported Recall, achieved by Claude-4.6-Opus in Quality Assurance Mode with 500 steps, leaving over half the bugs undetected.Table 1 reports Recall across both testing modes and four step budgets.
  • Main Results: Qwen3-32B-Thinking reaches 33.87%, outperforming Llama-3.1-70B at 14.52% and rivaling Qwen3-235B-A22B at 18.55%, showing reasoning is more parameter-efficient than scale alone.Performance generally improves with model size, but reasoning capability provides stronger efficiency gains for sustained multi-step bug discovery.
  • Main Results: Quality Assurance Mode consistently outperforms Player Exploring Mode because design artifacts and source code enable specification-driven testing.Experiments use four interaction budgets, T ∈{50, 100, 200, 500}, and evaluate Recall as the primary metric.
  • Validation: Krippendorff’s α reaches 0.901 for the 378 candidate annotations, while GPT-5.2 achieves the highest critic-agent correlation, ρ = 0.903, and becomes the default backbone.These results support the reliability of both human annotations and automated evaluation.
  • Step Budget Analysis: Easy bugs are largely found within 300 steps, Medium bugs reach about 30% at 500 steps, and Hard bugs show no clear saturation trend.The analysis links Easy bugs to perceptual checking, Medium bugs to short-horizon reasoning, and Hard bugs to longer-horizon reasoning.
  • Memory Ablation: The full memory module combines in-session and cross-session memory to reduce repeated states and avoid re-exploration, unlike either component alone.Without memory, recall saturates early; in-session memory leaves cross-session redundancy, while cross-session memory leaves within-session redundancy.

6 CONCLUSION … A.3 TEAM-LEVEL PLANNING PHASE

GBQA is presented as a scalable benchmark showing that state-of-the-art LLMs remain substantially limited in autonomous bug discovery, especially for long-horizon and state-dependent errors. Its hierarchical builder uses a Producer Agent, specialized teams, unified proposals, and dependency-aware planning to construct interactive game environments.

  • 6 CONCLUSION: GBQA exposes a substantial gap between current LLM agent capabilities and real-world quality-assurance demands, particularly for long-horizon and state-dependent bug discovery.The benchmark evaluates autonomous bug discovery in interactive game environments despite LLMs’ strong performance in code generation and repair.
  • A DETAILS OF THE GAME ENVIRONMENT BUILDER: The builder uses a hierarchical, studio-inspired multi-agent architecture rather than conventional prompt chaining to coordinate game-environment development.A Producer Agent maintains global project state, guides downstream development, and compiles the integrated environment after team deliverables are complete.
  • A.1 TOP-DOWN STUDIO ORGANIZATION: Three specialized teams—Design, Programming, and Art—are supervised by leaders who decompose directives, scale workers, validate deliverables, and synchronize progress with the Producer.The leaders are the Lead Designer, Technical Director, and Art Director, respectively, and actively manage execution rather than merely routing messages.
  • A.1 TOP-DOWN STUDIO ORGANIZATION: The teams work in isolated documentation, code, and asset workspaces, making the Producer’s process a distributed multi-workspace pipeline.The workspaces are ./project/docs, ./project/code, and ./project/assets.
  • A.2 PRODUCER-LEVEL PROPOSAL FORMATION: Before team execution, the Producer defines genre, reference titles, narrative and gameplay loops, and aesthetic guidelines in a unified project proposal.The proposal establishes the strategic direction for downstream development.
  • A.2 PRODUCER-LEVEL PROPOSAL FORMATION: The proposal serves as the authoritative specification: Design derives rules, Programming implements the environment and interaction APIs, and Art aligns assets with the specified style.The three teams translate the shared proposal into documentation, implementation, and visual production.
  • A.3 TEAM-LEVEL PLANNING PHASE: Team leaders convert high-level directives into atomic worker tasks by estimating workload and criticality and constructing a Task Dependency and Priority Graph.The graph encodes prerequisites, parallelization opportunities, and resource-aware priorities, with workspace-specific content across Design, Programming, and Art.

A.4 TEAM-LEVEL EXECUTION PHASE · A.5 SHARED SUPPORT PLATFORM AND SKILL BINDING · A.6 WORKSPACE REVIEW AND AGENT LIFECYCLE

The builder executes evolving task graphs through dynamically scheduled, skill-specialized worker agents that operate in isolated team workspaces. Validation-controlled merging and ephemeral agent lifecycles support adaptive, deterministic, and scalable production across Design, Programming, and Art teams.

  • A.4 TEAM-LEVEL EXECUTION PHASE: Dynamic runtime scheduling instantiates workers on demand, assigns ready atomic tasks, and rebalances resources as dependencies resolve, enabling elastic team scaling.Parallelizable operations proceed concurrently while dependent tasks remain queued until prerequisites are validated.
  • A.4 TEAM-LEVEL EXECUTION PHASE: Team leaders act as active schedulers by provisioning agents, enforcing dependency constraints, and optimizing throughput throughout production.The architecture coordinates specialized Design, Programming, and Art teams through structured planning–execution loops.
  • A.5 SHARED SUPPORT PLATFORM AND SKILL BINDING: Task-specific skill bundles are dynamically bound from a centralized Shared Support Platform, decoupling orchestration from capabilities and enabling precise role specialization.The platform organizes general, Design, Program, Art, and Meta Skills into a stratified architecture supporting all three teams.
  • A.5 SHARED SUPPORT PLATFORM AND SKILL BINDING: General, Design, Program, and Art skills provide file, web, document, implementation, asset-generation, and interface-styling capabilities tailored to each team.Examples include terminal and file operations, office-format authoring, game development and React scaffolding, and image-generation and UI/UX tools.
  • A.5 SHARED SUPPORT PLATFORM AND SKILL BINDING: Meta Skills enable runtime creation, refinement, and deletion of capabilities, supporting continuous expansion and adaptation without architectural rewrites.The platform can evolve skills as project requirements change, such as extending image generation with batch-processing pipelines.
  • A.6 WORKSPACE REVIEW AND AGENT LIFECYCLE: Workers commit outputs to team workspaces for structured leader validation against proposals, specifications, and dependencies before approved results are merged and tasks completed.After integration, the planning graph is updated and the worker is terminated.
  • A.6 WORKSPACE REVIEW AND AGENT LIFECYCLE: The standardized instantiate–execute–review–cleanup lifecycle treats agents as ephemeral compute units, avoiding state drift, resource contention, and context pollution.Agents are provisioned only for the active task frontier and decommissioned immediately after deliverables are integrated, supporting deterministic and scalable execution.
  • A.6 WORKSPACE REVIEW AND AGENT LIFECYCLE: CASTLE applies the pipeline by coordinating unified specifications and centralized review across room design, puzzle dependencies, backend implementation, interaction APIs, and visual assets.The Producer, Design, Programming, and Art teams contribute distinct deliverables under shared constraints.

B FRONTIER MODEL PERFORMANCE ON CODE RESOLUTION VS. BUG DETECTION

Frontier models perform strongly on conventional code resolution but degrade substantially on GBQA, revealing a capability gap in autonomous bug discovery. GBQA demands proactive exploration, anomaly detection, and coherent long-horizon reasoning beyond standard benchmarks.

  • B FRONTIER MODEL PERFORMANCE ON CODE RESOLUTION VS. BUG DETECTION: SWE-Bench evaluates localizing and patching known defects from explicit, well-scoped problems, whereas GBQA requires discovering latent anomalies through autonomous dynamic-environment exploration.The benchmarks therefore assess distinct capabilities, with GBQA requiring agents to surface bugs without explicit supervision.
  • B FRONTIER MODEL PERFORMANCE ON CODE RESOLUTION VS. BUG DETECTION: GBQA additionally requires coherent reasoning across long-horizon interactions, introducing compounding challenges that current coding and software-engineering benchmarks do not measure.SWE-Bench Verified scores were extracted from official vendor technical reports and system cards for strict alignment with publicly reported capabilities.
  • B FRONTIER MODEL PERFORMANCE ON CODE RESOLUTION VS. BUG DETECTION: Frontier models achieve strong SWE-Bench Verified results but substantially weaker GBQA performance, highlighting the greater complexity of autonomous bug discovery.Table 4 compares performance on both benchmarks and characterizes the gap as requiring capabilities beyond standard code resolution.

C PROMPT DESIGN IN GBQA

GBQA uses role-specific prompt architectures to coordinate game production, manage scoped implementation work, guide evidence-driven interactive QA, and evaluate predicted bugs against human-verified ground truth. Dynamic runtime variables and structured outputs support consistent execution, auditability, memory, and reproducible assessment.

  • C PROMPT DESIGN IN GBQA: The prompt architecture instantiates role- and context-dependent variables at runtime, defining operational responsibilities and corresponding prompts for each agent type.Placeholders include project context, task specifications, workspace state, skills, tools, and interaction history.
  • C.1.1 GAME PRODUCER AGENT: The Game Producer Agent defines the game’s global direction, including genre, references, concept, mechanics, presentation, player objective, and winning condition.It also specifies compact deterministic environments and stateful QA-relevant mechanics such as inventory rules, locks, visibility constraints, combination logic, and delayed updates.
  • C.1.2 TEAM LEADER AGENT: The Team Leader Agent converts the producer proposal into dependency-aware atomic tasks, schedules workers with suitable skills and tools, and reviews their workspace changes.Planning maintains a Task Dependency and Priority Graph, while execution follows repository-level inspection and explicit acceptance or revision decisions.
  • C.1.3 WORKER AGENT: Worker Agents execute one scoped atomic task within assigned boundaries, using equipped skills and allowed tools while returning concrete, auditable results for leader review.They create pull requests for valid changes and report blockers or no-change status instead of fabricating commits; team differences come from configuration and skill bundles.
  • C.2 PROMPTS FOR BASELINE INTERACTIVE AGENT: The baseline Interactive Agent uses backend APIs to expose reproducible bugs and state inconsistencies through evidence-driven reflection, one-command planning, and memory-supported summaries.Its prompt alternates exploration, local verification, and longer-horizon bookkeeping rather than pure task completion.
  • C.3 PROMPTS FOR EVALUATION: The evaluation Critic judges whether each predicted bug report matches a human-verified ground-truth bug using the supplied bug list and match threshold.This prompt defines structured comparison between agent reports and target-game reference bugs.

D REPRESENTATIVE GAME ENVIRONMENTS

GBQA comprises 30 interactive game environments spanning multiple genres, with CASTLE designated as the primary case study. CASTLE is a deterministic eight-room text adventure whose staged progression, stateful mechanics, backend-only interface, and three human-verified bugs support QA evaluation.

  • Representative Environments: GBQA contains 30 interactive game environments spanning multiple genres and gameplay patterns, with CASTLE selected as the primary case study.CASTLE is presented as a deterministic text adventure game with an eight-room topology.
  • World Structure and Progression: CASTLE uses a compact, non-trivial room graph with prerequisite-gated progression through keys, fragments, tools, and access conditions.The hall serves as a central hub, while the attic becomes accessible from the library only after the ladder is positioned.
  • Stateful Mechanics: CASTLE tests stateful QA mechanics through a six-item inventory limit, staged containers and locks, restricted visibility, and a light-dependent basement mechanic.Players must carry and ignite a valid light source before basement inspection becomes valid.
  • Backend Interface: The QA agent interacts with CASTLE exclusively through backend endpoints for session creation, command execution, and state retrieval.Responses expose textual observations and structured summaries of the room, exits, inventory, flags, turn counter, and visibility status, while excluding hidden developer metadata.
  • Ground-truth Bug List: The CASTLE dataset contains three human-verified bugs covering logic errors, description flaws, and data inconsistencies.These bugs represent distinct QA-relevant failure patterns.

E CASE STUDY: TOWARDS FULLY AUTONOMOUS AGENTIC CODING SYSTEMS

The CASTLE case study integrates autonomous QA and code repair into a fully closed-loop agentic coding system. Across three sessions, the system discovered and repaired all three released bugs without human intervention, achieving 100% discovery and fixing rates.

  • Experimental Setup: The pipeline combines a QA agent that explores CASTLE and generates reports with Claude Code, which modifies the codebase and returns patched versions.Both agents use Claude-4.6-Opus-Thinking, while QA uses in-session and cross-session memory for long-horizon reasoning and experience reuse.
  • Key Observations: Autonomous QA supplies the upstream development signal by discovering defects without human-written issue descriptions, turning QA from passive validation into active exploration.This discovery module targets the missing autonomous bug-discovery component in existing code-generation and bug-fixing benchmarks.
  • Closed-Loop Trajectory: 100% discovery and fixing rates were achieved on CASTLE: all three released bugs were discovered and repaired within three autonomous sessions.Table 7 summarizes the session-level discovery, verification, and remediation trajectory.
  • Closed-Loop Trajectory: Session 2 interleaves verification of prior fixes with discovery of BUG-1, showing that effective QA requires continuous exploration after apparent convergence.Session 1 found BUG-2 and BUG-3; Session 2 verified their repairs and found BUG-1; Session 3 verified the final repair and identified its root cause.
  • Key Observations: Evaluating discovery and repair jointly is necessary because isolated assessment of either component misses the dynamics of the full closed loop.The case study evaluates an end-to-end defect discovery and remediation pipeline rather than either module alone.

F LABELING INSTRUCTIONS · G LLM USAGE STATEMENT

Annotators replay candidate reports against the provided game build and design specification, validating bugs, assigning discovery difficulty, and recording minimal reproduction steps. They distinguish valid defects from duplicates, non-bugs, and uncertain cases using verified behavior, while LLMs were used only for language polishing.

  • F.1 TASK OVERVIEW: Annotators replay each candidate report in the playable game, using the design specification and existing accepted bugs to determine validity, difficulty, and reproduction steps.They may refine incomplete reproduction sequences by replaying nearby interactions, but the final annotation must be grounded in behavior actually verified.
  • F.3 DEFINITION OF A VALID BUG: A valid bug must be reproducible, behaviorally incorrect relative to the specification or clear interface expectations, and caused by the game implementation.Reports based only on difficulty, unsupported commands, unmet prerequisites, or insufficient evidence are rejected as valid bugs.
  • F.4 DIFFICULTY ANNOTATION CRITERIA: Discovery difficulty is assigned only for valid candidates and reflects how hard the bug is to find through gameplay, not how difficult it is to fix.The required output records validity, difficulty, matched bug ID, verified minimal reproduction steps, and a specific behavioral explanation.
  • F.5 DUPLICATE AND NON-BUG HANDLING: Duplicate reports describe the same underlying defect under materially the same reproduction condition, even when their wording differs.Annotators record the matched bug ID and briefly explain why the reports refer to the same issue.
  • F.5 DUPLICATE AND NON-BUG HANDLING: Reproducible behavior consistent with the design specification is labeled non-bug, while irreproducible or unresolved ambiguous cases are labeled uncertain rather than guessed.Examples include intended prerequisite failures, correct puzzle gating, and unusual observations that remain valid under the game rules.
  • F.7 WORKED EXAMPLE: The worked CASTLE example labels early disclosure of a bedside-drawer key as valid, medium difficulty, and BUG-2 after verifying the interaction sequence.The violation is that the room description exposes a hidden item before the relevant container is opened.
  • F.8 IMPORTANT CONSIDERATIONS •: Annotators must judge intended behavior rather than personal preference and provide precise minimal reproduction traces sufficient for another expert to trigger the behavior.They should record the shortest verified sequence that completes the reproduction chain.
  • G LLM USAGE STATEMENT: The authors state that large language models were used solely for language polishing and did not contribute to the research design, analysis, or conclusions.The authors retain full responsibility for the paper’s integrity and content.
Loading 2604.02648v1…