PaperlayerView exact revision on arXiv

Source-linked AI summary

GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?

Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, Zeno Zeng

arXiv:2608.21833v1cs.AIcs.CL

TL;DR

Game development benchmarks often assess only final artifacts or isolated stages, despite the need for logic, presentation, interaction, and playability to work together. GAMEXPERT-BENCH derives three lifecycle stages from human–agent trajectories and evaluates them through complementary execution-grounded tracks. Across the tracks, agents are more reliable at playable foundations and explicit requirements than at defect discovery, runtime verification, and preserving functionality across changes.

  • Problem

    Existing benchmarks often evaluate the final game artifact or an isolated development stage, although playable games require multiple components to work coherently during interaction.

  • Method

    GAMEXPERT-BENCH analyzes complete human–agent trajectories and operationalizes generation, bug diagnosis and repair, and multi-turn optimization as complementary execution-grounded benchmark tracks.

  • Results

    Across the three tracks, agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.

  • Takeaways & Limitations

    Evaluating what games actually do across creation, repair, and optimization provides a broader assessment of coding-agent game-development capability than initial generation quality alone.

  • Takeaways & Limitations

    GAMEGEN uses a blank workspace without provided templates or multimedia assets and does not prescribe a game engine or development toolchain.

Abstract

from arXiv · show

Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.

1 Introduction

Game development requires coherent interaction among logic, presentation, interfaces, and audiovisual elements, so evaluating only code or final artifacts is insufficient. GAMEXPERT-BENCH unifies generation, repair, and multi-turn optimization to assess this lifecycle and finds that explicit implementation is easier than autonomous verification and change control.

  • Playable games require gameplay logic, rendering, controls, interfaces, audiovisual content, and state transitions to function coherently during interaction.
  • Existing benchmarks mainly assess final artifacts or isolated development stages, leaving the complete user-facing development process jointly unevaluated.
  • Qualitative analysis of complete human–agent trajectories identifies generation, bug diagnosis and repair, and multi-turn optimization as recurring lifecycle stages.
  • 97 generation tasks span 11 genres, while 100 controlled repair tasks use 50 confidential Gold Games with 19–27 injected defects per level, and optimization contains 17 chains with 102 requests.
  • Track-specific protocols connect implementation evidence to executable behavior through live interaction, deterministic repair probes, product assessment, and regression checks.
  • Across tracks, agents implement explicit requirements and produce playable foundations more reliably than they discover defects, verify runtime behavior, and preserve functionality across changes.

2 Related Work

Prior work studies game generation, bug discovery and repair, or interactive refinement largely in isolation. GAMEXPERT-BENCH combines these capabilities as complementary tracks spanning the coding-agent game-development lifecycle.

  • Existing game-generation benchmarks evaluate multimodal or runtime behavior in projects, browser-native games, or Godot, generally within single-request final-artifact settings.
  • Game bug benchmarks cover visual quality assurance, autonomous discovery, behavioral GUI testing, and iterative repair, but do not jointly provide a verified clean game.
  • Interactive coding benchmarks reconstruct multi-turn sessions or automated feedback loops, whereas GAMEOPT derives optimization chains from real human–agent game-development trajectories.
  • GAMEXPERT-BENCH formulates generation, repair, and human-guided optimization as three controlled, complementary tracks spanning the lifecycle.

3 GameXpert-Bench Overview

GAMEXPERT-BENCH models coding-agent game development as creation, maintenance, and improvement of an executable artifact. Its three tracks use distinct starting conditions and artifact-grounded evaluation to measure this lifecycle.

  • The Game Development Lifecycle with Coding Agents: The user-facing lifecycle runs from an initial request to a final playable artifact and is derived from qualitative analysis of complete historical human–agent trajectories.
  • The Game Development Lifecycle with Coding Agents: Generation creates the initial playable game, Fix repairs defects, and Optimization improves the game through successive requests while preserving established functionality.
  • Benchmark Tracks: GameGen starts from a natural-language request, GameFix from a mutated Gold Game, and GameOpt from multi-turn optimization based on real human interaction.
  • Artifact Evolution: Across analyzed trajectories, artifact source size retains or increases across lifecycle stages, consistent with later work inheriting and extending earlier results.
  • Benchmark Tracks: The stages are recurring development modes rather than a fixed pipeline, since diagnosis may trigger repair and optimization may expose new defects.
  • Benchmark Tracks: GameGen realizes a design brief in a blank workspace, GameFix uses human-verified mutated games, and GameOpt replays successive product requests across gameplay, design, balance, art, interface, and audio.
  • Artifact-Grounded Evaluation: All tracks evaluate the executable artifact through runtime verification, deterministic gameplay probes, or evidence-grounded final-game criteria with regression accounting.

4 GAMEGEN: Single-Request Game Generation

GAMEGEN evaluates whether coding agents can create complete browser-native games from natural-language briefs in a blank workspace, without prescribed engines or assets. Results show strong playable foundations but persistent gaps in richness, runtime functionality, interface composition, and 3D generation.

  • Task Setting: GAMEGEN requires agents to synthesize complete browser-native games from natural-language briefs without templates, prescribed engines, or pre-existing assets.The benchmark includes 97 games across 11 genres, including 44 tasks requiring 3D rendering, with three difficulty levels spanning single mechanics to multiple interacting mechanics.
  • Evaluation Framework: The evaluation combines static code analysis, dynamic runtime validation, live interaction, behavioral rubrics, and human assessment.Checklist items receive credit only when the intended effect is validated during execution; four dimensions are normalized to [0, 100] and weighted equally.
  • Overall Performance: Claude-Opus-5 ranks first with an overall score of 79.7, leading completeness at 94.4, richness at 72.0, and visual quality at 80.4.Claude-Fable-5 achieves the best player experience at 74.4, while Kimi-K3 and Claude-Opus-4.8 score 71.3 and 69.9 overall.
  • Dimension-wise Performance: Across 15 models, completeness averages 77.5 versus richness at 46.1, showing a persistent gap between playable cores and diverse additional content.Player experience and visual quality average 61.5 and 68.2, respectively, indicating that functional breadth and player-facing polish remain distinct capabilities.
  • 2D versus 3D Game Generation: 3D games average 60.1 overall versus 65.9 for 2D games, a 5.8-point drop, with 14 of 15 models performing worse on 3D.The largest mean degradation is in completeness (−8.8), followed by player experience (−6.0), richness (−5.9), and visual quality (−2.4).
  • Interface Robustness: UI misalignment affects 221 of 1,455 model–game runs (15.2%), and every evaluated model exhibits the problem.The reported rates range from 7.2% for Claude-Fable-5 and 10.3% for Claude-Opus-5 to 19.6% for GLM5.1 and 22.7% for Seed-2.1-pro.
  • Runtime Functionality: Across 43,081 assessed event instances, 2,293 (5.32%) were coded as implemented but failed or were only partial during dynamic evaluation.Load or crash failures account for 56.0% of these cases, motivating runtime interaction alongside static inspection.

5 GAMEFIX: Game Bug Diagnosis and Repair

GAMEFIX is a controlled bug-repair benchmark that tests whether coding agents can diagnose and repair defects while preserving correct game behavior. Its results expose a sharp gap between repairing named bugs and autonomously discovering, verifying, and resolving hidden defects.

  • Benchmark design: GAMEFIX evaluates repair of human-verified web games with automatically injected defects, using deterministic executable behavioral tests rather than subjective grading.Reversible mutations provide exact gold patches and fixed ground truth for each task.
  • Benchmark design: 50 proprietary game levels receive 19–27 bugs each, producing 100 tasks per run across explicit-issue and hidden-defect query modes.The closed, human-reviewed game set reduces contamination concerns and supports scalable extension without re-annotation.
  • Evaluation protocol: Each repair task combines Fail-to-Pass tests for injected defects with Pass-to-Pass tests that detect regressions in previously correct behavior.Tests reset the level, simulate inputs, inspect executable state, and require all parameterized cases to pass.
  • Evaluation protocol: The Cliff measures the drop in macro average@3 from Explicit Issue to Self-Discovery, isolating autonomous bug-discovery ability.The full bug set remains the denominator, so undiscovered objective bugs count as failures.
  • Results: STRICT is difficult to saturate because it is high only when models repair nearly every bug across runs; even Claude Opus 5 reaches just 39.0 out of 100.The median model is near 14, leaving substantial headroom for near-perfect multi-bug repair.
  • Results: The Explicit Issue span is about 13 points, versus about 38 points under Self-Discovery, while the Cliff ranges from 7.6 for Opus 5 to 32.8 for Hy-3.The Cliff narrows as STRICT rises, indicating that stronger models more consistently discover, verify, and control repairs autonomously.
  • Results: The remaining gap reflects six interacting abilities: self-discovery, behavioral verification, value recovery, multi-bug planning, regression control, and stopping criteria.Observed mechanisms include discovery failure and treating the reported checklist as a scope boundary; rabbit-holing and multi-agent conflicts add planning and orchestration failures.

6 GAMEOPT: Human-Guided Game Optimization

GAMEOPT evaluates whether coding agents can improve working games through six-turn, trajectory-grounded request chains while preserving existing functionality. Results show strong overall performance can conceal uneven capabilities, non-monotonic turn behavior, and inconsistent integration across game dimensions.

  • Benchmark design: GAMEOPT uses multi-turn optimization chains to improve mechanics, level layout, balance, visuals, interface, and audio without breaking prior functionality.Requests are grounded in historical human–agent co-creation trajectories and use fixed replay inputs.
  • Benchmark design: 17 self-contained JavaScript games form six-turn chains totaling 102 optimization turns.The collection contains 701 acceptance criteria: 604 positive rubric items and 97 regression checks.
  • Evaluation protocol: Final products are judged using hidden requirements, integration challenges, and regression checks based on code, rendered output, or both.Criteria require admissible evidence from reachable code paths, runtime logs, screenshots, or audio traces; unsupported claims and dead code do not count.
  • Results: Claude-Opus-5 ranks first at 93.96, followed by Claude-Fable-5 at 89.31 and Kimi-K3 at 84.66.Across 15 models, the median is 74.24, with a 58.07-point range from Claude-Opus-5 to Seed-2.1-Pro.
  • Results: Numeric balance (66.11) and level design (66.36) have the lowest six-dimensional means, while audio is highest (81.97).Claude-Opus-5 ranges from 80.80 in balance to 100.00 in art, showing that high overall performance does not imply an even profile.
  • Results: Turn-level performance is non-monotonic, ranging from 77.30 at turn one to 81.97 at turn six, with every model reaching its lowest value at turns two, three, or four.Turn and content are confounded, so the pattern does not establish long-context stability.

7 Conclusion

The conclusion presents GAMEXPERT-BENCH as an execution-grounded evaluation of coding agents across generation, repair, and human-guided optimization. Its central finding is that plausible implementations are easier than rich, verified, and regression-free games, with the appendix documenting corpus composition and evaluation details.

  • Conclusion: GAMEXPERT-BENCH evaluates coding agents across game generation, diagnosis and repair, and human-guided optimization.The benchmark is execution-grounded and assesses what games actually do, not only what agents write.
  • Conclusion: The benchmark’s central finding is that producing a plausible implementation is easier than delivering a rich, verified, and regression-free game.The conclusion frames this gap across all three development stages.
  • Corpus and appendix: The appendix provides representative prompts, shared rubrics, subset leaderboards, and per-game event pass rates for GAMEGEN.The 2048 rubric separates Completeness from Richness, while Tables 12 and 13 report 2D and 3D subset results.
  • Corpus and appendix: GAMEGEN contains 97 games across 11 categories, including 53 2D and 44 3D games.Simulation and Management is the only category without a 3D task.
  • Corpus and appendix: Per-game event pass rates reveal substantial task-level variation that aggregate model scores hide.Strong models encounter isolated difficult games, while lower-ranked models sometimes perform well on particular tasks.

B.1 F2P and P2P test definitions

GAMEFIX defines repair success through behavioral tests that distinguish restoring broken behavior from preserving unaffected behavior. Its authoring gate isolates co-injected bugs and requires stable, non-contaminating tests before tasks are shipped.

  • Test definitions: A test case is a hand-written boolean predicate over observable runtime behavior, expressed as a tolerance band rather than equality with a gold constant.The gold patch certifies test construction but does not determine whether a candidate repair receives credit.
  • Test definitions: Fail-to-Pass tests require injected-bug behavior to recover, while Pass-to-Pass tests require previously correct behavior to remain unchanged.A bug counts as fixed only when its F2P test passes and all associated P2P tests remain passing.
  • Multi-bug isolation: 19–27 co-injected bugs per task require isolation checks to prevent neighboring mutations from masking failures or creating spurious regressions.The authoring gate tests gold-green, coupled-base-red, single-bug-red, and P2P survival conditions.
  • Multi-bug isolation: Bugs failing isolation or stability checks are replaced until each task contains at least 20 isolated bugs.Gold-game P2P behavior must remain green for five consecutive runs with zero flakiness.

B.2 The six agentic abilities the benchmark measures

The benchmark decomposes game bug repair into six agentic abilities because multi-bug tasks require several distinct skills to operate together. Behavioral verification and regression control are linked aspects of running the game through completion, while explicit issue reports reduce several discovery demands.

  • Six agentic abilities: The benchmark separates game-repair skill into six agentic ability axes rather than treating repair as a single code-editing ability.Each 19–27-bug task requires multiple abilities to act together because mutations are behaviorally visible only during play.
  • Six agentic abilities: Behavioral verification and regression control are complementary sides of running the game to completion.Verification concerns building a real test harness; regression control concerns what that harness catches.
  • Six agentic abilities: Five of the six ability axes appear mainly under Self-Discovery, whereas Explicit Issue reports supply localization and a completeness target.Explicit Issue tasks also include a guard against silent regression.

B.3 Full leaderboard with secondary metrics

Table 15 reports secondary performance, resource-use, and completion metrics for all 17 evaluated models. The metrics are derived from the same live report module as the main leaderboard, with some fields unavailable for frameworks that do not expose them.

  • Table 15 reports pass@3, pass3, average@3 counts at thresholds 100/95/90, mean turns, and uncached input/output tokens.pass@3 means any of three runs solves the task, while pass3 means all three runs solve it.
  • 17 models are included in the secondary-metrics table.
  • Counts are out of 100 tasks for pass@3 and pass3, or expected tasks for average@3.
  • Models evaluated under two agent frameworks are listed once per framework, while other models use Claude Code.

B.4 Reasoning-level trajectory analysis

The trajectory analysis uses only signals exposed by each model provider and framework. Because frontier models generally withhold raw chain-of-thought traces, the analysis instead examines observable planning, outputs, tool use, edits, and outcomes.

  • Trajectory analysis excludes hidden reasoning and uses provider-exposed signals instead.These signals include reasoning summaries or planning preambles, visible outputs, tool-use sequences, code edits, and final task outcomes.
  • The analysis relates observable signals to failure cases across selected tasks.

C GAMEOPT Details

The GAMEOPT appendix details a 17-chain JavaScript collection evaluated in an offline browser sandbox. It summarizes chain inventory and rubric composition, with rendered output sometimes required as acceptance evidence.

  • Evaluation setting: Each starting snapshot runs in the offline browser sandbox.Acceptance criteria may therefore require rendered output as evidence.
  • Supplementary examples: The appendix provides three request chains and one complete hidden rubric as representative supplementary examples.

C.1 Track Overview

GAMEOPT evaluates multi-turn optimization chains using structured acceptance criteria and regression checks. The examples span game dimensions and show that evaluation extends beyond satisfying the visible request to preserving behavior and meeting hidden requirements.

  • Track overview: GAMEOPT contains 17 chains, 102 turns, and 701 acceptance criteria.The intended budget is one P0 criterion and one regression check per turn.
  • Track overview: Acceptance-criterion percentages use all criteria as the denominator.diff_analysis criteria derive from adjacent human-version changes, while real_user_followup criteria restate later complaints from the same user.
  • Track overview: All 17 chains cover every game dimension exactly once, but AUDIO is always turn six.This partially confounds turn position with dimension, limiting separation of those effects.
  • Representative chains: The request chains span interface, gameplay, level design, balance, art, and audio optimization.Table 20 reports six requests for each representative example, including total, P0, and rendered-output criterion counts.
  • Hidden rubric: The hidden Hero Arena rubric contains seven criteria, including three P0 criteria.Two P0 criteria come from later user follow-ups, and another adds throttling from an adjacent human-version diff.
  • Hidden rubric: Satisfying the visible request is only the first evaluation layer.The hidden rubric also checks explicit trigger-to-output behavior, throttling, and regression-sensitive conditions.
Loading 2608.21833v1…