Source-linked AI summary
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, Benyou Wang
TL;DR
Evaluating agents that generate playable games requires testing complete artifacts in real engines through observable interaction, not ordinary code inspection. GameCraft-Bench applies this framework to 140 Godot tasks, finding that the strongest agent reaches only 41.46% and agents still struggle to produce coherent, playable games.
Problem
Existing benchmarks do not jointly evaluate real-engine grounding, complete game artifacts, and interaction-based playability.
Method
GameCraft-Bench evaluates complete Godot projects across 140 tasks and 15 game families using replayed interactions and rubric-guided multimodal assessment.
Results
41.46% overall is achieved by the strongest agent, while most agents score below 40% and struggle across mechanics, content, visuals, and presentation.
Takeaways & Limitations
Current frontier coding agents can produce recognizable mechanics or runnable prototypes but remain far from reliable end-to-end generation of coherent games.
Takeaways & Limitations
The benchmark relies on a multimodal judge, so results may be affected by judge-model bias, API drift, or limitations in visual understanding.
Abstract
from arXiv · showhide
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.
1 Introduction
GameCraft-Bench frames end-to-end game generation as producing complete, engine-native interactive systems whose gameplay is verified through executable interaction. It instantiates this evaluation in Godot with 140 tasks across 15 game families and finds that current coding agents remain far from reliable end-to-end generation.
- Desiderata for Game Generation: End-to-end game generation requires Engine Grounding, Artifact Completeness, and Interactive Verification.These desiderata respectively concern development in a real game engine, delivery of a launchable game with necessary artifacts, and evaluation through gameplay interaction.
- The Existing Benchmarks Fail to Meet the Desiderata: Existing benchmarks satisfy parts of this contract but do not jointly cover real-engine development, complete artifacts, and interaction-based evaluation.OpenGame-Bench targets web games without gameplay-interaction judgment, while GameDevBench uses real engines but studies only part of the setting; WebGameBench remains outside real-engine development.
- How GameCraft-Bench Fills the Gap: GameCraft-Bench uses Godot, complete launchable projects, and replayed interactions to evaluate generated games rather than relying on static inspection alone.The benchmark is designed to jointly satisfy the three evaluation desiderata through an interaction-grounded framework.
- How GameCraft-Bench Fills the Gap: 140 tasks across 15 game families comprise GameCraft-Bench, requiring agents to generate complete game projects from natural-language specifications.The benchmark covers diverse gameplay requirements and instantiates the proposed framework in Godot.
- Observations on GameCraft-Bench: 41.46% overall is the strongest agent’s score, while most evaluated agents score below 40%.Failures span core mechanics, content depth, functional visuals, art and presentation, visual feedback, and task-completion behavior, with performance varying across game families.
2 What Should Be a Good Game Generation Benchmark?
A good game-generation benchmark must evaluate the complete mapping from a natural-language specification and concrete runtime environment to a playable artifact. It therefore requires Engine Grounding, Artifact Completeness, and Interactive Verification, with success determined by observable responses to player actions.
- Problem Definition: AI game generation maps a high-level specification to a playable artifact produced within a development and runtime environment.The specification includes rules, mechanics, goals, content, and presentation; the artifact must be launchable in the target environment.
- Three Desiderata: The benchmark’s three desiderata are Engine Grounding, Artifact Completeness, and Interactive Verification.These constrain where the game is constructed, what artifact is delivered, and how it is judged.
- Engine Grounding: Engine Grounding preserves engine-level semantics, including scene hierarchy, scripting lifecycle, asset loading, input dispatch, physics, rendering, configuration, and launch procedures.This forces agents to solve integration problems in a real game engine rather than imitate game logic through abstract programming.
- Artifact Completeness: Artifact Completeness evaluates the complete launchable game project rather than isolated scripts, assets, prototypes, or scenes requiring human assembly.Required components include project metadata, entry scenes, scripts, assets, UI, input mappings, configuration files, and runtime resources.
- Interactive Verification: Interactive Verification judges the action–response loop, exposing failures such as unresponsive controls, incorrect collisions, inactive enemies, unreachable objectives, missing feedback, and broken state transitions.These failures may remain invisible until the game is actually played.
3 The GameCraft-Bench
GameCraft-Bench evaluates end-to-end game generation in Godot by requiring complete game projects and replayable interaction traces, then judging executable gameplay evidence against hidden rubrics. Its benchmark covers 140 tasks across 15 game families and emphasizes full artifact delivery, interactive verification, content depth, and presentation.
- End-to-End Evaluation: GameCraft-Bench requires agents to submit a complete Godot game project and replayable demonstration traces, with traces serving as standardized evaluation evidence rather than game content.The verifier launches the project, replays traces, and observes gameplay in the Godot runtime.
- End-to-End Evaluation: A build gate assigns BUILD = 0 and a final score of zero when the project fails to launch or no submitted trace can be parsed.Scoring begins only after the submission becomes an executable artifact.
- End-to-End Evaluation: Replay converts fixed mouse and keyboard traces into standardized gameplay evidence, avoiding dependence on the verifier’s autonomous exploration policy.Each valid trace is replayed in a fresh Godot instance using a fixed 1280 × 720 viewport.
- Scoring: The hidden rubric scores Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation, weighted 0.15, 0.35, 0.15, and 0.35, respectively.Content Depth and Art and Presentation receive priority because complete games should extend beyond minimal mechanics and functional prototypes.
- Benchmark Composition: The benchmark contains 140 tasks across 15 game families spanning demands such as continuous control, state management, progression, exploration, and narrative or presentation-heavy interaction.The task suite is authored by 12 experienced annotators using public specifications and hidden requirement-level rubrics.
4 Benchmarking Results
Across 140 tasks and seven frontier coding-agent configurations, end-to-end game generation remains highly challenging: the strongest completed configuration reaches only 41.46% overall. Agents perform better on mechanics than on content and presentation, often producing partial or recognizable local interactions without complete, coherent systems.
- Experimental Setup: Seven configurations were evaluated on all 140 GameCraft-Bench tasks under the same benchmark interface and environment.The configurations span Claude Code, Codex, Kimi Code, and Code Buddy with the listed frontier models.
- Main Results: 41.46% overall was achieved by the strongest completed configuration, Opus-4.7 high under Claude Code.GPT-5.5 high followed at 39.49%, Kimi-K2.6 reached 30.65%, MiMo-V2.5-Pro 24.10%, and DeepSeek-V4-Pro 2.15%.
- Main Results: DeepSeek-V4-Pro often violated the demonstration-trace requirement, creating a large gap between build pass rate and valid-trace rate.Build pass measures launchability, whereas valid trace requires at least one submitted demonstration to be replayable.
- Category-level Results: 55.34% on Core Mechanics was Opus-4.7 high’s score, compared with 39.48% on Content Depth.GPT-5.5 high scored 54.36% and 38.61%, while Kimi-K2.6 scored 39.76% and 28.07% on the same categories.
- Category-level Results: Mechanics are consistently stronger than content and presentation, indicating that agents can often create partial interaction loops but struggle to assemble complete, coherent interactive systems.The benchmark pattern is also described as recognizable local mechanics that fail to become complete games.
5 In-depth Analysis
The analysis shows that playable game quality depends on rendered-feedback debugging and closing the build–replay–evaluation loop, not merely on code or tool volume. The multimodal judge is stable across repeated evaluations, broadly aligned with humans but mildly permissive, while rubric categories are unevenly coupled.
- Perception-Guided Debugging: Rendered interaction exposes player-facing failures invisible in source code and helps agents debug issues such as camera framing, UI readability, visual feedback, and level layout.Without rendered feedback, agents may continue editing while unable to assess whether the player-facing game state is coherent.
- Tool Usage: 56.3% of MiMo-V2.5-Pro’s tool calls are shell executions across 140 tasks, but execution effort and tool volume weakly predict final playability.Figure 8 reports near-zero score correlation with total tool calls, r = +0.016, while code-reading and editing account for 16.5%.
- Judge Stability: 0.0037 is Kimi-K2.6’s standard deviation on Card Game over repeated GPT-5.5 evaluations, versus 0.0038 on Simulation, indicating stable judge scores.Opus-4.7 high has standard deviations of 0.0050 on Card Game and 0.0036 on Simulation; these variations are smaller than performance gaps across agents and families.
- Human Calibration: The multimodal judge broadly aligns with human scores but is slightly more permissive overall, with humans stricter on Content Depth and Art and Presentation and the judge stricter on Functional Visuals.At the family level, Card Game is nearly matched between human and judge scores.
- Rubric Correlations: Core Mechanics correlates with Content Depth at r = 0.61 and Functional Visuals at r = 0.53 for Kimi-K2.6, whereas Art and Presentation correlates weakly with Functional Visuals at r = 0.11.The reported pattern suggests that stronger interaction loops often expose more game state and feedback, while presentation is less coupled to other categories.
6 Related Work
Related work spans autonomous software-engineering agents, interactive GUI-agent evaluation, and emerging game-generation benchmarks. Existing game benchmarks evaluate different parts of the end-to-end contract, motivating GameCraft-Bench’s interaction-grounded assessment.
- Coding Agents and Software Engineering Evaluation: Coding agents have progressed from code-completion models to autonomous systems supporting repository navigation, multi-file editing, and iterative debugging.Frameworks including SWE-agent, OpenHands, ChatDev, and MetaGPT evaluate agents on realistic software-development tasks.
- GUI Agents and Interactive Evaluation: GUI-agent benchmarks evaluate end-to-end task completion in desktop and web environments through visual perception and native interface actions.Mind2Web, WebArena, and OSWorld emphasize interactive settings, while CogAgent and OSAtlas demonstrate computer-use behaviors.
- Game Generation Benchmarks: Existing game-generation benchmarks cover different parts of the evaluation contract: OpenGame-Bench targets web games with primarily static or page-level judgment, while GameDevBench studies localized tutorial-derived edits in existing Godot projects.These benchmarks therefore differ from evaluating complete, interactively playable games generated from specifications.
7 Conclusion
The paper introduces an interaction-grounded framework and GameCraft-Bench for evaluating end-to-end game generation in Godot. It argues that reliable evaluation must preserve the engine environment, require complete artifacts, and judge gameplay through interaction, because current agents remain unreliable.
- Conclusion: The framework evaluates end-to-end game generation through interaction rather than treating it as ordinary code synthesis.Its evaluation is grounded in the resulting playable artifact and observed gameplay.
- Conclusion: GameCraft-Bench instantiates the framework as a Godot-based benchmark for coding agents.The benchmark targets game-generation systems operating within a real game engine.
- Conclusion: The benchmark contract preserves the engine environment, requires a complete artifact, and judges the artifact through interaction.These three requirements define the paper’s central evaluation claim.
- Conclusion: Current frontier agents remain far from reliable: they often produce recognizable mechanics or runnable prototypes but struggle to assemble coherent games.The conclusion distinguishes partial functionality from complete, coherent game outcomes.
Limitations
GameCraft-Bench is scoped to lightweight, reproducible 2D Godot evaluation rather than broader game-development settings. Its replay-based multimodal judging may have evaluation biases and does not measure subjective fun.
- Benchmark scope: The benchmark covers 2D game generation in Godot but excludes Unity, Unreal, 3D games, multiplayer, large-scale physics, and long-form production workflows.This scope supports lightweight, reproducible headless evaluation.
- Benchmark scope: Its current verifier scores visual gameplay evidence only, limiting evaluation of audio-dependent aspects of some game types.The supplied passage specifically identifies rhythm, horror, and sports games as examples.
- Evaluation limitations: Multimodal judging of replayed gameplay evidence may be affected by judge-model bias, API drift, and limitations in visual understanding.These factors can influence evaluation results even when the gameplay evidence is unchanged.
- Evaluation limitations: The benchmark does not measure whether a generated game is subjectively fun.Instead, it assesses specification adherence and realization of requested mechanics, content, visual state, and presentation in an executable artifact.
A Evaluation Details
GameCraft-Bench evaluates submissions in a fixed Godot sandbox through replayable demonstrations and multimodal video judging. The protocol specifies constrained project and trace formats, deterministic replay, and rubric-based scoring over sampled gameplay recordings.
- Runtime Environment: The reference environment uses Ubuntu 22.04, Python 3.12, Godot 4.6.2, Xvfb, xdotool, and ffmpeg in a local subprocess sandbox.Each task allows 7200 seconds for the agent and 1800 seconds for verification.
- Submission Format: Each submission includes a Godot project and one to ten JSON demonstration traces containing timed mouse or keyboard events in a fixed 1280×720 viewport.Traces can deterministically initialize states such as battles, upgrade screens, or late-game setups.
- Replay and Judge: For every valid trace, the verifier launches a fresh Godot instance, replays inputs at 30 frames per second, records video, and samples frames every 0.5 seconds.The default configuration evaluates at most ten demos and a deterministic 20-second window per demo.
B Full Family Results
Table 6 reports family-level and benchmark-level results for each model, covering core mechanics, content depth, functional visuals, art and presentation, and overall score.
- B Full Family Results: Table 6 evaluates each model on core mechanics, content depth, functional visuals, art and presentation, and overall score.Results are reported at both the family and benchmark levels, with values expressed as percentages.
C Case Study
The Strategy-Skirmish case study specifies a complete, polished dark-fantasy tactical game rather than a prototype, centered on deliberate, lethal squad battles. Its rubric evaluates whether the resulting artifact combines playable mechanics, meaningful content depth, readable visuals, and coherent presentation.
- Evaluation Rubric: The rubric groups evaluation into Core Mechanics, Content Depth, Functional Visuals, and Art & Presentation.Examples require a playable tactical movement loop, at least three functionally distinct controllable unit types, visually distinct battle states, and a consistent authored dark-fantasy style.
- Game Specification: The instruction requires a complete, shippable dark-fantasy tactical skirmish in Godot 4, framed as a polished vertical slice rather than a prototype.The requested experience uses a limited palette, high contrast, and a tense tactical-survival tone.
- Game Specification: The core loop has players position an outnumbered squad in turn-based battles while enemies advance, flank, and attack after End Turn.Movement ranges are shown on the grid, and control returns to the player after the enemy phase.
- Game Specification: Combat is lethal and readable: attacks use range, reduce persistent HP, remove dead units, and make specialist differences central to tactical decisions.Specialists may differ in movement range, attack patterns, HP pools, or abilities.