Source-linked AI summary
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, Chris Donahue
TL;DR
Multimodal agents lack a challenging, verifiable benchmark for game development, where dense codebases and visual assets must be handled together. The paper introduces GameDevBench from web and video tutorials and evaluates agents across these tasks. Agents perform poorly, especially on more multimodal tasks, while simple screenshot and video feedback improves GPT-5.4 from 41.1% to 52.0%.
Problem
Multimodal-agent progress and evaluation testbeds lag behind coding-agent progress, despite game development combining complex software projects with visual and temporal assets.
Method
The paper introduces GameDevBench, a Godot benchmark built from web and video tutorials, with deterministic tests and screenshot- and video-based feedback methods.
Results
Agents struggle most on tasks requiring deeper multimodal understanding, while visual feedback raises GPT-5.4 from 41.1% to 52.0% pass@1.
Takeaways & Limitations
GameDevBench exposes a sharp gap between current agent capabilities and the multimodal demands of game development, while simple visual feedback improves performance.
Takeaways & Limitations
Agents frequently select incorrect images or sprites, such as walking sprites instead of attacking sprites, when tasks require multimodal understanding.
Abstract
from arXiv · showhide
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.
1. Introduction
GameDevBench targets game-development agents in a domain that combines dense software projects with multimodal assets and visual scene understanding. The benchmark shows that agents remain challenged by these tasks, while screenshot- and video-based feedback improves performance.
- Game-development tasks combine large projects, static and temporal visual elements, and deterministic code-based verification.
- GameDevBench is presented as the first benchmark for evaluating agents on complex, multimodal game-development tasks in a modern game engine.
- Tasks span sprite animations, character controllers, colliders, shaders, particle effects, and other common game-development topics.
- Tasks average more than three times the code changes of SWE-Bench and are evaluated with Godot tests plus verified reference solutions.
- 53.8% is the best agent success rate with multimodal support, while average success falls from 51.4% on gameplay tasks to 33.0% on 2D graphics tasks.
- Screenshot feedback through an MCP server and recorded game-scene video both improve agent performance across almost all evaluated models.
2. Benchmark Construction
The benchmark is constructed by converting Godot tutorials and their solution repositories into tested game-development tasks. Human review then checks task correctness, resolvability, ambiguity, and test strictness.
- 2. Benchmark Construction: GameDevBench tasks are distilled from online tutorials because tutorials provide common use cases and accompanying solution repositories.
- 2.1. Stage 1: Data Preparation: All tutorials are filtered to Godot 4 materials with corresponding GitHub repositories using permissive open-source licenses.
- 2.1. Stage 1: Data Preparation: The video source began with 102 YouTube tutorials, averaged 29 minutes each, and yielded 57 usable tutorials after filtering.
- 2.1. Stage 1: Data Preparation: The web source produced 99 tutorial folders containing tutorial text, downloaded visual media, GitHub repositories, and metadata.
- 2. Benchmark Construction: Tasks must follow tutorial instructions, derive files from existing repositories, and test only explicitly requested features.
- 2.4. Stage 4: Human Annotation.: A preliminary review found 43% of 41 tasks issue free, while 50% required minor updates.
- 2.4. Stage 4: Human Annotation.: Eight human annotators reviewed all tasks for correctness, resolvability, ambiguity, conflicting instructions, overly strict tests, and other issues.
3. GameDevBench
GameDevBench is designed as a diverse, multimodal, and complex benchmark for Godot game-development tasks. It combines varied task and editor categories with deterministic testing and flexible solution strategies.
- Task categories: Tasks are categorized by skill set into 2D graphics and animation, 3D graphics and animation, gameplay logic, and user interface.
- Task categories: Tasks are also categorized by editor type, using the scene, script, and contextual editors as a proxy for the editors users would need.
- Benchmark features: 82.4% of tasks contain additional assets such as images, fonts, shaders, audio, or other resources, directly testing multimodal capabilities.
- Task diversity: 333 tasks span 2D graphics and animation, 3D graphics and animation, user interface, and gameplay logic.
- Benchmark features: Reference solutions average 4.7 files, 114.1 changed lines, and 3.2 distinct filetypes, exceeding SWE-Bench complexity by more than threefold in lines and file changes.
- Verification: Godot’s testing framework directly checks multimodal game behaviors, while test-based verification supports comparing code-generation and editor-based solution strategies.
4. Evaluation
GameDevBench evaluates models and harnesses on complex game-development tasks, testing both code-based work and multimodal feedback. Agents perform best on gameplay logic, struggle more with multimodal tasks, and generally improve when given screenshots and video, though costs and harness effects vary.
- Evaluation setup: Agents solve game-development tasks using local CLI harnesses that provide access to project files and the Godot application.The evaluation uses native model harnesses and OpenHands for selected cross-harness comparisons.
- Multimodal feedback: Screenshot and video configurations expose editor state, scene visuals, and temporal gameplay information to agents.The screenshot MCP shows the scene, node tree, and inspector, while runtime video captures temporal elements and the camera view.
- Results by skill: 51.4% of tasks are solved for gameplay logic, compared with 33.0% for 2D graphics and animation and 32.0% for UI.3D graphics tasks fall between these categories at 38.4%, using averages across eight agents with multimodal feedback.
- Results by feedback: 41.1% to 52.0% is GPT-5.4’s performance increase when both MCP and video feedback are provided.Multimodal tooling improves performance across almost all models, with exceptions for GPT-5.4 Mini in OpenHands and Gemini 3 Flash.
- Harness comparison: Harness effects vary by model: Gemini 3 Flash falls from 45.4% to 30.3% in OpenHands, while Claude Haiku 4.5 rises from 16.5% to 17.7%.GPT-5.4 Mini shows mixed results across baseline and multimodal conditions.
- Cost: Multimodal feedback usually increases cost, with GPT-5.4 reaching up to a 3.3x increase when feedback is enabled.Gemini and Claude generally incur smaller cost increases, while Gemini 3 Flash in its native harness is identified as most cost-efficient.
- Error analysis: Agents commonly fail to understand multimodal assets and game-development patterns such as node hierarchies, signals, and resource assignments.Observed errors include selecting incorrect sprites, placing nodes at incorrect tree levels, dropping signals, and assigning resources incorrectly.
5. Related Works
Prior agentic benchmarks are mostly unimodal or focused on narrow multimodal settings, while GameDevBench evaluates full game development by combining software-style code output with multimodal inputs. This distinguishes it from game-playing and single-feature game-development systems.
- Agentic benchmarks: Most software-development benchmarks remain largely unimodal, while multimodal software benchmarks primarily target frontend JavaScript.Multimodal agents are also commonly evaluated on computer use and web navigation, where they operate in an action space.
- GameDevBench: GameDevBench combines multimodal input with code output, bridging software-development and computer-use evaluation settings.Game development requires agents to operate on visual game artifacts while producing code-based solutions.
- Game playing: Game-playing research treats games as interactive environments for evaluating AI capabilities, including systems such as Deep Blue, AlphaGo, Cicero, and SIMA 2.This line of work evaluates gameplay or interaction rather than constructing complete game projects.
- Game development: Prior game-development work often replaces or generates a single component, including characters, physics, assets, or levels, rather than combining them in a game engine.GameDevBench directly evaluates the integrated capability of developing a full game.
6. Conclusion
GameDevBench shows that current agents struggle with game-development tasks, especially those requiring deeper multimodal understanding, and that frontier models outperform non-frontier models substantially. Simple screenshot-and-video feedback improves performance, including a 10.9 percentage-point gain for GPT-5.4.
- Contribution: GameDevBench is the first benchmark for evaluating agents on game-development tasks and is built by converting YouTube and web tutorials into benchmark tasks.The benchmark targets an evaluation gap involving complex game-development work.
- Findings: Agents struggle especially with tasks requiring deeper multimodal understanding, with frontier and non-frontier models differing by up to 48.7% pass@1.This finding summarizes the benchmark’s principal performance pattern.
- Visual feedback: 10.9 percentage points is GPT-5.4’s pass@1 gain, rising from 41.1% to 52.0% with screenshot and video feedback.The increase corresponds to a 26.5% relative improvement.
- Implications: The findings highlight improving multimodal capabilities through training or visual-feedback methods as an important direction for agent development.The paper also speculates that addressing these needs could improve performance beyond software and game development.
Impact Statement
GameDevBench evaluates multimodal game-development agents in a domain combining large-scale software engineering with rich multimodal reasoning. Its task-construction process converts tutorials and repositories into independent, testable tasks with documented evidence and validation.
- GameDevBench targets game development as a multimodal agentic domain combining large-scale software engineering with rich multimodal reasoning.
- Task creation uses tutorial transcripts for learning objectives and GitHub repositories as ground truth for completed implementations.Repositories are adapted to the benchmark structure rather than copied verbatim.
- Tasks are designed to span diverse skills and categories, with multimodal reasoning tasks such as spritesheet processing and animation-sound alignment especially desirable.
- The workflow tests source repositories, analyzes their structure and dependencies, creates ground-truth implementations, and validates tasks against explicit instruction evidence.Validation includes checking that tests map to requested features and that instruction steps point to transcript or repository evidence.
- The task-construction guide converts tutorials and repositories into specific, testable tasks centered on node creation and inspector configuration.Tasks are documented with source evidence, requested nodes or properties, validation criteria, file modifications, difficulty, categories, and multimodal status.
E. Task Examples
GameDevBench examples span graphics, animation, 3D scene construction, user interfaces, gameplay logic, and reusable game objects. Tasks can be completed through editor actions or direct code and scene-file edits.
- GameDevBench tasks can be solved either through editor actions or by directly editing code and scene files.The figures match editor actions to the corresponding file modifications.
- Isometric Crusader Animation: The character example combines physical collision with animated sprites, including multiple named animation loops and a feet-aligned rectangular collider.
- 3D Scene Construction: The 3D example populates an empty scene with environment lighting, a shader-driven water plane, background spheres, and a camera.
- Menu System: The menu example builds Launch, Pause, and Restart screens with styled controls, a transition overlay, exported references, and button signal connections.
- RTS Unit: The RTS-unit example combines a sprite, collision shapes, neighbor-avoidance detection, and a selection-highlight aura shader, with scripting as its main focus.
F. Task Statistics
GameDevBench includes task statistics capturing substantially varied skill demands and distributions. Some sprite-animation tasks require processing thousands of sprites, and reported means exclude values beyond 3 standard deviations.
- GameDevBench task statistics reflect drastically different skill requirements and data distributions across tasks.
- Some sprite-animation tasks require processing thousands of sprites.
- Table 3 reports means after excluding values more than 3 standard deviations from the mean.
G.1. Common Game Development Patterns
A representative Godot failure shows that agents may identify the correct property and value but place them under the wrong scene component. The error reflects confusion about property ownership between nodes and sub-resources.
- GPT-5.4 writes the correct sub_emitter property and NodePath value but places them under ParticleProcessMaterial instead of GPUParticles2D.
- The sub_emitter property belongs to GPUParticles2D, so placing it on the material resource makes the configuration meaningless.
H.1. Full Benchmark Result with Token Count and Cost
The full benchmark evaluates agent performance across core and multimodally demanding variant tasks, while reporting token usage and cost for efficiency comparisons. Nearly every model configuration performs worse on hard tasks, with Gemini 3 Pro as the exception.
- Full benchmark results: Stronger models generally achieve higher pass@1, and multimodal inputs improve performance across models.Table 4 also reports average token usage and cost as reference measures for effectiveness and efficiency.
- Full benchmark results: Full pass@1 is lower because task variants explicitly increase the requirement for multimodal understanding.The core set comes from the benchmark pipeline, while the full set adds variants derived from core tasks.
- Efficiency reporting: Tokens and costs are reported as per-task medians over all records contributing to each result row.Token quantities use k = 1,000 tokens.
- Difficulty split: Almost every model configuration performs worse on the hard subset.The benchmark annotates tasks into easy and hard subsets.
- Difficulty split: Gemini 3 Pro improves on the harder subset, which the authors hypothesize reflects strong inherent multimodal understanding.This is the only reported exception to the general hard-subset degradation.
I. Failure Analysis
The failure analysis identifies incorrect game-development patterns and multimodal-grounding failures as the most common error modes, while a contamination check finds little evidence of tutorial memorization.
- Failure modes: Failures most often involve incorrect game-development patterns or failures of multimodal grounding.The analysis covers the four best-performing model configurations and uses an LLM-as-a-judge procedure.
- Failure modes: Failure-mode percentages are computed over total failures and are not mutually exclusive.A single failed task may exhibit multiple error modes.
- Contamination check: The contamination test provides the first half of each of 117 tutorial transcripts and compares generated continuations with held-out text using ROUGE-L and BLEU-4.A score of 1 would indicate perfect memorization.
- Contamination check: Both evaluated models obtain very low continuation-overlap scores, suggesting no evidence of systemic memorization.The comparison uses the true held-out continuation as reference.