PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Game: Can Coding Agents Build the Games We Want?

Xiaoyu Chen, Lai Wei, Jin Wang, Xiangyu Zou, Ruochen Fan, Enze Luo, Mingzhe Yao, Jiahui Zhu, Yuhua Wen, Linghe Kong, Weiran Huang

arXiv:2609.33678v3cs.AI

TL;DR

Game evaluation must distinguish visible outcomes from the interactions that produce them. SWE-Game benchmarks five development activities using executable reference games, shared runtime instrumentation, and visual assessment. Opus5 leads across all five task types, while executable checks align more closely with human-labeled behaviors than a video-based VLM judge.

  • Problem

    Visible gameplay alone may not show whether the implementation reproduces required interactions, so evaluation needs explicit behavioral targets and execution evidence alongside visual assessment.

    Source: §1 · p. 2

  • Method

    SWE-Game derives 247 tasks from 41 executable reference Godot games and uses shared instrumentation to test independently implemented games, complemented by visual rubrics for construction tasks.

    Source: §1 · p. 2

  • Results

    Opus5 achieves the highest overall score in all five task types, and executable checks reach 92.59% balanced accuracy versus 78.41% for a video-based VLM judge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across six models, results show a persistent gap between producing playable projects and faithfully implementing intended mechanics and content.

    Source: §5 · p. 10

    The benchmark has 41 reference games, limited coverage of large projects and long-term development, and evaluates each model with a single agent framework.

    Source: LIMITATIONS AND FUTURE WORK · p. 10

Abstract

from arXiv · show

We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five task types cover development from a brief, implementation from a game design document, skeleton completion, repair of 83 injected-fault cases, and Godot-to-Unity porting. Reference materials specify the intended gameplay, while a shared instrumentation interface lets evaluator-owned drivers and probes execute actions and observe independently implemented games. Evaluation combines engine-state checks, certified reference-input replay, and agent-authored feature demonstrations to assess mechanic correctness, demonstrated playability, and behavioral restoration and preservation after repairs. Game-specific vision-language rubrics separately assess presentation. Across six models, Opus5 achieves the highest overall score in all five task types. Best overall scores remain below 60 out of 100 across the three construction tasks, with Brief-to-Game reaching 50.38. Analysis of reviewed submissions identifies requirement omissions and gameplay logic errors as predominant implementation problems. On human-labeled behaviors from 100 agent-built games, executable checks achieve 92.59% balanced accuracy, compared with 78.41% for a video-based VLM judge. Rubric-based visual scores reach a Spearman correlation of 0.829 with human ratings of 200 gameplay clips. Together, these results characterize current agent capabilities across game-development activities and support combining runtime evidence with visual assessment.

1 INTRODUCTION

SWE-Game connects game-development tasks to executable reference games and runtime evidence, addressing the difficulty of assessing whether implementations reproduce intended interactions. Its evaluation spans five development activities and combines functional checks with visual assessment.

1 INTRODUCTION

  • The benchmark addresses the gap between visible gameplay and verified interactions by requiring explicit behavioral targets and execution evidence alongside visual assessment.A game may display hazards correctly while failing to implement their effects, and reaching a victory screen may not show that required interactions occurred.Source: §1 · p. 2
  • SWE-Game derives 247 tasks from 41 reference Godot games across 13 gameplay categories in 2D and 3D, covering five development activities.The activities are brief-to-game generation, GDD-to-game implementation, skeleton completion, bug repair, and Godot-to-Unity porting.Source: §1 · pp. 2–3
  • A shared instrumentation interface lets evaluators bind semantic gameplay roles to independent project structures, then execute actions and inspect relevant engine state.Evaluator-owned drivers and probes can check interactions such as player-hazard contact and the required health change or respawn.Source: §1 · pp. 2–3
  • Evaluation combines interaction tests and input replays with matched no-input controls, repair restoration and preservation checks, and visual rubrics for construction tasks.Porting uses a corresponding Unity execution contract; visual rubrics assess presentation, feedback, and asset integration.Source: §1 · p. 2
  • Opus5 achieves the highest overall score in all five task types; executable checks reach 92.59% balanced accuracy versus 78.41% for a video-based VLM judge.Rubric-based visual scores correlate with human ratings of 200 gameplay clips at 0.829 Spearman correlation; reviewed submissions predominantly show requirement omissions and gameplay logic errors.Source: §1 · p. 2

2 RELATED WORK

Prior game-development benchmarks vary in task coverage and evaluation, while related work also explores interactive repair and diagnostic testing. SWE-Game connects multiple development contracts to runtime checks of required behavior.

Game development benchmarks

  • Existing benchmarks span project editing, game construction, browser-game interaction, and deterministic mechanics checks, with differing task and evaluation coverage.The cited examples include GameDevBench and JamBench, GameEngineBench, GameCraft-Bench, WebGameBench, and GameLogicBench.Source: §2 · pp. 3–4
  • SWE-Game covers five development contracts over a shared project foundation and exposes runtime evidence for behavioral and perceptual outcomes.The authors position this as addressing limits of open-ended or narrowly defined tasks and coarse-grained rule-based tests.Source: §2 · p. 4

Interactive repair and diagnostic testing

  • Interactive repair methods use execution feedback, but additional iterations need not improve outcomes; SWE-Game instead assesses results against explicit requirements.The cited mutation-test study found more iterations with scientific-debugging prompts but similar success rates to ordinary iterative prompting.Source: §2 · p. 4
  • SWE-Game uses evaluator-owned probes and input replays for required interactions, with matched no-input controls and reference, faulty, and repaired runs for repair assessment.These checks target action dependence, behavioral restoration, and preservation.Source: §2 · p. 4

3 SWE-GAME BENCHMARK

SWE-Game evaluates game-development tasks against executable references, with task modes spanning construction, repair, and porting. Its protocol connects behavioral requirements to runtime evidence and separate visual assessment.

3.2 TASK FORMULATION

  • The benchmark derives 247 tasks from 41 games, with one task per game in each mode except Bug Repair, which has 83 cases.Construction ranges from authoring a game from a brief to completing a skeleton; repair and porting supply faulty or Godot projects, respectively.Source: §3 · p. 4, §3.2 · pp. 4–5

3.1 REFERENCE GAME CONSTRUCTION

  • The 41 reference Godot games define objectives, controls, mechanics, success and failure conditions, and asset roles, with levels progressing from introducing mechanics to combining them.The projects, GDDs, assets, and gameplay recordings become reference materials for task construction.Source: §3.1 · p. 4

3.3 GAME INSTRUMENTATION INTERFACE

  • A shared instrumentation interface maps differently named objects and scene hierarchies to semantic roles, standardized actions, and observable values for evaluator-owned drivers and probes.Submissions provide bindings; evaluator instruments working project copies to record positions, contacts, and state changes.Source: §3.3 · pp. 5–6

3.4 EVALUATION PROTOCOL

  • Construction tests combine engine-state checks and certified reference-input replays with agent-authored feature demonstrations, rejecting outcomes also reached by matched no-input controls.Mechanics and playability combine these forms of runtime evidence and action-dependent demonstrations.Source: §3.4 · p. 6
  • Repair evaluation applies identical scenarios to reference, faulty, and repaired versions, while porting checks Unity build readiness and runtime behavior.Repair credit measures restoration and preservation; a repair need not match the reference patch.Source: §3.4 · p. 6
  • Construction visual evaluation scores evaluator-recorded gameplay frames against reference evidence and asset previews with 41 game-specific rubrics comprising 658 items in four groups; objective scores are normalized by an 85-point ceiling.Visual items are capped by triggered deficiency ceilings, averaged within groups, then combined with weights 0.10, 0.18, 0.27, and 0.45, renormalized over applicable groups, to produce V in [0,1]. Porting uses a weighted aggregate of mechanics, playability, structure, visual quality, and stability.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

The experiments evaluate six model–framework configurations across all 247 tasks, examining task performance, resource use, reference-video effects, and agreement with human judgments. Opus5 leads all five tasks, while the evaluation-validity analysis finds executable checks more accurate than a video-based judge for behavior and visual rubrics correlated with human ratings.

4.1 EXPERIMENTAL SETUP

  • The study evaluates six models on all 247 tasks, with each reported configuration pairing a model with its agent framework and tools.Table 3 reports mean component and overall scores on a 0–100 scale.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 MAIN RESULTS

  • 50.38 in Brief-to-Game to 83.46 in Bug Repair: Opus5 achieves the highest overall score in all five tasks.The runner-up varies by task, and every model scores higher on GDD-to-Game than Brief-to-Game, although inputs and scoring criteria differ.Source: §4.2 · p. 7, Table 3 · p. 8
  • In porting, Structure scores exceed Playability and Visual for GPT-5.6 Luna and Opus5; skeleton-completion Content scores range from 11.97 to 36.93.Opus5’s porting scores are 90.53 Structure, 65.88 Playability, and 58.40 Visual; Restoration varies more across models than repair’s Retained routes or Preservation contracts.Source: §4.2 · p. 7

4.3 RESOURCE USE AND ERROR ANALYSIS

  • In Brief-to-Game, Opus5’s highest score uses 8.49M input tokens, 127.34k output tokens, and 111.37 tool calls, versus GPT-5.6 Luna’s 2.56M, 34.05k, and 57.15.The resource analysis compares scores with mean cumulative input and output tokens across runs; model-framework configurations differ in token accounting and tools.Source: §4.3 · pp. 7–8
  • Requirement omissions predominate for GPT and Opus, while gameplay logic errors are GLM’s largest category; these patterns motivate distinct validation checks.Omissions call for tracking task obligations; logic errors call for exercising triggers and inspecting state changes, while engine-integration errors motivate checking connections among scripts, scenes, and resources.Source: §4.3 · pp. 8–9

4.4 REFERENCE-VIDEO ABLATION

  • With reference video, VLM totals rise by 5.60 points for GPT-5.6 Luna and 2.23 for Opus5, while component changes are mixed.Both conditions evaluate the same 41 games, and only the agent’s input changes; Visual Presentation rises by 6.48 and 9.00, respectively.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 EVALUATION VALIDITY

  • Executable checks achieve 92.59% balanced accuracy versus 78.41% for the video-based VLM judge; visual rubric scores correlate with human ratings at Spearman ρ = 0.829.The functional comparison uses 1,200 human-labeled assertions from 100 games; visual scores cover 200 clips.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

SWE-Game combines executable reference games and runtime evidence to assess game construction, completion, repair, and cross-engine porting. Across six models, results reveal a gap between producing playable projects and faithfully implementing intended mechanics and content.

5 CONCLUSION

  • SWE-Game uses executable reference games and a shared instrumentation interface to connect gameplay requirements with runtime evidence, alongside visual assessment of behavioral correctness and perceptual quality.Across six models, the benchmark reveals a persistent gap between producing playable projects and faithfully implementing intended mechanics and content.Source: §5 · p. 10

LIMITATIONS AND FUTURE WORK

The benchmark’s reference-game collection and evaluation scope remain limited, and future work will expand them. The collection spans projects of varied structures and scales, while the paper also reports data-provenance and AI-use practices.

LIMITATIONS AND FUTURE WORK

  • The 41-game benchmark has limited coverage of large projects and long-term development; future work will expand the collection and evaluate longer workflows.The authors also propose controlled-budget framework comparisons and broader human assessment to refine visual rubrics and clarify their relation to player experience.Source: LIMITATIONS AND FUTURE WORK · p. 10
  • The authors report using generative AI for literature organization and scientific writing, reviewing AI-assisted code, and taking responsibility for the final work.They state that AI was not used to fabricate experimental results or replace human judgments of game validity.Source: AI USE STATEMENT · p. 10
  • The projects and assets came from public sources with provenance and licenses recorded; generated projects run in isolated environments without external services or private data.The authors caution users to respect copyright, licensing, and attribution when creating or releasing game content.Source: ETHICS · p. 10

A REFERENCE GAME COMPOSITION

  • The reference collection contains 24 2D and 17 3D games, ranging from compact single-mechanic projects to multi-stage games with interacting mechanics and persistent progression.Together, the projects contain 168,335 lines of gameplay GDScript and 1,587 scenes; code size ranges from 1,153 to 11,024 lines, with a median of 3,367.Source: A REFERENCE GAME COMPOSITION · p. 13, Table 6 · p. 13

B TASK CONSTRUCTION AND DEVELOPMENT MODES

SWE-Game’s task construction spans five development modes that vary the specification and starting implementation, covering new builds, scaffold completion, repair, and engine porting. Unity porting tasks focus on preserving intended gameplay under engine-specific project constraints, with instrumentation contracts defining how entities, actions, state, and outcomes are observed.

B.1 BRIEF-TO-GAME

  • Mode 1 gives developers a concise design target, asset palette, and reference video while leaving concrete rules and implementation choices open.The brief is a design target, not an implementation specification.Source: §B.1 · p. 14

B.2 GDD-TO-GAME

  • Mode 2 supplies a reviewed task GDD and observable behavior claims in DEMONSTRATIONS.md, while executable predicates and reference solutions remain evaluator-only.Its Beacon Relay example specifies fuse pressure, the return-trip challenge, stomp time refunds, and fixed readable motion.Source: §B.2 · pp. 14–15

B.3 SKELETON COMPLETION

  • Mode 3 starts from a newly generated, implementation-neutral scaffold that fixes public integration surfaces rather than exposing the reference game’s internal architecture.It includes addressable placeholder scenes, InputMap bindings, a levels manifest, and a player adapter with initially placeholder numeric slots.Source: §B.3 · p. 17

B.4 BUG REPAIR

  • Mode 4 packages a faulty Godot project with a player-facing symptom report and gameplay requirements, using validated injected faults and stable case IDs.The repaired project is expected to preserve unaffected behavior.Source: §B.4 · p. 18

B.5 GODOT-TO-UNITY PORTING

  • Mode 5 asks developers to port sanitized Godot source into a prebuilt Unity workspace, preserving behavior and presentation described by the GDD and video.The benchmark uses Godot 4.5.1 and targets Unity Editor 6000.3.23f1.Source: §B.5 · pp. 18–19
  • Unity ports must preserve intended controls, mechanics, progression, content, and visible feedback while meeting project, build, and input-witness requirements.Gameplay code belongs under Assets/Game/; supplied SDK and version files are protected, BUILD.md specifies a reproducible Linux Player build, and ops.json provides an input witness. Assets must be imported into Unity, not supplied by bundling or launching the Godot runtime.Source: §B.5 · p. 19

C GAME INSTRUMENTATION INTERFACE

  • The instrumentation interface uses shared Godot contracts and a separate Unity contract to specify semantic entities, actions, state bindings, scenes, and outcomes.The Godot schema requires a player role and lists optional gameplay roles, action channels, numeric slots, and level/endings declarations; the interface is broader than the Godot level manifest.Source: C GAME INSTRUMENTATION INTERFACE · p. 20

D MODE-SPECIFIC EVALUATION CRITERIA

Table 8 organizes evaluation criteria by task mode, while Table 9 lists component weights for construction and porting. Construction, repair, and porting use different criterion combinations.

D MODE-SPECIFIC EVALUATION CRITERIA

  • Construction modes share mechanics, content, and playability measures, with mode-specific design or scaffold criteria and visual assessment.Table 8 summarizes the criteria associated with each task-mode component.Source: D MODE-SPECIFIC EVALUATION CRITERIA · p. 20, Table 8 · p. 21
  • Table 9 lists component weights for construction and porting, and indicates that Bug Repair uses multiplicative aggregation.A dash marks components that do not apply.Source: Table 9 · p. 21
  • Repair and porting use scorecards tailored to behavior preservation and cross-engine fidelity, respectively.Table 8 presents the evaluation criteria by task mode.Source: D MODE-SPECIFIC EVALUATION CRITERIA · p. 21, Table 8 · p. 21

D.1 SCORE AGGREGATION

The aggregation rules combine weighted component scores for construction and porting, while Bug Repair multiplies case-level factors before averaging cases.

D.1 SCORE AGGREGATION

  • For construction tasks, objective components account for 85% and the VLM score accounts for 15%.Table 9 lists component weights for construction and porting as percentages of the final score.Source: §D.1 · p. 21
  • Construction and porting task scores use component scores weighted by their mode-specific percentages, summing across applicable components.Component scores are defined on a 0–100 scale, and the applicable component set is denoted Cm.Source: §D.1 · p. 22
  • Bug Repair case scores multiply restoration, retained-route, preservation-contract, and validity-gate factors; overall scores average 83 case scores.The reported overall score is the arithmetic mean across cases, with repair factors multiplied within each case before averaging.Source: §D.1 · p. 22

E EXPERIMENTAL CONFIGURATION AND RESOURCE ACCOUNTING

The experimental setup specifies sandboxing and engine versions, then reports resource-accounting conventions for the recorded model runs.

E EXPERIMENTAL CONFIGURATION AND RESOURCE ACCOUNTING

E.1 RESOURCE ACCOUNTING

  • Figure 4(a) and (b) report arithmetic means over each model's recorded runs, counting cache reads and writes once and reused context each time it is processed.Output totals retain reported usage without adding reasoning subtotals again.Source: §E.1 · p. 22

F FAILURE ANALYSIS

The analysis categorizes implementation problems in reviewed submissions from three models. It uses seven categories and computes percentages separately for each model.

F FAILURE ANALYSIS

  • Reviewed submissions from GPT-5.6 Luna, Opus5, and GLM5.3 Flash are analyzed across seven implementation-problem categories.The categories include requirement omissions, gameplay logic errors, engine-integration errors, delivery errors, visual or asset mismatches, state-lifecycle errors, and other issues.Source: F FAILURE ANALYSIS · p. 22
  • Each identified problem receives one primary category, although a single game may contribute multiple problems.Source: F FAILURE ANALYSIS · p. 22
  • Category percentages are calculated separately for each model, using that model’s total categorized problems as the denominator.Source: F FAILURE ANALYSIS · p. 22

G CASE STUDY

This section examines how visual resemblance and gameplay correctness may diverge and defines a constrained visual-judge procedure for assessing rubric attainment from candidate frames. It supports reporting functional and visual performance separately alongside overall scores.

G CASE STUDY

  • The 2D and 3D case studies show that visually plausible games can fail on progression or interaction, while functional routes may have lower visual fidelity.The examples contrast reproducing layouts, platforming routes, interactive objects, and visual progression with coordinating traversal, beacon activation, movement continuity, and completion feedback.Source: G CASE STUDY · p. 22, Figure 5 · p. 23, Figure 6 · p. 23

H.1 BRIEF-TO-GAME

  • Brief-to-Game treats the brief as a design target, not a supplied GDD, so agents must author their own design before implementation.The workflow calls for defining the experience, design pillars, core loop, scope, and mechanic behavior before coding.Source: §H.1 · p. 24

H.2 GDD-TO-GAME

  • GDD-to-Game uses the supplied GDD as its specification, with independently replayed demonstrations covering observable features; unexercised requirements remain uncovered.The renderer notes that the mode-specific instruction retains legacy full-clear wording, while the shared demonstration protocol does not require whole-game completion.Source: §H.2 · p. 25

H.3 SKELETON COMPLETION

  • Skeleton Completion provides a small importable integration scaffold whose public entry, adapter, manifest paths, groups, and action IDs must be preserved, while gameplay code can be authored freely.Agents must implement the stated observable behaviors, drive numeric properties from live state, and produce operations that genuinely clear the game.Source: §H.3 · pp. 25–26

H.4 BUG REPAIR

  • Bug Repair starts from a complete project with a coherent defect bundle and requires reproducing the failure, tracing its cause, and making a behavior-preserving repair.Agents must retest the affected and a nearby regression path without deleting the broken feature or unbinding an action.Source: §H.4 · p. 26

H.5 GODOT-TO-UNITY PORTING

  • Godot-to-Unity requires a native Unity port preserving observable mechanics, progression, outcomes, retry semantics, content identity, audiovisual feedback, and public GB addresses.Observable state must reflect real gameplay consequences; the prompt permits redesigning Unity scenes and C# structure rather than preserving Godot node paths.Source: §H.5 · pp. 26–27

H.6 VISUAL JUDGE

  • The visual judge assesses game-specific items across candidate segments, treating missing coverage as unknown and observed absence of required achievements as a shortfall.Footage cannot certify hidden code, exact input latency, unseen collisions, audio, or deterministic implementation.Source: §H.6 · p. 27
  • Attainment q ranges from 0 to 1 and reflects realized relationships plus the importance, scope, and severity of remaining deficiencies.Any score below 1 requires an observed deficiency tied to a supplied normative clause; judges must not add obligations beyond the rubric.Source: §H.6 · pp. 27–28
  • The judge returns attainment rather than a transformed score, while the host calculates scoring curves and applies deficiency caps.Source: §H.6 · p. 28
  • For every applicable item, the judge must assign numeric attainment from the candidate PNG, using visible state and a conservative score when the full route is unseen.Use 0.0 when no criterion is visibly met; do not return unknown solely because the full route is not visible.Source: §H.6 · p. 28
  • The response must be strict JSON with one item per requested rubric identifier and only supplied clause, requirement, cap, and response IDs.Each item includes evidence, deficiencies and checks, applied caps, limitations, and a score rationale.Source: §H.6 · p. 28
Loading 2609.33678v3…