PaperlayerView exact revision on arXiv

Source-linked AI summary

A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

Seonho Lee, Wonryeol Jeong, Alberto Cereser, Inha Kang, Hyeonjong Kim, Seungmin Kwak, Dongmin Park

arXiv:2609.39564v1cs.AI

TL;DR

Coding agents must preserve intended designs, not just produce plausible games, where requirements span game logic, visual rendering, and player interactions. A2Z GameSpec-Bench evaluates games against fixed, dependency-aware contracts using code inspection and replay and playtesting policies. Requirement-specific feedback improves GDD Fidelity over self-revision after two rounds.

  • Problem

    Faithfully implementing a game is challenging because its intended design spans requirements for game logic, visual rendering, and player interactions.

    Source: §1 · pp. 1–2

  • Method

    A2Z GameSpec-Bench evaluates games from long-form GDDs using fixed dependency-aware contracts and requirement-linked code inspection, replay, and adaptive playtesting.

    Source: §1 · p. 2, §5 · p. 14

  • Results

    10.9%: requirement-specific feedback improves GDD Fidelity relative to self-revision after two rounds.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A2Z GameSpec-Bench supports systematic comparison of specification-following and targeted revision toward the intended design.

    Source: §1 · p. 3, §5 · p. 14

    The benchmark primarily covers synthetic 2D single-player Phaser games, and generalization to human-authored specifications, other engines, and broader software tasks remains unestablished.

    Source: J. Limitations · p. 68

Abstract

from arXiv · show

Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce A2Z GameSpec-Bench, a benchmark of 100 long-form GDDs for evaluating end-to-end game development by agents. We measure faithfulness by checking whether the game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a dependency-aware contract that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated test policies for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure detection. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond implementation judgments and provides targeted feedback to support more faithful game development. Code and datasets are available at https://a2z-gamespec-bench.github.io.

1. Introduction

The introduction frames fidelity as implementing connected design requirements across game logic, visuals, and play—not merely producing plausible or runnable games. It presents A2Z GameSpec-Bench as a fixed-contract, multi-channel evaluation and reports gains from requirement-specific revision feedback.

1. Introduction

  • Long-form GDD fidelity requires preserving rules and their relationships across implementation, rendered behavior, and actual play.Source: §1 · p. 2
  • A2Z GameSpec-Bench evaluates end-to-end game development from 100 long-form GDDs by asking agents to deliver source projects and playable builds.Source: §1 · p. 2
  • Each GDD becomes a fixed Dependency-Aware Contract recording rules, constraints, and prerequisite relations rather than treating requirements as an independent checklist.Source: §1 · p. 2
  • Evaluation combines source-code inspection with requirement-specific Test Policies for scenario-based replay, visual evidence, and adaptive playtesting.Source: §1 · p. 2
  • Judgments and evidence link to the same requirements to support GDD Fidelity scoring and revision feedback.Source: §1 · p. 3
  • 10.9%: requirement-specific feedback improves overall GDD Fidelity relative to self-revision after two rounds from the same initial builds.Source: §1 · p. 3

2. Related Work

Related work spans specification-based evaluation, structured contracts, and benchmarks for agentic game development. A2Z distinguishes itself by fixing a dependency-aware contract from each long-form GDD and linking complementary evidence to the same requirements.

Specification-Based Evaluation

  • Specification-based benchmarks decompose prompts into criteria and use visual checks, code execution, or interaction evidence, but game requirements may depend on unmet prerequisites.Source: §2 · p. 3

Structured Evaluation Contracts

  • Structured-evaluation work represents prerequisites, constraint composition, or interaction flows, while A2Z applies related contract principles to GDDs and distinguishes local tests from specified gameplay paths.Source: §2 · pp. 3–4

Agentic Game Development and Evaluation

  • Game-generation benchmarks use varied methods, including replay, GUI playtesting, initialized runtime states, and code inspection with live validation.Source: §2 · p. 4
  • A2Z fixes a Dependency-Aware Contract from each long-form GDD before inspecting generated builds, then links judgments and evidence to requirements across agents and revision rounds.Source: §2 · p. 4

3. A2Z GameSpec-Bench

A2Z GameSpec-Bench evaluates games built from long-form GDDs against a fixed contract of requirements and dependencies. It combines source inspection, scenario replay, and adaptive playtesting, linking evidence and judgments to requirements for scoring and revision feedback.

3.1. Benchmark Setting

  • The benchmark evaluates a coding agent’s source project and playable build against a GDD-derived contract withheld during initial generation.The corpus contains 100 GDDs split into 50 Small and 50 Big designs.Source: §3 · p. 4, §3.1 · p. 5

3.2. Dependency-Aware Contract

  • The Dependency-Aware Contract records rules, constraints, and prerequisite relations, making interdependent requirements explicit rather than treating them as an independent checklist.Its dependency edges connect rules when one updates state referenced by another rule’s preconditions or emits an event that triggers it.Source: §3.2 · p. 5

3.3. Contract-Guided Evaluation

  • Contract-guided evaluation combines source-code inspection, fixed-input scenario replay, and adaptive playtesting to gather evidence about requirements’ visual and interactive outcomes.Replay policies specify input sequences and timing; adaptive policies choose inputs in response to runtime observations.Source: §3 · p. 4, §3.3 · p. 5

3.3.1. Source-Code Evaluation

  • Source-code evaluation checks whether each rule’s specified conditions, triggers, and effects are implemented, then weights rules by downstream reach and incorporates invariant pass rates.The default weight floor is α=0.5, limiting a rule’s weight to at most twice that of a rule with no downstream dependencies.Source: §3.3.1 · pp. 5–6

3.3.2. Scenario-Based Replay Assessment

  • Scenario-based replay uses fixed input sequences to assess replay-observable requirements against a visual rubric, with judgments supported by frame evidence.Adaptive frame selection retrieves relevant recorded moments to address brief responses and delayed outcomes.Source: §3.3.2 · p. 6

3.3.3. Adaptive Playtest

  • Adaptive playtests use executable policies that respond to runtime observations, record traces, and assess whether required situations and outcomes occur through player inputs.Targeted adversarial tests initialize preconditions for rules left unverified during normal play, then require player inputs to trigger and assess effects.Source: §3.3.3 · pp. 6–7
  • Each tested rule is judged satisfied, violated, or unverified from its execution trace, with a strict majority determining the verdict and adversarial tests extending normal-play coverage.Unverified means the required situation or a conclusive outcome was not established; it does not count as a violation.Source: §3.3.3 · p. 7

3.4. Requirement-Level Evidence and Feedback

  • Overall GDD Fidelity equally weights the three evaluation axes, while runtime fidelity averages replay and adaptive-playtest scores; requirement-linked evidence supports feedback across revisions against the same contract.The contract remains fixed while revised builds are assessed on all three axes.Source: §3.4 · pp. 7–8

4. Experiments

The experiments assess agents across long-form GDDs and multiple evaluation axes, then examine diagnosis, revision, reliability, component choices, and extension to 3D games. Results show that broader designs are harder to implement faithfully and that requirement-level feedback improves fidelity over self-revision.

4.1. Experimental Setup

  • Across nine agents and 100 GDDs, the same contracts and replay scenarios are used for every agent, with three builds per GDD and three independent judgments per build.The benchmark reports overall fidelity, source-code, replay, adaptive-play scores, and verifiable rate.Source: §4.1 · p. 8

4.2. Main Benchmark Results

  • Mean GDD Fidelity falls from 71.1 on Small GDDs to 51.1 on Big GDDs, with source-code, replay, and adaptive-playtest scores also declining.The three axis-specific decreases are 19.1, 20.6, and 20.3 points, respectively.Source: §4.2 · pp. 8–9

4.3. Complementarity of the Evaluation Axes

  • Across 100 GPT-5.6-Sol games, omitting source-code, replay, or adaptive-play evaluation reverses 20.5%, 8.9%, or 11.2% of full-evaluation orderings, respectively.Source-code and runtime judgments also disagree: only 76.8% of rules with full source-code scores are confirmed satisfied in playtesting.Source: §4.3 · p. 9

4.4. Diagnosing Integration Failures

  • Dependency context raises coverage of recorded playtest violations from 71.1% to 80.2%, adding 51 violations across 26 games to the inspection set.The added targets include cases that received full enumerated source credit; the improvement is 9.1 points, with a 95% paired-game bootstrap interval of 6.1–13.0 points.Source: §4.4 · p. 10, Table 3 · p. 10

4.5. Iterative Revision with Three-Axis Feedback

  • After two rounds on 50 Big GDDs, Source + Replay + Playtest reaches 74.3 mean GDD Fidelity versus 67.0 for self-revision, a 7.3-point gain (10.9% relative).The conditions share initial builds and a revision agent; qualitative cases show feedback repairing visual, interaction, and state-update errors that self-revision leaves unresolved.Source: §4.5 · pp. 10–11

4.6. Evaluation Reliability

  • On Big GDDs, contract construction achieves 93% citation overlap across five runs versus 57% for a naive judge; the top three agents retain their positions under every nonnegative axis weighting.For the All aggregate, 30 of 36 agent-pair orderings are invariant across the full weighting domain.Source: §4.6 · p. 11

4.7. Component Analysis

  • Under matched second-pass budgets, adversarial playtesting raises judgment coverage to 48.5%, versus 41.0% for another normal pass; adaptive frame selection is preferred in 60.5% of non-tied comparisons.Dependency weighting at α=0.5 changes source-code scores by 0.38 points on average and at most 3.71 points, while stronger weighting at α=0.1 leaves model rankings unchanged.Source: §4.7 · pp. 11, 13, Table 4 · p. 11

4.8. Extension to 3D Games

  • The contract representation and three evaluation axes extend to three Three.js builds, with engine-specific interfaces adapted for execution, input, observation, scenario initialization, and frame capture.The evaluated games are Sonic: Cascade Coast, Diablo Cathedral, and Rocket League.Source: §4.8 · p. 13, Figure 8 · p. 14

5. Conclusion

A2Z GameSpec-Bench evaluates faithfulness to long-form GDDs using fixed dependency-aware contracts and linked evidence, and supports targeted revision. Its documents are generated and validated through a pipeline designed to make their requirements checkable.

5. Conclusion

  • The benchmark links fixed Dependency-Aware Contracts to source-code inspection and agent-generated Test Policies for replay and adaptive playtesting, with evidence tied to the same requirements.It connects requirement-level evaluation with targeted revision across 100 long-form GDDs.Source: §5 · p. 14

A.1. Brief-to-GDD Pipeline

  • The 100 GDDs are divided into Small and Big splits and derive from briefs expanded first into creative visions, then into detailed specifications.The brief fixes premise, mechanic, and scope; the creative vision establishes intended experience before the GDD specifies entities, rules, formulas, stages, screens, and assets.Source: A. Game Design Document Dataset · p. 19, §A.1 · p. 19

A.2. Harness Optimization

  • Harness search balances CV fidelity against buildability, then selects candidates through repeated item-level and CV-level majority comparisons.All 100 benchmark GDDs use the harness selected at iteration 8, but selection does not guarantee that every GDD is defect-free.Source: §A.2 · p. 20

A.3. Admission Criteria

  • Admission criteria target missing, contradictory, unreachable, or ambiguous requirements that can make conformance scores difficult to attribute to implementation.Canonical identifiers, ordered rule tables with an ELSE row, and ledgers make repeated values, totality, determinism, and derived calculations more checkable.Source: §A.3 · pp. 20–21

A.4. Validation and Repair

  • Candidates are validated against G1–G4 and accepted only after three consecutive rounds with no detected violations.Repairs are made at canonical definitions, logged, and followed by full re-validation when a document changes; remaining evaluator and interpretation errors can persist.Source: §A.4 · p. 22

A.5. Dataset Splits and Statistics

The dataset divides 100 GDDs by game scope, not document length, and illustrates how shared document conventions express requirements at different scales.

A.5. Dataset Splits and Statistics

  • Small GDDs specify compact core tasks, while Big GDDs cover broader content and interacting systems; split assignment is based on game scope rather than document length.The benchmark’s Big designs are complete 2D single-player games, excluding 3D or 2.5D presentation and mandatory multiplayer.Source: §A.5 · p. 22
  • Big GDDs contain about 1.87× as many tokens and 1.55× as many outcome requirements as Small GDDs, across 11 commercial genres versus eight Small task families.Each split contains 50 documents.Source: §A.5 · p. 22, Table S3 · p. 23, Table S4 · p. 23

A.6. Examples of GDDs

  • Both splits use prose plus labeled rule-table rows and canonical constants, while Big-split rules can connect several tables across a single player action.The examples trace a Small GDD requirement to contract rules and show a Big GDD slide action spanning stop, camouflage, and turn rules.Source: §A.6 · p. 24

B.1. Dependency Structure in Game Specifications

The paper measures causal structure in game specifications and builds fixed contracts that represent rules, dependencies, and execution evidence. Its analyses examine contract stability, dependency interpretation, and the difference between local and dependency-adjusted rule passes.

B.1. Dependency Structure in Game Specifications

  • 38.4% is the corpus-macro average rate of causal sentences across three external game-design corpora, with corpus rates ranging from 32.0% to 45.3%.The pooled sentence-level rate is 42.0%; the analysis uses CiRA predictions on retained prose.Source: §B.1 · p. 27, Table S7 · p. 27

B.2. Rule Generation

  • Contract generation fixes vocabulary and source entries before formalizing rule conditions and effects, then validates repeated generations before freezing one contract.Read/write facts and updates are independently extracted three times; code assembles a dependency skeleton, and the acceptance check requires reachability agreement of at least 0.9.Source: §B.2 · pp. 27–29

B.3. Properties of the Accepted Contract

  • The accepted contract separates triggers, preconditions, and effects, derives dependencies from rule fields, and preserves source associations and unformalizable values.State links connect writes to inspected attributes; event links connect emitted events to matching triggers.Source: §B.3 · p. 29

B.4. Contract Example

  • The traces_left example maps GDD rows to rule triggers, preconditions, and effects, with repeated rule nodes showing occurrences in a recorded playtest.The execution view is not the full static contract graph; a missing prerequisite can leave downstream behavior unverified without automatically making every downstream rule violated.Source: §B.4 · p. 29

B.5. Stability of Contract Generation

  • Across repeated generations, dependency-graph reachability agreement, edge Jaccard similarity, and role preservation are each at least 0.999, while trigger-name agreement is 0.94 and predicate-structure agreement is 0.70.The repeated generations preserve the same rule entries and source associations; predicate variation increases score changes of at least 0.25 by 5.0 points on Small and 2.3 points on Big.Source: §B.5 · pp. 29–30, Table S8 · p. 31, Table S10 · p. 32

B.6. Value Compatibility

  • After removing state-dependency edges with provably incompatible written values, dependency-weighted rule-score rankings remain stable at Spearman 0.99 on Small and 1.00 on Big.The authors retain attribute-level edges as potential rather than guaranteed prerequisites.Source: §B.6 · p. 32

B.7. Dependency-Adjusted Rule Pass Rate

  • 72.4% of dependency-linked rules pass locally in source-code judgments, versus 22.7% when all direct and indirect predecessors must also pass.This is a graph-based reachability proxy; edges encode potential prerequisites, so exclusion indicates possible blocking rather than demonstrated runtime unreachability.Source: §B.7 · p. 32

C. Experimental Details

The benchmark uses a common Phaser development environment and combines source-code judgments with runtime replay and adaptive playtesting. Source-code evaluation uses a prompted model judge with fixed-schema evidence requirements.

C. Experimental Details

  • The 100 main tasks use Phaser 4.1, TypeScript, and Vite for 2D single-player browser games, with a shared project and runtime interface specification.Each of three open-weight models generates three games per GDD without best-of-k selection; serving uses approximately 1,100 GPU-hours.Source: C. Experimental Details · p. 33

D.1.1. Prompt

  • Runtime evaluation serves builds in Chromium, replays recorded inputs, and uses observation-driven keyboard and pointer inputs for adaptive playtesting.Source-code evaluation requires one model call per build and a validated fixed-schema judgment with an evidence entry for every rule.Source: C. Experimental Details · p. 33, §D.1.1 · p. 34

D.1.2. Implementation

The evaluation combines source-code judgments, scenario-based visual replays, and adaptive playtests, linking evidence to fixed contract requirements. Each axis uses distinct procedures to test implementation or behavior against those requirements.

D.1.2. Implementation

  • Source-code judges assess each rule’s causal commitment and each invariant, then combine dependency-weighted rule scores with invariant scores.Rules receive continuous scores based on whether code gates triggers on stated conditions and produces specified effects; invariants are binary checks of whether a property holds across reachable gameplay paths within scope.Source: §D.1.2 · pp. 34–35

D.2.1. Prompt

  • The replay judge selects recorded frames for rubric items, then scores those frames with cited visual evidence.The selection call uses the rubric, input trace, render timestamps, and an observation budget; a deterministic tool resolves requests to frames, and the scoring call returns a score, rationale, and frame citations for each item.Source: §D.2.1 · p. 36

D.2.2. Implementation

  • Each build supplies input policies for every canonical scenario, while the contract defines what is tested and what visual evidence is collected.The replay protocol uses contract-linked rubric items and deterministic frame retrieval; replays are independently judged three times, then aggregated by item rule and category weights.Source: §D.2.2 · p. 36

D.3.1. Prompt

  • Adaptive playtesting uses separate normal-play and adversarial bot authors plus a trace judge, with objectives specifying target rules and dependencies.Adversarial tests may establish a rule’s precondition through a declared scenario or minimal state injection, then use player input to test the trigger and effects; absent or inconclusive evidence is unverified, not violated.Source: §D.3.1 · pp. 36–37

D.3.2. Implementation

  • Bots use a shared runtime interface to observe game state, issue inputs, and record labeled snapshots that the evaluator checks against rule conditions, triggers, and effects.A verdict requires evidence that the required situation was reached; each rule receives three judgments, with strict majority deciding its verdict and no majority treated as unverified.Source: §D.3.2 · pp. 37, 39

E. Additional Analysis on Benchmark Results

The additional analyses compare agents using pairwise outcomes and Elo ratings, then examine how stable evaluation targets remain across repeated judgments. They find that a stable score can coexist with substantial variation in the requirements a naive judge extracts.

E. Additional Analysis on Benchmark Results

  • The pairwise protocol compares six proprietary agents on matched GDDs, giving a tie half credit and calculating Overall from each game’s three-axis mean.Elo ratings use Bradley–Terry fits; pooled ratings combine Small and Big GDDs, and uncertainty is estimated with 2,000 GDD-level bootstrap resamples.Source: E. Additional Analysis on Benchmark… · p. 40
  • Claude-Fable-5.1 has the highest pooled Elo on all four axes; GPT-5.5’s win rate against GPT-5.6-Sol is 76.5% in source-code evaluation, 61.0% in Playtest, and 12.9% in replay assessment.The pairwise matrices report win rates, with ties worth half a win; Elo summarizes outcomes against all opponents.Source: E. Additional Analysis on Benchmark… · p. 40, Figure S11 · p. 41

F. Ablations on Evaluation

  • The ablation analyses vary one evaluation choice at a time, including whether a judge reconstructs requirements and whether source-code scores weight rules by dependency reach.They compare changed judgments against the same judge’s self-disagreement while holding other inputs, such as builds or scenario replays, fixed.Source: F. Ablations on Evaluation · p. 44, §F.1 · p. 44

F.1.1. Requirement-Set Consistency

  • For the median GDD, naive-judge requirement counts vary by 60% of their mean, only 35% of items match across runs, and 9% appear in all five runs.Cross-run matching uses a Jaccard threshold of 0.5; median match rates are 31% for Big GDDs and 38% for Small GDDs, with 7% and 14% of items, respectively, appearing in all five runs.Source: §F.1.1 · p. 44, Figure S14 · p. 45
  • For the median unchanged build, the gap between a naive judge’s highest and lowest scores is 0.05, despite substantial variation in its extracted requirements.The naive judge reconstructs requirements on each run, whereas the contract-based judge scores fixed rules and invariants.Source: §F.1.1 · p. 44

F.1.2. Ablation On Dependency-Weighted Scoring

These ablations examine dependency weighting, replay-judge settings, adaptive playtesting, and complementary evidence channels. Their findings show that evaluation choices affect agent comparisons and the requirements for which conclusive execution evidence is collected.

F.1.2. Ablation On Dependency-Weighted Scoring

  • Negative score gaps occur in 11 of 18 root-defective games versus 6 of 16 comparison games, but the gaps are small relative to judge variability.The sign reflects score distribution across the contract; positive gaps in root-defective games arise primarily when failures are more prevalent among leaf rules.Source: §F.1.2 · pp. 45–46
  • Dependency weighting reverses at least one agent-pair source-code ordering on 23 of 100 GDDs, with rule judgments and invariant scores held fixed.The comparison contrasts uniform weighting (α=1) with downstream-reach weighting (α=0); the figure marks GDDs with a strict reversal.Source: §F.1.2 · p. 46, Figure S15 · p. 46

F.2. Scenario-Based Replay Assessment

  • At high effort, Luna’s replay judgment takes 41 seconds and costs $0.007 per call, versus 78 seconds and $0.048 for the Sol extra-high reference.At extra-high effort, substituting Luna changes mean replay score by −0.005 with agreement of 0.85; lower effort raises mean scores, especially on visual and art items.Source: §F.2 · pp. 46–47, Table S11 · p. 47

F.3. Adaptive Playtest

  • Mean judgment coverage rises from 21.7% after normal play to 74.4% after adversarial testing, adding 41.1 percentage points of satisfaction and 11.6 points of violation evidence.Coverage increases for every agent and rises more on Big GDDs, from 8.2% to 68.3%, than on Small GDDs, from 35.3% to 80.5%.Source: Figure S17 · p. 47, §F.3 · p. 48, Table S12 · p. 49
  • On 86 dependency-linked rules, a second normal test raises coverage from 40.5% to 41.0%, while an adversarial second test raises it to 48.5%.The build, contract, test policy, interface, and budgets are held fixed; combined testing exceeds the 45.0% coverage of adversarial testing alone.Source: §F.3 · p. 48, Table S13 · p. 49

F.4. Complementarity of Evaluation

  • Three examples show source-code scores of 1.00 paired with playtest violations: a mistimed condition, a duplicated effect, and an outdated value.The examples come from Rainwright, Strata Keepers, and Chroma Bastion, respectively; the authors say linking both evidence types to the same requirements makes discrepancies identifiable for revision.Source: §F.4 · p. 50
  • The axis-omission analysis recomputes scores as the mean of the two retained axes and counts strict game-pair ordering reversals within each split.Ties in either ordering remain in the denominator but are not counted as reversals.Source: §F.4 · p. 50

F.5. Sensitivity to Evaluation-Axis Weights

Evaluation-axis weights can change agent rankings, while requirement sets extracted through dependency-aware contracts show greater repeat-run stability than naive judgments. The appendix also reports how weighting affects rank robustness and contract coverage.

F.5. Sensitivity to Evaluation-Axis Weights

  • 30 of 36 All-aggregate pairwise orderings remain invariant across the full weight domain, and the top three agents retain their order for Small, Big, and All.When each axis receives at least half its default contribution, 34 of 36 All-aggregate orderings are invariant and every agent stays within one rank of its equal-weight position.Source: §F.5 · p. 50

Click ENTER ZONE

  • A system scoring 68.4 versus 61.0 in source-code evaluation but 49.3 versus 54.6 in runtime fidelity loses the aggregate ordering when source inspection is emphasized.GPT-5.6-Sol scores higher than GPT-5.5 in replay, 56.1 versus 51.6, while emphasizing scenario-based replay changes their order; reporting all three axes alongside the aggregate makes these differences explicit.Source: Click ENTER ZONE · p. 52

G.1.1. Rule Coverage and Run-to-Run Consistency

  • The contract raises mean rule coverage against the pooled reference set from 0.84 to 0.88 on Small and from 0.62 to 0.80 on Big.The reference pools rules cited across five runs of both methods; coverage measures references to that recovered set, not all possible GDD rules.Source: §G.1.1 · p. 53, Table S15 · p. 53
  • On Big GDDs, requirement citation overlap is 93% for repeated contract extractions versus 57% for naive judgments.Among naive items matchable across runs, judgment status agrees in 91% of Small pairs and 87% of Big pairs, so instability largely concerns which requirements are selected.Source: §G.1.1 · p. 53

G.1.2. Comparison With Enumerated Evaluation

Dependency context finds runtime violations missed by enumerated source inspection, while the appendix also examines judge agreement, replay efficiency, adaptive playtesting, and frame selection. Together, the comparisons show how requirement-linked evidence can reveal failures and support more targeted evaluation.

G.1.2. Comparison With Enumerated Evaluation

  • Dependency context raises inspection coverage of recorded violated targets from 398/560 (71.1%) to 449/560 (80.2%), a gain of 9.1 percentage points.It adds 51 violated targets across 26 games; the 95% paired game bootstrap interval for the gain is 6.1–13.0 points.Source: §G.1.2 · p. 54, Table S16 · p. 54
  • Among 2,446 paired rules, 619 disagree on full-credit status between enumerated and contract-based source evaluation, and retrospective review finds omissions in 17 rules across 13 Big games.Examples include a missing failure-and-penalty branch, an omitted retry-checkpoint update, a missed landscape badge, and an unchecked occupied tile.Source: §G.1.2 · pp. 54–55

G.2.1. Source-Code Evaluation

  • GPT-5.6-Luna at high effort reaches 0.72 rule-level correlation with reference source judgments at an estimated 0.15× the independent GPT-5.6-Sol re-run cost.The independent re-run correlation is 0.74; Luna invariant agreement ranges from 0.73 to 0.77, versus 0.83 for the re-run.Source: §G.2.1 · p. 55, Table S17 · p. 55

G.2.2. Scenario-Based Replay Assessment

  • At 640 pixels, replay assessment correlates 0.967 with native-resolution per-item scores; reducing to 480 pixels lowers agreement with 640-pixel assessment to 0.921.A native-width frame uses about 4.7K image tokens, compared with 2.0K at 1280 pixels and 0.5K at 640 pixels; 480 pixels also makes text harder to read.Source: §G.2.2 · pp. 55–56, Figure S20 · p. 56, Figure S21 · p. 56

G.2.3. Adaptive Playtest

  • On fixed traces, GPT-5.6-Luna agrees with the GPT-5.6-Sol high reference at 0.910 on Small and 0.888 on Big, while token use per vote falls to 51K–71K.The independent Sol re-run agrees at 0.959 on both splits; reported per-game playtest cost with three judgments per rule falls to $0.08–0.11 from $0.65–0.73.Source: §G.2.3 · p. 56, Table S18 · p. 57

G.3. Fixed-Rate Sampling vs. Adaptive Frame Selection

  • Adaptive frame selection receives 60.5% of item preferences and 35 game wins, versus 39.5% and 13 for fixed-rate sampling, with two ties.Selected frames can capture intermediate transitions, such as a Confirm button becoming enabled or a drawn path being removed, that uniform samples omit.Source: §G.3 · p. 57, Table S19 · p. 57, Figure S22 · p. 58

G.4. Gameplay Agent vs. Adaptive Playtest

  • Across ten Big games, adding adversarial results raises recorded requirement judgment coverage from 10.0% with normal play to 73.4%.Unlike milestone attainment, the evaluation links a named violated requirement to execution evidence; Toybox Rider's milestone checks miss its sequence, duration, and tutorial-obstacle conditions.Source: §G.4 · p. 59, Table S20 · p. 59

H. Additional 3D Evaluation Results

The additional 3D cases show that contract-based evaluation can distinguish implementation credit from rendered and gameplay fidelity, and identify concrete revision targets.

H. Additional 3D Evaluation Results

  • Sonic receives a source-code score of 99.71 but Replay and Playtest scores of 53.50 and 80.00, respectively.The cases demonstrate applicability of the evaluation structure beyond the main Phaser setting.Source: H. Additional 3D Evaluation Results · p. 60, Table S21 · p. 60
  • For Sonic, the evaluator identifies a rail-landing minimum-speed rule that can increase speed when projected tangent velocity is smaller, pointing to a targeted velocity-update revision.The suggested follow-up is to test low-speed rail landings against the fixed contract.Source: H. Additional 3D Evaluation Results · p. 60

I.1. GDD-Specific Revision Cases

Matched probes compare initial builds, self-revision, and Source + Replay + Playtest against specific GDD requirements. The selected cases illustrate repairs in interaction sequences and state or feedback behavior, alongside representative generated gameplay from Small and Big games.

I.1. GDD-Specific Revision Cases

  • Across the selected revision cases, Source + Replay + Playtest satisfies requirements left unresolved after self-revision, including pointer-following fragment movement and oxygen-triggered Hull loss.These probes use matched preconditions and inputs; the figure's selected outcomes do not establish complete GDD compliance.Source: §I.1 · p. 61, Figure S23 · p. 62
  • In Toybox Rider, Source + Replay + Playtest opens a discard confirmation while preserving the run, then returns to Pause with the same values after cancellation.The initial build stays in Pause and self-revision restarts immediately with zero run Bolts.Source: §I.1 · p. 61
  • In Night Shift Rx, Source + Replay + Playtest pairs CODE BLUE on boss timeout with zero Life and the required red-grey ring; self-revision leaves Life near 79.8 and the ring green.The comparison starts with 0.05 seconds remaining and advances the game by 0.1 seconds.Source: §I.1 · p. 61
  • In Alias Alchemy Shop, Source + Replay + Playtest retains the chosen alias and renders the shop, whereas self-revision selects a candidate but encounters a TypeError during shop rendering.The comparison tests the full selection-to-shop sequence, not candidate selection alone.Source: §I.1 · p. 61
  • Selected Small-game examples include selective clicking, visual discrimination, ring switching, memory input, cup-shuffle tracking, shape sorting, dice banking, and combination dialing.The examples show prompts, controls, and feedback across distinct generated interfaces.Source: §I.1 · pp. 63–64
  • Selected Big-game examples span fishing and rewards, traversal and defense, tactical deployment, fleet management, shadow traversal, soul encounters, persistent shop progress, and excavation research.The examples retain gameplay context, controls, state information, and interface feedback.Source: §I.1 · p. 63, Figure S26 · p. 66, Figure S27 · p. 67

J. Limitations

The benchmark primarily covers synthetic 2D single-player games developed in Phaser; broader generalization and evaluator reliability remain uncertain.

J. Limitations

  • Results cover 100 synthetic GDDs for 2D single-player games developed in Phaser.Whether they generalize to human-authored specifications, other engines, or broader software tasks remains to be established.Source: J. Limitations · p. 68
  • Three additional Three.js games show applicability to 3D environments, but not performance across a broad range of engines or genres.Source: J. Limitations · p. 68
  • Generative-model-based contract construction and evidence interpretation leave omissions and evaluator bias possible despite fixed contracts and repeated judgments.Source: J. Limitations · p. 68

Related papers

Loading 2609.39564v1…