PaperlayerView exact revision on arXiv

Source-linked AI summary

Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate

Hugh Xuechen Liu, Kıvanç Tatar

arXiv:2605.07342v1cs.LGcs.AIcs.SE

TL;DR

Compile-pass rate is an incomplete signal for multi-component executable artifacts because declarative structure, imperative logic, and engine-specific runtime conventions can fail independently. Mage evaluates compile success, runtime success, structural fidelity, and mechanism adherence across four open-weight models, three seeds, and 858 generation attempts. 43% mean runtime-pass rate for direct NL-to-C# generation is paired with mechanism F1 of 0.12, whereas structural IR conditioning reaches mechanism F1 up to 1.00.

  • Problem

    Compile-pass rate is an incomplete signal for multi-component executable artifacts because declarative structure, imperative logic, and engine-specific runtime conventions can fail independently.

    Source: §1 · p. 1

  • Method

    Mage evaluates compile success, runtime success, structural fidelity, and mechanism adherence across four open-weight models, three seeds, and 858 generation attempts.

    Source: §1 · p. 2

  • Results

    43% mean runtime-pass rate for direct NL-to-C# generation is paired with mechanism F1 of 0.12, whereas structural IR conditioning reaches mechanism F1 up to 1.00.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The findings show that compile rate is anti-correlated with domain-specific functional correctness, while behavior-only and full-scene IR are statistically indistinguishable.

    Source: §1 · p. 2, §8 · p. 9

    Runtime success measures clean builder execution rather than whether a human player can complete the win condition, and the study covers only Unity 2022.2 and four open-weight models.

    Source: §7 · p. 9

Abstract

from arXiv · show

Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') -- compile success, runtime success, structural fidelity, and mechanism adherence -- applied to 858 generation attempts across four open-weight LLMs (7B--30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C\# generation achieves the highest runtime-pass rate (43\% mean) yet produces structurally vacuous scenes (mechanism $F_1 \approx 0.12$). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure ($F_1$ up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar $p = 1.0$), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the benchmark, replay logs, and per-record metrics for independent verification.

1 Introduction

The paper introduces Mage, a controlled benchmark for testing when compile success diverges from functional correctness in executable Unity scene synthesis. Across 858 attempts, structural IR conditioning trades runtime rate for substantially greater mechanism fidelity, while behavior-only and full-scene IR perform indistinguishably.

Benchmark and dataset

  • The benchmark covers 26 hand-crafted Unity goal patterns and 858 generation outputs, with controlled prompt templates, API rules, models, and seeds.Only IR level, model identity, and random seed vary across conditions.Source: §1 · pp. 1–2

Evaluation methodology

  • Mage evaluates compile success, runtime success, structural fidelity, and mechanism adherence to expose failures that compile-rate-only evaluation misses.The study applies the protocol to four open-weight models ranging from 7B to 30B parameters.Source: §1 · p. 2

Empirical findings

  • 43% mean runtime-pass rate for direct NL-to-C# generation coexists with mechanism F1 of 0.12, revealing structurally and behaviorally vacuous scenes.The strongest model reaches 80% runtime success, but the scenes remain indistinguishable from a no-trace baseline.Source: §1 · p. 2
  • 14% runtime-pass rate under behavior-level IR accompanies mechanism F1 of 0.82–1.00, reversing the apparent advantage of direct generation.Structural grounding recovers domain-faithful behavior despite reducing runtime success.Source: §1 · p. 2
  • Behavior-only and full-scene IR are statistically indistinguishable across all four axes, with McNemar p = 1.0 for every model.This indicates saturation at the tested input-level granularity choices.Source: §1 · p. 2

Empirical characterization

  • The paper frames failures through domain completeness, API-mapping adequacy, and LLM execution fidelity.This three-factor decomposition interprets when structural grounding fails to produce correct code.Source: §1 · p. 2

2 Related work

Existing code-generation benchmarks and scene-generation systems provide limited evidence for executable, domain-faithful game scenes. The paper addresses this gap with an automated multi-axis benchmark for structural and behavioral evaluation.

Code-generation evaluation

  • Existing benchmarks judge standalone or repository-scale code mainly by fixed test-suite success, missing signals when compilability and functional correctness diverge in multi-component artifacts.This limitation affects HumanEval, APPS, and SWE-bench-style evaluation.Source: §2 · p. 2

Scene and game generation

  • Prior scene-generation work produces spatial arrangements, structured descriptions, or level geometry but generally does not generate runtime-executable code for a specific engine API.The paper distinguishes these systems from executable Unity scene synthesis.Source: §2 · p. 2

Structured conditioning

  • A prior semantic-level Unity IR study reported 0% compilation across thousands of attempts, motivating investigation of which structural grounding level is sufficient.The cited result establishes a difficult baseline rather than resolving the granularity question.Source: §2 · p. 2

Benchmark gap

  • No existing benchmark provides multi-axis automated evaluation for LLM-generated executable game scenes.Mage proposes such a benchmark and evaluation protocol.Source: §2 · pp. 2–3

Structured conditioning

  • Structured conditioning spans input-level schema injection, parameter-efficient tuning, and decoding-time grammar constraints, but these approaches do not directly provide Mage’s executable-scene evaluation.The paper situates its IR conditioning within this broader design space.Source: §2 · p. 3

3 Benchmark Construction

The benchmark uses 26 verified Unity mini-games and automatically extracts behavior-only or full-scene IRs from flattened ground-truth scenes. Controlled prompt templates then isolate the effect of IR content on runtime scene generation.

3.1 Game Design Patterns and Goal Playable Concept

  • The benchmark contains 26 single-screen Unity 2D mini-games spanning resource, competitive, spatial, and stealth goal patterns with explicit win and lose conditions.Goal patterns enable automated structural and behavioral evaluation without expert judgment for every output.Source: §3.1 · p. 3

3.2 Ground-truth scenes

  • Each ground-truth scene includes prefab assets, MonoBehaviour scripts, and scene-level configuration, then undergoes prefab flattening and manual playthrough verification.Flattening expands prefab instances into constituent scene objects and components.Source: §3.2 · p. 3
  • Ground-truth scenes serve both as IR sources and as comparison targets for structural fidelity and mechanism adherence, with flattened Unity and parsed JSON releases.The parsed representation records scene graphs, component types, inspector values, script bindings, and condition chains.Source: §3.2 · p. 3

3.3 IR extraction pipeline

  • The automated extraction pipeline parses scenes, builds demand manifests, extracts referenced data, traces behavior paths, and optionally extracts all scene components.These stages are shared across the two granularity levels except for the scope of scene extraction.Source: §3.3 · pp. 3–4
  • Behavior-only IR captures gameplay logic and referenced configuration, whereas full-scene IR additionally includes camera, boundaries, passive obstacles, and rendering components.Both levels share Steps 1–4; full-scene executes Step 5 over all GameObjects.Source: §3.3 · p. 4, Table 1 · p. 4
  • The two IRs are deliberate structural endpoints: behavior-only is minimal for reproducing win/lose chains, while full-scene is maximal short of runtime state.Intermediate granularities are left for follow-up work.Source: §3.3 · p. 4

3.4 Prompt templates

  • Three prompt templates generate one runtime-builder C# file under no-IR, behavior-only, and full-scene conditions, with ground-truth-derived IR supplied rather than generated.The GameBuilder Awake() method constructs the scene and gameplay classes are defined inline.Source: §3.4 · p. 4
  • API-mapping rules remain identical across conditions, so generated-code differences reflect injected IR content rather than prompt-engineering differences.No-schema receives pattern markdown, while IR conditions receive the corresponding IR JSON.Source: Figure 1 · p. 4, §3.4 · p. 5

4 Evaluation Protocol of Multi-Axis Game Evaluation: ‘Mage’

Mage evaluates Unity scene generation through four axes spanning compilation, execution, structural fidelity, and gameplay mechanism adherence. Its definitions and paired statistical procedures make runtime and domain-faithfulness failures measurable beyond compile status.

4 Evaluation Protocol of Multi-Axis Game Evaluation: ‘Mage’

  • Mage defines four axes: compile success, runtime success, structural fidelity, and mechanism adherence, linking execution outcomes to ground-truth scene similarity.The implementation retains backward-compatible metric labels in the released harness.Source: §4 · p. 5

4.1 Compile and runtime success

  • Runtime success requires compilation, GameBuilder.Awake(): OK, and a clean-exit signature, making it a strict subset of compile success.The released batch harness emits both runtime markers and replay logs.Source: §4.1 · p. 5

4.2 Structural fidelity

  • Structural fidelity computes multiset F1 for scripts, GameObject names, component types, and tags against the ground-truth scene.Scene-settings blocks and the default Untagged tag are excluded from their respective component and tag measures.Source: §4.2 · p. 5
  • Structural fidelity is defined only for runtime-pass records, so its denominator varies by condition and is reported with each result.This restriction means structural scores do not cover failed runtime artifacts.Source: §4.2 · p. 5

4.3 Mechanism adherence

  • Mechanism adherence statically compares win/lose-path steps, effects, and conditions with no runtime execution, so it applies to all 858 attempts.Lose-path scores are null for the 16 patterns without lose paths.Source: §4.3 · pp. 5–6

4.4 Statistical evaluation

  • Pass@k uses an unbiased estimator over three seeds and macro-averages across 26 patterns, with pass defined as runtime success.The evaluation reports k ∈ {1, 2, 3}.Source: §4.4 · p. 6
  • Condition comparisons use paired McNemar exact tests on per-pattern any-pass aggregation, reporting p-values, discordance odds ratios with 95% Woolf confidence intervals, and Cohen’s h.Each model-pattern pair contributes one paired observation after aggregating three seeds.Source: §4.4 · p. 6
  • Released replay logs and per-record metrics reproduce every reported number without rerunning generation or replay.This supports independent verification of the evaluation results.Source: §4.4 · p. 6

5 Empirical Study

The empirical study evaluates four open-weight LLMs on executable Unity scene generation using controlled no-schema and two IR conditions. Runtime success favors direct generation, while fidelity favors IR conditioning; within-IR granularity is indistinguishable, and compile failures reveal condition-specific semantic bottlenecks.

5.1 Setup

  • The study analyzes 858 generation attempts from four 7B–30B models across 26 patterns, three conditions, and three seeds, excluding one full-scene scenario for context overflow.All conditions use identical prompt templates and API mapping rules; only IR level, model identity, and random seed vary.Source: §5.1 · p. 6

5.2.1 Compile-correctness divergence

  • 43% mean runtime success for direct NL-to-C# exceeds behavior-only IR’s 14% and full-scene IR’s 21%, with Qwen3 reaching 80% without schema.Qwen3 falls from 80% no-schema runtime success to 37% with behavior-only IR and 42% with full-scene IR.Source: §5.2.1 · pp. 6–7, Table 2 · p. 7
  • Mechanism win-step F1 is approximately 0.115 for all no-schema cells but rises to 0.82–1.00 on the strongest model and 0.61–0.97 across IR-conditioned cells.Structural fidelity is evaluated only on runtime-pass records, whereas mechanism adherence covers all 858 attempts.Source: §5.2.1 · p. 7, Table 3 · p. 7

5.2.2 Within-IR granularity saturation

  • McNemar p = 1.0 across all four models shows no statistical difference between behavior-only and full-scene IR at any evaluated axis.IR-versus-no-schema comparisons are significant for three of four models on at least one IR level, with no-schema having higher runtime success.Source: §5.2.2 · p. 7

5.2.3 Failure mode taxonomy

  • Semantic compile errors dominate, led by CS0122 with 991 occurrences, CS0029 with 730, CS0136 with 504, and CS1061 with 495.CS0246 occurs only under IR conditioning, while CS0122 is much more frequent with behavior-only IR than full-scene IR.Source: §5.2.3 · pp. 7–8, Table 5 · p. 8

6 Discussion

The discussion explains why compile-pass rate diverges from functional correctness and organizes the findings around domain completeness, API-mapping adequacy, and LLM execution fidelity.

Compile rate alone as a misleading evaluation signal

  • Compile rate measures parser-accepted programs rather than complete game artifacts, allowing structurally vacuous scenes to appear successful.No-IR generation often produces small, locally consistent programs that omit the target scene’s defining structure and behavior.Source: §6 · p. 8

A three-factor interpretation

  • Domain completeness explains why behavior-level information can suffice, although behavior-only inspector listings increase CS0122 errors to 862 versus 117 under full-scene IR.The three-factor framing treats richer information as not uniformly beneficial.Source: §6 · p. 8
  • API-mapping adequacy creates a visible translation cost: CS0246 occurs 469 times under IR conditioning and zero times under no-schema generation.The fixed rule block does not exhaustively cover the explicit IR-to-API translation required by IR conditions.Source: §6 · p. 9
  • LLM execution fidelity dominates cross-model variation, with behavior-only IR runtime success ranging from 1% to 37% across models.These differences occur under the same input, while the decomposition remains interpretive rather than causally identified.Source: §6 · p. 9

Failure distribution reshapes under IR conditioning

  • IR conditioning reshapes failures toward Unity semantic errors because models must resolve additional types, members, and API references.CS0246 and CS0122 concentrate under IR conditions rather than reducing failure overall.Source: §6 · p. 9

Implications for benchmark design

  • Benchmarks for heavy domain-specific frameworks should report parser-level, runtime-level, and domain-fidelity outcomes instead of relying on a single pass signal.The paper releases a benchmark and harness designed to support evaluation beyond compile-pass rate.Source: §6 · p. 9

7 Limitations •

The study’s conclusions are bounded by its runtime criterion, single-engine setting, limited model coverage, sample size, and interpretive factor decomposition.

Runtime playability not measured

  • Runtime success does not measure whether a human player can complete the win condition, although the released harness supports playability evaluation.The protocol measures clean execution rather than full human-playable completion.Source: §7 · p. 9

Single engine

  • The protocol is validated only on Unity 2022.2 and 26 goal patterns, so cross-engine generality remains untested.Godot and three.js are identified as natural next settings for replication.Source: §7 · p. 9

Limited model coverage

  • Four open-weight models from 7B–30B active parameters under one hyperparameter configuration do not constitute an exhaustive model survey.Closed-source models, larger open-weight models, and specialized code agents remain future work.Source: §7 · p. 9

Sample size suitable for evaluation, not training

  • The 26 patterns support evaluation comparisons but are insufficient for de-novo training.Follow-up work should plan for frozen pretrained encoders.Source: §7 · p. 9

Three-factor decomposition is interpretive

  • The three-factor decomposition organizes observations but does not independently measure causal contributions.Controlled factorial interventions are needed for causal estimates.Source: §7 · p. 9

8 Conclusion

The paper introduces a benchmark for structural IR conditioning in Unity scene generation and shows that runtime success can trade off against domain-faithful structure.

8 Conclusion

  • Across 858 attempts, direct natural-language-to-C# generation has the highest runtime-pass rate yet produces structurally and behaviorally vacuous scenes with mechanism F1 ≈0.12.The study covers 26 patterns, two IR granularities, four open-weight LLMs, and four evaluation axes.Source: §8 · p. 9
  • Structural IR conditioning lowers runtime-pass rate but recovers domain-faithful structure and mechanism, reaching mechanism F1 up to 1.00.This establishes the central runtime-versus-fidelity trade-off reported by the study.Source: §8 · p. 10
  • Behavior-only and full-scene IR are statistically indistinguishable, indicating input-level granularity saturation for this benchmark.The paper reports this result alongside released patterns, extraction tools, harnesses, outputs, replay logs, and per-record metrics.Source: §8 · p. 10
  • The four-axis protocol is designed for extension across engines, model families, and generation strategies.Future work will test whether the compile-correctness divergence persists across those settings.Source: §8 · p. 10

A.1 Goal Playable Concepts

The goal-playable concepts are Unity mini-games whose scenes, controls, interactions, and win conditions instantiate distinct gameplay patterns.

A.1 Goal Playable Concepts

  • In “Ownership,” the player moves a blue box with WASD and turns colliding white boxes blue by claiming them.The initial scene contains one large blue box and multiple small white boxes.Source: §A.1 · p. 12
  • The “Ownership” win condition is to own all eight white boxes, with no lose condition.The color change indicates that a white box has been owned.Source: §A.1 · p. 12

A.2 Prompt templates

This appendix section specifies prompt variants and the associated evaluation scope, preprocessing, dataset metadata, metric mappings, and IR schemas. It also illustrates the Unity scene IR format and distinguishes behavior-only from full-scene IR while maintaining shared API rules.

A.2 Prompt templates

  • The three prompt templates share API-mapping rules but differ in IR content and instructions: no schema, behavior-only IR, or full-scene IR.All generate a runtime-built Unity scene in one C# file, prohibit Editor APIs, and enforce portability-oriented implementation constraints.Source: §A.2 · pp. 12–15

A.3 Per-pattern results

  • Table 6 reports per-pattern structural-fidelity and mechanism-adherence aggregates, while compile and runtime rates remain aggregate elsewhere.Fidelity averages four structural subdimensions over IR-conditioned runtime-pass records; mechanism averages win-path steps F1 over all 858 records.Source: §A.3 · p. 16, Table 6 · p. 16

A.4 Compile error taxonomy

  • The released harness reproduces the per-cell compile-error distribution underlying Table 5, with full model-condition splits in metric CSVs.This passage describes the released error-taxonomy support without supplying individual error frequencies.Source: §A.4 · p. 16

A.5 Dataset metadata

  • The dataset’s Croissant 1.0 metadata file documents artifacts, checksums, and field-level schema and is validated with the official mlcroissant library.The metadata covers patterns, ground-truth scenes, parsed JSON, both IR levels, generation outputs, and replay logs.Source: §A.5 · p. 16

A.6 Mechanism adherence: lose paths

  • Lose-path mechanism adherence is evaluated only for the 10 of 26 patterns with explicit lose conditions, using 30 records per cell.The other 16 patterns terminate only on win; Table 7 reports means over those 10 lose-defining patterns.Source: §A.6 · p. 17, Table 7 · p. 17

A.7 Prefab flattening

  • Prefab instances are flattened in Unity Editor 2022.2 because resolving cross-file fileID references complicates IR extraction.The flattened scenes replace prefab blocks with constituent objects and are manually verified as functionally equivalent canonical ground truth.Source: §A.7 · p. 17

A.8 Metrics implementation mapping

  • The released harness maps compile and runtime success to m1_compile and m1_exec, structural fidelity to four F1 labels, and mechanism adherence to six win/lose F1 labels.The code repository documents this correspondence between paper-facing metric names and implementation identifiers.Source: §A.8 · p. 17

A.9 IR schema and example

  • Both IR levels share pattern, behavior, condition_path, prefab_refs, tags, and metadata fields, while full-scene IR additionally provides a scene GameObject list with components and transforms.The example shows behavior-only content and a full-scene addition with Unity-internal m_* fields stripped.Source: §A.9 · pp. 17–18
  • The IR example specifies a MonoBehaviour data object with a GoalManager class and goal-tracking fields, including goalCount 8 and currentCount 0.The excerpt also includes setGoal: 1.Source: §A.9 · p. 19
  • The example encodes a transform position with x, y, and z coordinates all set to 0.Source: §A.9 · p. 19
  • The serialized scene is explicitly truncated, with 10 more GameObjects omitted from the example.Source: §A.9 · p. 19

Related papers

Loading 2605.07342v1…