PaperlayerView exact revision on arXiv

Source-linked AI summary

JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

Jianwen Sun, Chuanhao Li, Zizhen Li, Yukang Feng, Fanrui Zhang, Yifei Huang, Yu Dai, Kaipeng Zhang

arXiv:2606.19830v2cs.SEcs.CL

TL;DR

Project-level code engineering on professional game engines lacks large-scale datasets and deterministic evaluation. The paper constructs JamSet and JamBench with a Godot-based verification pipeline and evaluates generation and completion tasks. Results show a strong project-scale capability cliff, limited Code Agent gains beyond compilation, and effective fine-tuning with JamSet.

  • Problem

    Project-level code framework generation on professional game engines remains largely unexplored because large-scale datasets and deterministic evaluation methods are lacking.

  • Method

    The paper builds JamSet and JamBench from Godot projects using deterministic verification, then evaluates theme-driven generation and multi-granularity code completion with compilation, SCS, and BAS.

  • Results

    Project scale is a hard capability bottleneck: runtime pass rates drop from 80.4% on small projects to 5.7% on large ones, while Code Agents improve compilation without improving runtime behavioral quality.

  • Takeaways & Limitations

    JamSet supports training that improves compilation and structural completeness while narrowing systematic engineering-practice gaps from human developers.

  • Takeaways & Limitations

    The dataset focuses on Godot and preliminary training validation without thorough ablation experiments, excluding other major engines and limiting training conclusions.

Abstract

from arXiv · show

Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.

1 Introduction

Project-level code framework generation on professional game engines remains underexplored because large-scale datasets and deterministic evaluation methods are lacking. The paper addresses this gap with Godot-based datasets, a verification pipeline, and benchmark evaluations that expose scale-related limitations.

  • Project-level code framework generation on professional game engines remains largely unexplored because large-scale datasets and deterministic evaluation methods are absent.
  • Game behavior is context-dependent, making hand-written tests expensive to scale, VLM- or LLM-based scoring subjective, and unit tests insufficient for runtime interactions.
  • Godot’s text-based project format and headless execution support LLM-compatible processing and automated runtime behavior collection.
  • 8,133 verified projects were distilled from over 240,000 repositories, with 300 manually verified projects forming JamBench and the remainder forming JamSet.
  • 80.4% versus 5.7% runtime pass rates on small versus large projects reveal a capability cliff as project scale increases in Task2a.
  • Code Agents improve compilation pass rates without improving runtime behavioral quality, while fine-tuning on JamSet improves compilation, structural completeness, and human-like engineering practices.

2 Related Works

Prior work advances game assets, gameplay design, web games, and local engine modifications, but existing benchmarks and evaluation methods remain limited for project-level professional-engine code.

  • Game-generation research covers art assets, gameplay design, web games, and local modifications on professional engines, including Godot, Unreal Engine 5, and Unity.
  • Existing game-domain datasets are substantially smaller than project-level research needs, ranging from 20 tasks in AutoUE to 2,219 single-file samples in V-GameGym.
  • Table 1 distinguishes related works by whether they support both from-scratch generation and multi-granularity completion.
  • Hand-written scripts are precise but costly, VLM- or LLM-based scoring is scalable but subjective, and unit testing misses game-specific runtime behavior.
  • The proposed framework addresses these evaluation limitations with a deterministic Godot headless pipeline spanning compilation through runtime behavior collection.

3 Dataset Construction

The dataset construction process combines Game Jam and repository collection with filtering, annotation, deterministic behavior collection, and manual benchmark verification. It produces 8,133 verified projects and separates 300 benchmark projects from 7,833 training projects.

  • Game Jam entries provide open-source, time-constrained game projects, while collection from three major platforms yields 37,588 repositories and 5,034 identified Godot projects.
  • A code-length analysis found ρ = 0.445 (p < 0.001) and motivated thresholds of game_lines ≥1,200 and addon_lines <1,000.
  • Approximately 96% of over 240,000 candidate repositories were filtered out for missing files, compilation errors, version incompatibility, or runtime crashes.
  • Structured annotations include automatically extracted project manifests, LLM-assisted gameplay descriptions and genre labels, evaluation configurations, and hybrid asset annotations.
  • Deterministic behavior collection runs projects for 60 seconds, generates input strategies from eval_config, and retains 8,133 projects with meaningful runtime behavior.
  • The benchmark samples 300 projects for manual verification, while 7,833 remaining projects are converted into multi-turn dialogue training data spanning 21K to 197K tokens.

4 Benchmark Design

JamBench evaluates project-level Godot engineering through theme-driven generation and multi-granularity completion, using deterministic verification and complementary structural and behavioral metrics.

  • Task 1: Theme-driven generation: Task 1 requires complete Godot projects from Game Jam themes, optionally adding gameplay descriptions to separate creative planning from engineering implementation.Experiments use 50 real Game Jam themes and repeat each experiment three times.
  • Task 2: Multi-granularity code completion: Task 2 evaluates completion at function-, script-, and full-script-level granularity while preserving scene files and providing project context.Each granularity clears 30–50% of relevant code, and the test set contains 300 benchmark projects.
  • Dataset structure: Figure 3 organizes the corpus by engine trends, code-size/rating relationship, three code-size tiers, and the top 10 genres across 8,133 projects.The tiers use 4K and 15K code-line boundaries, while the quality threshold is 1.2K lines based on Spearman ρ=0.445 (p<0.001).
  • Deterministic verification: The four-level pipeline uses L1 for file integrity, L2 for compilation correctness, L3a for runtime stability, and L3b for deterministic behavior collection.During evaluation, L1/L2/L3a are pass/fail tests, while L3b supplies data for BAS rather than filtering projects.
  • Evaluation metrics: L1, L2, and L3a provide pass rates, while SCS and BAS quantify complementary aspects: static structure and runtime behavior.Low SCS with compilation passing indicates minimal code, whereas high SCS with low BAS indicates structurally complete but behaviorally incorrect output.
  • Evaluation metrics: SCS measures static completeness across seven Godot project dimensions, including scripts, scenes, inputs, functions, nodes, non-empty functions, and signals.For Task 1, reference values are same-genre dataset means; for Task 2, references come from the original project.
  • Evaluation metrics: BAS compares runtime behavior across seven numeric dimensions and one signal-overlap set dimension, using only dimensions with non-zero coverage above 50%.Task 2 uses the original project's behavior data as references, while Task 1 uses dataset means.
  • Runtime behavior collection: Behavior collection preprocesses project structure offline, generates deterministic rule-based inputs, and executes games headlessly for 60 seconds to collect seven-dimensional behavior data.The first 10 seconds navigate menus and the remaining 50 seconds perform gameplay interaction, aiming to trigger active nodes rather than complete the game.

5 Experiments

The experiments evaluate direct LLMs, fine-tuning, and Code Agents across generation and completion tasks using verification pass rates plus structural and behavioral scores. Results show that models can compile projects more reliably than they can produce complete, behaviorally aligned game frameworks, especially as project scale grows.

  • 70.7% average L3a pass rate on Task 1 contrasts with much lower SCS and BAS scores, indicating compilable but structurally minimal projects.
  • Gameplay descriptions improve most models’ SCS and BAS but reduce compilation pass rates, exposing a trade-off between design complexity and engineering capability.
  • Qwen3.5-27B-SFT improves over its base model in both compilation rate and SCS, supporting JamSet’s training-data effectiveness.
  • Task 2 shows a sharp performance cliff from Small to Large projects across all models and completion granularities.
  • 82.7% versus 77.3% L3a on Task 1a for Claude+Agent, and 84.0% versus 72.7% for DeepSeek+Agent, show improved compilation with agent assistance.
  • Claude+Agent changes SCS only from 0.41 to 0.42 and BAS from 0.11 to 0.13, showing minimal structural or behavioral improvement.

6 Analysis

Analysis identifies project scale, architectural design, and engineering practices as central barriers beyond compilation. Case studies show that verification can pass minimal shells or miss cross-file behavioral drift.

  • Script-level completion achieves higher compilation pass rates than function-level completion, while function-level completion achieves higher SCS and BAS.Function-level restoration must remain consistent with retained interfaces and logic, whereas script-level rewriting permits internally consistent code.
  • Function-level completion demands contextual precision because a single incorrect function can break compilation, so completion granularity does not form a simple difficulty progression.
  • Over 30 percentage points higher average L3a pass rates from Code Agents do not produce corresponding SCS or BAS gains.
  • Model-generated projects differ from real projects by relying on hard-coded key events and concentrating logic in single scripts rather than using abstract input interfaces and autoloads.Real projects average 6.9 input actions, and 76.3% use autoload scripts for global state management.
  • Compilation alone cannot distinguish a runnable shell from a real game: projects may pass verification while containing minimal core logic and low SCS.Qwen’s project included 129-line and 119-line manager modules but only 2 lines in player_controller.gd.
  • Engine-specific Godot type constraints can cause compilation failure even when a model’s general-purpose algorithm is correct.
  • A functionally equivalent implementation can compile and run while omitting state variables required by other scripts, producing high SCS but low BAS.The reported case has SCS 0.56 and BAS 0.27.

7 Conclusion

The paper introduces JamSet and JamBench as a project-level game-code dataset and benchmark for a professional engine. Its evaluations identify scale, architectural design, and engineering-practice gaps, while fine-tuning narrows those gaps.

  • JamSet and JamBench are presented as the first project-level game code framework dataset and benchmark on a professional game engine.
  • A deterministic four-level Godot headless pipeline distills 8,133 verified 2D game projects from over 240,000 repositories.
  • The benchmark evaluates from-scratch generation and multi-granularity completion using Structural Completeness Score and Behavioral Alignment Score.
  • Frontier-model evaluation finds project scale to be a hard capability bottleneck, Code Agent gains limited to compilation, and systematic gaps from human engineering practices.
  • Fine-tuning on JamSet effectively narrows the observed engineering paradigm gaps.

Scope and Limitations

The work deliberately evaluates reproducible code frameworks rather than complete games with subjective art and audio assets. Its dataset and benchmark are currently limited to Godot, and training exploration remains preliminary.

  • The study evaluates code frameworks instead of complete games with art and audio because asset assessment is subjective and can undermine reproducibility.
  • A well-structured code framework is treated as a meaningful self-contained objective because it supports arbitrary asset integration and flexible design extensions.
  • The dataset covers Godot but not Unity, Unreal, or other engines with significant market share.
  • Model-training exploration is preliminary and lacks thorough ablation experiments because the work focuses on dataset construction and benchmark design.

Ethics Statement

The paper uses open-source game projects and automated model-assisted annotation while keeping verification model-independent, and commits to public release. It positions the framework as more scalable and reproducible than several existing code-engineering benchmarks.

  • Open-source projects are drawn from Ludum Dare, itch.io, Global Game Jam, and GitHub, with only openly licensed projects included.
  • Developer identity markers are removed, and the dataset contains no personal privacy information.
  • Language and vision-language models generate annotations only; all four verification levels are automated and use no model judgment.
  • The authors will publicly release the complete dataset, evaluation framework, and tool code.
  • The framework evaluates runtime behavior without pre-written test cases, unlike unit-test-based benchmarks, while avoiding per-project manual setup.
  • The work claims differentiated advantages in scale, automation, and evaluation dimensions across game and general code benchmarks.

B Additional Experiment Results

Additional experiments show that completion granularity and agent use affect compilation more than higher-level project quality. Fine-tuning data construction and behavioral coverage provide supporting benchmark details.

  • Task 2 Results: Script-level completion achieves higher compilation pass rates than function-level completion for both direct LLMs and Code Agents.In the Small tier, Claude reaches 88% versus 84% for direct completion and 94% versus 90% with an agent.
  • Task 2 Results: Agents significantly improve compilation rates, but Structural Completeness Score and Behavioral Alignment Score remain largely unchanged.For Medium-tier 2b, Claude’s L3a rises from 62% to 76%, while Medium 2a SCS is 0.80 with an agent versus 0.82 for the LLM and BAS is 0.52 versus 0.51.
  • Interpretation: Compilation is only a minimum threshold; scene structure, cross-file coordination, and runtime interaction are the higher-level quality indicators.
  • Training Data: JamSet training projects are reverse-engineered into multi-turn dialogues that generate files in dependency-topological order.
  • Training Data: Large projects are excluded from fine-tuning because they exceed the 32K-token cutoff; training uses 7,300 Small- and Medium-tier samples for three epochs.
  • Behavioral Coverage: Behavioral coverage averages 3.9 non-zero BAS dimensions per project, with platformers highest at 4.8 and card games lowest at 2.5.

C.4 Benchmark and Evaluation Statistics

The benchmark spans theme-driven generation and multi-granularity completion across three project-size tiers. Its evaluation uses structured annotations, deterministic input strategies, and fixed runtime execution to compare behavior reproducibly.

  • Benchmark Statistics: Benchmark projects have median code sizes of 2,468, 5,782, and 23,180 lines for Small, Medium, and Large tiers.
  • Benchmark Statistics: Task 2 removes 30–50% of functions for function-level completion, 30–50% of scripts for script-level completion, and 100% of script content for full-script completion.
  • Evaluation Pipeline: Offline annotation identifies player nodes, score and health mechanisms, key signals, menu/gameplay scenes, and expected genre-specific behaviors.
  • Evaluation Pipeline: Rule-based input generation converts annotations into deterministic action sequences, input modes, and menu-navigation strategies without LLM calls.
  • Evaluation Pipeline: Runtime execution uses menu navigation followed by gameplay interaction, with fixed ordering and safeguards against pause-related disruptions.The collection window uses the first 10 seconds for menus and the remaining 50 seconds for gameplay interaction.
  • Menu Navigation: Menus are categorized as absent in 31.2% of projects, simple in 52.4%, and complex in 16.4%, requiring progressively broader fallback handling.

D.5 Silent Project Analysis

Silent projects are excluded because they produce no detectable behavioral signals, and the analysis attributes silence mainly to visual-only feedback, custom input systems, or passive game designs. The training example illustrates dependency-ordered multi-turn project generation.

  • Silent Project Analysis: 416 of 8,549 runtime-stable projects, or 4.9%, produced no meaningful behavioral change during the 60-second collection window.
  • Causes of Silence: Visual-feedback-only games account for 44% of silent projects because their interaction effects are not represented in detectable node-property state changes.
  • Causes of Silence: Custom input systems account for 32% of silent projects by bypassing Godot’s InputAction system or relying on unsupported gestures and raw events.
  • Causes of Silence: Passive or cutscene-style games account for 24% of silent projects and include interactive fiction, autoplay narratives, and autonomous procedural art generators.
  • Silent Project Analysis: Silent projects are excluded because their absent behavioral signals make BAS computation undefined, while their tier rates remain similar across Small, Medium, and Large projects.The reported silence rates are 5.1%, 4.6%, and 4.7%, respectively.
  • Training Example: JamSet examples simulate complete game construction through a blueprint followed by dependency-ordered generation of project, scene, and script files.The representative sample contains 16 turns, 15 files, and approximately 4,200 lines.
Loading 2606.19830v2…