Source-linked AI summary
GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments
Brian La, Sejoon Chang, Ben Kim, Junyoung Bae, Aamish Ahmad Beg, Sei Chang, Gonzalo Gonzalez-Pumariega, Kanav Goyal
TL;DR
Coding-agent benchmarks largely focus on general software engineering, while game-engine C++ changes must also work across runtime behavior and interacting systems. GameEngineBench evaluates scoped implementations in executable Unreal projects using 110 tasks from nine repositories. Across configurations, the strongest reaches 55.5% pass@1, and agents continue to struggle with runtime integration.
Problem
Existing coding benchmarks do not directly measure whether native C++ edits integrate correctly with running game engines, where behavior spans networking, object lifecycles, and interacting systems.
Source: §1 · pp. 1–2Method
Results
Across evaluated configurations, the strongest reaches 55.5% pass@1, while performance is not strictly ordered by overall score.
Source: §6 · p. 7Takeaways & Limitations
Frontier coding agents do not reliably integrate gameplay-facing C++ systems, with failures persisting despite syntax and compilation requirements being met.
Source: §8 · p. 10The authors identify benchmark-coverage expansion and methodological extensions to strengthen evaluation reliability as future work.
Source: §7 · pp. 9–10
Abstract
from arXiv · showhide
Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5\% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.
1 Introduction
GameEngineBench targets a gap in coding-agent evaluation: native C++ changes in game engines must satisfy runtime behavior across networking, object lifecycles, and interacting systems. It evaluates those changes in executable Unreal projects and analyzes agents’ runtime and integration failures.
1 Introduction
- Game-engine changes must satisfy C++ compilation and runtime requirements spanning networking, object lifecycles, callbacks, assets, and existing gameplay systems.A patch may compile yet fail by updating only local state, missing a lifecycle transition, or breaking another system’s interaction.Source: §1 · pp. 1–2
- Game engines also support applications beyond entertainment, including simulation, digital twins, XR, robotics prototyping, virtual production, and interactive training.Mature, executable, stateful, and often public game repositories provide a practical source of tasks for evaluating this broader software setting.Source: §1 · p. 2
- The strongest of 12 evaluated configurations reaches 55.5% pass@1, while 31 tasks remain unsolved by every configuration.Failures cluster around authority, state synchronization, object lifecycles, initialization, and integration with surrounding systems.Source: §1 · p. 2
- GameEngineBench evaluates scoped native C++ edits in functioning Unreal projects using runtime tests and judge auditing for behavioral correctness rather than reference similarity.Its 110 tasks come from nine publicly available repositories; models edit native source files, while projects may use non-code assets at runtime.Source: §1 · p. 2
2 Related Work
Prior coding benchmarks cover isolated synthesis, repository issue resolution, long-horizon tasks, and game creation, but GameEngineBench focuses on runtime-integrated C++ changes in existing Unreal projects. Its distinguishing scope includes server/client correctness and coordination among tightly coupled engine behaviors.
2 Related Work
- Earlier benchmarks cover isolated function synthesis, repository issue resolution, execution-environment tasks, and long-horizon or behavior-focused agent work.GameDevBench emphasizes multimodal game-development tasks, while AutoUE targets end-to-end 3D game creation rather than constrained C++ changes in existing projects.Source: §2 · pp. 2–3
- GameEngineBench uniquely combines existing game-engine repositories, scoped native C++ edits, server/client correctness, and LLM-reviewed runtime behavior.The benchmark shifts toward implementing missing behavior inside an existing runtime system rather than localized bug repair.Source: Table 1 · p. 3, §2 · p. 3
- Across 110 tasks, reference solutions add a mean of 511 lines and a median of 362 lines, exceeding the SWE-bench Verified and SWE-bench Pro reference-edit sizes reported by DeepSWE [10].This characterizes the benchmark’s focus on larger implementation tasks within existing runtime systems.Source: §2 · p. 3
- Correct Unreal solutions coordinate server authority, replication, object lifecycles, and engine-managed subsystems rather than treating them in isolation.A compilable solution can still fail if it runs on the wrong machine, updates only local state, cleans up too late, or registers a component after it is needed.Source: §2 · pp. 3–4
3 Benchmark Design
GameEngineBench supplies buildable Unreal projects with scoped C++ edit instructions, then validates tasks and tests them for behavioral correctness. Its authoring process combines test validation with iterative LLM-judge review and task refinement.
3 Benchmark Design
- Each task gives the model a buildable starting state, editable C++ files, and public instructions, while withholding the test suite until implementation is complete.Afterward, tests are injected and run through Unreal Play-in-Editor automation; judge auditing assesses requested behavior rather than reference-solution similarity.Source: §3 · p. 4
- Task calibration iterates through package and build validation, reference-solution tests, LLM-judge review, task or test revision, and revalidation.Validation checks that the starting state fails on the intended missing behaviors; judge review identifies tests that are too narrow, underspecified, or over-constrained.Source: §3 · p. 4
- Reference-solution screenshots illustrate the gameplay and engine-system behaviors evaluated by benchmark tasks.The figure’s examples are illustrations of evaluated behavior, not a report of task outcomes.Source: Figure 2 · p. 5
4 Current C++ Task Set
The evaluated task set contains 110 tasks from nine open-source Unreal projects and covers gameplay and systems programming within existing codebases. Its primary-area labels coexist with cross-cutting runtime requirements, while some development domains receive lighter coverage.
4 Current C++ Task Set
- The current set contains 110 active tasks from nine Unreal projects, embedded in their original codebases and spanning gameplay and systems programming.This requires working with existing C++ code, engine callbacks, networking setup, and gameplay architecture rather than isolated programming problems.Source: §4 · p. 5
- Tasks span gameplay mechanics, multiplayer, AI and world orchestration, animation, UI and sessions, plugins, object management, data logic, movement, XR, and rendering.Coverage is intentionally lighter for audio, performance, platform support, security, and tooling.Source: §4 · p. 5
- Each task receives one primary game-development area label, though many also exercise authority, state synchronization, object lifecycle, and subsystem initialization.The categories are assigned once for readability, not to imply tasks involve only one runtime concern.Source: Figure 3 · p. 6
5 Evaluation Protocol
The protocol scores behavioral correctness on a single attempt, using an LLM judge to assess whether implementations satisfy the task rather than resemble a reference. Agents work in isolated project copies, and injected tests run through Unreal’s Play-in-Editor automation.
5 Evaluation Protocol
- pass@1 is the fraction of tasks solved by one attempt, with success determined by whether an LLM judge finds the requested behavior satisfied.The judge-based measure is used because tests cannot capture every valid implementation and interaction path.Source: §5 · p. 6
- The LLM judge reviews the behavior specification, tests, execution results, code edits, and reference solution, with cross-family judging supported to mitigate self-judging bias.The review assesses behavioral correctness, not similarity to the reference.Source: §5 · p. 6
- Each agent runs in an isolated copy of the task start state; after solving, tests are injected and executed in Unreal’s Play-in-Editor listen-server mode, with workspaces and logs preserved for auditing.The CLI gives agents the same task payload and requires the modified project to compile.Source: §5 · p. 6
6 Results
Across twelve configurations, the strongest reaches 55.5% pass@1, yet 31 tasks remain unsolved by every configuration. Performance varies by model and task area, and stronger runs tend to fail on runtime behavior and integration rather than compilation alone.
6.1 Success Rates
- 29.1% pass@1 for GPT-5.5 at xhigh reasoning effort trails claude-fable-5 at max effort (55.5%), claude-opus-4-8 at max effort (23.6%), and Gemini 3.1 Pro (18.2%).The GPT-5.5 reasoning-effort sweep rises from 9.1% at medium to 19.1% at high and 29.1% at xhigh.Source: §6.1 · p. 7
6.2 Efficiency Tradeoffs
- claude-fable-5 has the strongest pass@1, while GPT-5.5 at xhigh is faster and less expensive in this run; cheaper agents may handle simpler tasks at scale.Efficiency should be considered alongside the pass@1 leaderboard.Source: §6.2 · pp. 7–8
6 Results
- 55.5% pass@1 is the strongest configuration’s score, while 31 tasks remain unsolved across all evaluated configurations.Figure 4 presents success rates by configuration over the active 110-task set.Source: §6 · p. 7, Figure 4 · p. 8
6.3 Solved Sets Are Complementary
- 79 tasks are solved by at least one configuration, but 31 are unsolved by all twelve, and no task is solved by every configuration.Some tasks are solved by only one or two configurations, so changing models can recover specific behaviors, but model selection alone does not cover the benchmark.Source: §6.3 · p. 8
6.4 Coverage Varies By Task Area
- Save and persistence have the largest unresolved share; at least one task also remains unsolved in each of several other task areas.Those areas include weapons/combat, serialization, UI/game-feature work, AI/world orchestration, animation/locomotion, engine systems/pooling, core gameplay, rendering plugins, and inventory/interaction.Source: §6.4 · p. 8
6.5 Failure Mechanisms
- Higher-performing wrappers usually compile and execute, while their remaining errors are mostly behavioral rather than purely syntactic.Figure 5 decomposes outcomes into passes, test failures, judge-disputed cases, and compile failures.Source: Figure 5 · p. 9, §6.5 · p. 9
- Failures can leave plausible Unreal C++ incomplete at runtime, such as adding an OnRep hook without replicating the source state needed for visible client UI updates.Other examples include missing local-controller checks and omitted constructor defaults, activation timing, or teardown behavior.Source: §6.5 · p. 9
7 Limitations and Next Steps
GameEngineBench covers gameplay-facing systems but omits several important areas of C++ game engineering, and its open-source repositories and PIE listen-server harness constrain how fully it represents professional environments. The authors propose broader task coverage and more reliable evaluation through wrapper standardization and stronger test coverage.
Coverage limitations
- The 110-task set covers gameplay-facing systems but not audio, memory, performance, platform support, editor tools, build systems, or security-sensitive code.Open-source repositories also expose gameplay and framework code more readily than proprietary engine modifications, rendering work, platform integrations, performance infrastructure, or security-sensitive systems.Source: §7 · p. 9
Evaluation boundary
- PIE listen-server automation tests runtime behavior, authority, replication, and lifecycle correctness, but does not fully reflect dedicated-server fleets, console hardware, profiling, large QA matrices, or live operations.Imperfect test suites may miss valid implementations, making evaluation sensitive to test design and judge calibration.Source: §7 · p. 9
Task expansion
- Future task expansion should add coordination across multiple engine systems and underrepresented C++ areas such as audio, performance, editor tooling, build/platform code, and security-sensitive gameplay infrastructure.The authors intend these additions to improve representation while preserving engine-integrated runtime challenges.Source: §7 · pp. 9–10
Evaluation reliability
- Standardizing execution wrappers would make comparisons fairer because observed pass@1 differences partly reflect wrapper effects, while broader test coverage and calibration could reduce reliance on judge intervention.LLM review sometimes converts test failures into valid solutions, indicating that tests do not always capture the full behavior space.Source: §7 · p. 10
8 Conclusion
GameEngineBench evaluates whether generated C++ changes satisfy runtime behavior in executable Unreal Engine repositories. Its results indicate that agents still struggle to integrate reliably with gameplay-facing systems, while the Unreal-focused benchmark leaves other game-development areas for future expansion.
8 Conclusion
- GameEngineBench evaluates runtime behavioral requirements through tasks extracted from executable Unreal Engine repositories, while leaving several game-development areas for future expansion.The benchmark requires outputs to fit behavioral requirements and integrate with surrounding game systems, rather than only meeting narrowly specified instructions.Source: §8 · p. 10
- The 110-task set shows that frontier coding agents do not reliably integrate with gameplay-facing C++ systems, even when their code meets syntax and compilation requirements.The paper states that the observed performance gaps reflect broader challenges in engine- and framework-integrated software development.Source: §8 · p. 10
- GameEngineBench is specific to Unreal Engine and serves as a focused benchmark and foundation for more comprehensive game-engine programming evaluations.The broader challenges described involve generated code operating within existing systems and respecting realistic runtime constraints.Source: §8 · p. 10
A Sample Tasks
Two sample tasks illustrate that benchmark difficulty can come from coordinating several Unreal systems, not just from edit size. Both examples remain unsolved across evaluated configurations and require consistent runtime behavior across interacting systems.
A Sample Tasks
- Both sample tasks remain pass@1 failures for every evaluated configuration and expose different forms of engine-level coordination.The examples were selected to illustrate the kinds of Unreal work represented in the task set.Source: A Sample Tasks · p. 12
- The Zombie System task coordinates zombie spawning, AI behavior, death handling, rewards, server-authoritative damage, replicated feedback, and round progression across eight source files.Correctness also depends on spawn and patrol metadata, melee range, and attack/death feedback to clients.Source: A Sample Tasks · p. 12
- Together, the examples show that hard tasks are not defined only by edit size; they require consistent behavior across interacting Unreal systems.The Zombie System is broad and multiplayer-focused, while the Generated Map Orchestrator is narrower in file count but still integration-heavy.Source: A Sample Tasks · p. 12
- The Generated Map Orchestrator task must populate a procedural grid, reuse pooled actors, replicate map-size changes, synchronize peers, and send one readiness signal after local generation completes.The task also involves editor preview behavior and avoiding premature or duplicate signals and actor leaks across regenerations.Source: A Sample Tasks · p. 12
B Supplemental Result Figures
The appendix provides audit views that expose task-level outcomes, metadata breakdowns, complementarity, compiler behavior, and resource use behind the headline pass@1 results. These views add diagnostic detail while preserving the main text’s aggregate-results focus.
Figure 9
- Figure 9 shows configuration-by-task outcomes, with long red columns marking tasks unsolved across configurations and mixed columns showing outcomes that vary by model choice.The task-level matrix preserves detail hidden by aggregate pass@1 and underpins capability-frontier and complementarity claims.Source: B Supplemental Result Figures · p. 12, Figure 9 · p. 15
Figure 10
- Figure 10’s domain and requirement-category breakdowns are directional evidence because several categories contain few tasks, and help identify areas contributing to the unsolved set.The breakdowns can guide future task expansion; the figure shows pass@1 by game-system domain and requirement category.Source: B Supplemental Result Figures · p. 12, Figure 10 · p. 15
Figure 11
- Figure 11 shows both broadly solved tasks and a substantial no-solve region, while pairwise counts of tasks uniquely solved by one configuration indicate the leaderboard is not simply nested.The pairwise matrix counts tasks solved by the row configuration but missed by the column configuration.Source: B Supplemental Result Figures · p. 13, Figure 11 · p. 16
Figure 12
- Figure 12 reports wrapper-visible compiler invocations and errors, but is not a complete comparison across agent interfaces.The views distinguish configurations that actively iterate through Unreal compilation from cases where compile behavior is less frequent or less visible in saved traces.Source: B Supplemental Result Figures · p. 13, Figure 12 · p. 16
Figures 6 and 13
- Figure 13 offers available-data cost and token diagnostics rather than a definitive pricing comparison, and indicates that stronger configurations often spend more inference, compilation, or review budget per task.Figure 6 separately compares pass@1 with median solve time, mean model cost, and mean output tokens per task.Source: B Supplemental Result Figures · p. 13, Figure 6 · p. 13, Figure 13 · p. 16
C Complementarity Examples
Model successes are not fully nested: configurations miss tasks solved by others, and examples involve distinct AI-initialization and map-lifecycle challenges.
C Complementarity Examples
- Off-diagonal task successes show capability is not fully nested, so leaderboard rank alone misses distinct game-engine reasoning strengths.The examples span AI control and procedural-map lifecycle; additional runs may reveal distinct capability pockets, while unsolved tasks point to needs for better training, tooling, or task-specific reasoning.Source: C Complementarity Examples · p. 17
- Task 15 is solved only by GPT-5.5 at xhigh and combines behavior-tree setup, blackboard observation, client notification, team setup, EQS lookup, and behavior-tree components.Correctness depends on several systems becoming available at the right time.Source: C Complementarity Examples · p. 17
- Task 19 is solved only by claude-fable-5 at max and combines procedural map generation, pooled actors, replication, asynchronous work, and readiness signaling.The task also requires runtime map-size changes and reuse of actors across regenerations.Source: C Complementarity Examples · p. 17
D Reproducibility Notes
- A stronger public release should bundle explicit OS, hardware, wrapper-version, and prompt-metadata files alongside the benchmark artifacts.The repository already contains run scripts, task packages, and saved benchmark snapshots.Source: D Reproducibility Notes · p. 17