PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine

Eray Turkel, Mengsha Sun, Kartik Ayyar, Sean Dunigan, Jack Lu, Vlad Shcherban, Hsiang-Shun Shih, Xin Wang, Tiantian Zhang

arXiv:2610.02563v1cs.LGcs.AI

TL;DR

Outcome-only coding benchmarks cannot distinguish understanding from guessing, while game-engine tasks also require discovering hidden system structure. OpenGameEval evaluates agents in reproducible Roblox sessions and measures both verified outcomes and exploration. Results show different task strengths among similarly performing frontier models and an association between task-relevant inspection and success.

  • Problem

    Outcome-only coding benchmarks cannot distinguish an agent that understands a system from one that guesses, even though game-engine tasks require discovering relevant scripts and object structure.

    Source: §1 · p. 1

  • Method

    OpenGameEval runs agents in reproducible stateful game-engine sessions, separates observation from action, and checks edited scenes and simulated play while annotating task dependencies.

    Source: §1 · p. 2

  • Results

    Frontier models reach similar aggregate pass rates by solving different tasks, while inspecting expected objects before acting is associated with 13.4pp higher pass rates on scene-only tasks and 9.8pp on script-only tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OpenGameEval provides a testbed for studying how agents act in stateful worlds, with exploration measured against task-specific dependencies.

    Source: §1 · p. 2, §9 · p. 10

    The evaluation covers one engine and one language, so its trajectory signal may not transfer unchanged to other engines.

    Source: §8 · p. 9

Abstract

from arXiv · show

We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in reproducible, stateful game-engine sessions and scores each run with executable checks, both on the edited scene and in a simulated play session. Most agentic coding benchmarks require exploration but score only final task success. OpenGameEval separates observation tools from editing tools in its eight-tool action space, so exploration can be measured directly. We measure the pass rates and exploration behavior of 13 frontier models on 84 human-curated core tasks, with 16 attempts per task. The tasks are hard for current models. The best model solves 51.7% of tasks on a single attempt and 39.4% five times out of five, and no tested model solves six of the tasks. Models at the frontier reach similar pass rates by solving different tasks: splitting tasks by the kind of work they require spreads the top five by 5.0pp on script-authoring tasks and 12.5pp on scene-change tasks. Exploration behavior predicts whether a run succeeds. Holding task and model fixed, a run that inspects every object a reference solution touches before acting on it passes 13.4pp more often than a run that inspects none of them on scene-only tasks, and 9.8pp more often on script-only tasks. We release the task suite, its place files, the per-task annotations, a plugin that runs the tasks inside Roblox Studio, and an updated leaderboard under the MIT license at https://github.com/Roblox/open-game-eval.

1 Introduction

OpenGameEval measures both task outcomes and agents’ exploration in a stateful game-engine setting where prompts leave relevant system details undisclosed. Across 13 models, results show low task success, differing strengths across task types, and an association between inspecting task-relevant objects and success.

1 Introduction

  • Outcome-only evaluation cannot distinguish an agent that understood the system from one that guessed, motivating measurement of what agents learn through exploration.Existing coding benchmarks score outputs against fixed tests, while the same passing result can arise from understanding or guessing.Source: §1 · p. 1
  • Game-engine tasks require agents to discover which scripts, server-authority choices, and object-hierarchy locations matter, with some correctness observable only during play.For example, health regeneration may require finding the health scripts and player rig, then checking behavior after damage in a running game.Source: §1 · pp. 1–2
  • OpenGameEval runs agents in reproducible, stateful editor sessions and separates observation from action in an eight-tool interface.It also verifies edited scenes and play-time behavior and uses human-written reference solutions annotated with the objects and scripts a correct run should consult.Source: §1 · p. 2
  • The strongest model solves 51.7% of core tasks on one attempt and 39.4% five times out of five; frontier models also solve different tasks and exploration predicts success.The five leading models’ spread grows from 1.1pp overall to 5.0pp on script-authoring and 12.5pp on scene-change tasks; inspecting expected objects before acting is associated with 13.4pp and 9.8pp higher pass rates on scene-only and script-only tasks, respectively.Source: §1 · p. 2
  • The release includes task definitions, place files, per-task annotations, a Roblox Studio plugin, and a leaderboard under the MIT license.The measurement scheme can also apply to other tool-using agent evaluations that distinguish observation from action and annotate task dependencies.Source: §1 · p. 2

2 Related Work

OpenGameEval brings together in-engine software development, gameplay-based correctness checks, and measured exploration. Related benchmarks measure end-to-end success or progress, while prior work on exploration and trajectories differs in its tasks or level of analysis.

2 Related Work

  • The authors know of no prior work combining deployed-game-engine development, correctness based on simulated gameplay and unit tests, and instrumented exploration.GameDevBench is the closest work, but uses Godot unit tests rather than a driven gameplay session with a server and client; exploration and trajectories are another difference.Source: §2 · pp. 2–3
  • Other stateful-agent benchmarks report end-to-end success, and Terminal-Bench checks final state rather than commands or task-specific dependencies.The cited systems include WebArena, OSWorld, AgentBench, GAIA, and Terminal-Bench.Source: §2 · p. 2
  • AgentBoard’s progress rate measures how close an agent’s state has come to the goal, whereas OpenGameEval measures the fraction of annotated task dependencies consulted.AgentBoard’s rate is the running maximum over steps and reaches one at completion.Source: §2 · p. 2
  • Park et al. (2026) find exploration error predicts success across models; their correlation analysis differs from OpenGameEval’s repeated-run comparison holding task and model fixed.They also report qualitatively different exploration behavior among models with indistinguishable success rates.Source: §2 · pp. 2–3
  • Mehtiyev & Assunção (2026) find context gathering before editing and validation predict success after controlling for task difficulty, using behavioral features shared across tasks.OpenGameEval instead scores investigation against each task’s annotated dependencies.Source: §2 · p. 3

3 The OpenGameEval Framework

OpenGameEval evaluates agents in real Roblox place files with a fixed tool-calling scaffold and separates observation tools from state-changing actions. Its checks cover both the edited scene and behavior in a subsequent play session.

3 The OpenGameEval Framework

  • The framework uses seventeen curated Roblox experiences and evaluates 113 tasks across populated games and bare placefiles, with reference solutions and unit tests for every task.The 113 tasks use eleven experiences; 70 are set in populated games and 43 start from bare placefiles.Source: §3 · p. 3
  • The framework is designed for fidelity, harness agnosticism, and verifiability, using real placefiles and machine-checkable assertions.The experiences range from an empty baseplate to populated games such as a racing game, laser-tag arena, suburban town, and modern city.Source: §3 · p. 3
  • The eight-tool interface separates observation from action, allowing comparisons of whether models differ in observation, action, or their mix rather than only tool-call counts.The engine calls hierarchy nodes such as parts, models, folders, scripts, and services “Instances.”Source: §3 · p. 3
  • Each attempt uses one prompt and a continuing tool-calling loop with a 300-second or 200-call limit, and results reflect the model together with this fixed scaffold.Almost all core attempts end when the model stops calling tools; 1.4% time out and none reaches the call ceiling.Source: §3 · p. 3
  • Sampling parameters are fixed per model family, reasoning effort is high where available, and attempts were collected between May and July 2026.Source: §3 · p. 4
  • Each task must pass both scene-state checks and play-session checks after the edits.The worked school-lights task prompt omits the eight subfolders, their LightsOn booleans, and the switch scripts; the check reads the resulting instance state.Source: §3 · p. 4, Figure 1 · p. 4

4 The OpenGameEval Benchmark

The benchmark separates 84 core feature-building tasks from 29 debugging tasks and analyzes the core suite independently. Core tasks vary in whether they require script authoring, scene changes, or both.

4 The OpenGameEval Benchmark

  • The 113 scored tasks comprise separate suites of 84 core tasks and 29 debugging tasks, with estimates never pooled across suites.Most core tasks open on populated games; the rest begin in an empty world, while debugging tasks provide a planted defect to repair.Source: §4 · p. 4
  • No model solves six core tasks in 16 attempts, although each has a human-written reference solution that passes its checks.Four additional tasks are solved on every attempt, leaving 74 tasks with between-model information; estimates still use all 84, and the ten tasks do not change rankings or paired comparisons.Source: §4 · p. 4
  • Core tasks split into 59 script-authoring tasks and 25 scene-change tasks, but these categories overlap because 29 tasks require both script editing and scene execution.The stricter exploration analysis sets aside those 29 and uses 30 script-only and 25 scene-only tasks.Source: §4 · p. 4
  • Table 1 reports all 13 models on the same 84 core tasks at 16 attempts per task, with pass rates, tool-use measures, errors, and timeouts.Timeouts count as failures, and model comparisons use paired comparisons rather than marginal pass-rate estimates.Source: Table 1 · p. 5

5 Metrics

The metrics distinguish outcome reliability, tool-use behavior, and investigation coverage, while paired comparisons account for shared task difficulty.

Outcome metrics

  • pass@1, pass@5, cons@5, and all@5 estimate per-task success, subsample success, consensus, and unanimity from 16 attempts using unbiased U-statistics.pass@5, cons@5, and all@5 are the probabilities that a uniformly random subsample of five attempts contains at least one, three, or five successes; the gap between pass@5 and all@5 is the consistency gap.Source: §5 · p. 5

Process metrics

  • Exploration measures how much of a task’s annotated dependencies the agent observes before acting; post-edit verification does not count as investigation.Source: §5 · p. 5

Inference

  • Model comparisons use paired per-task pass@1 differences, task-resampled bootstrap intervals, and a Wilcoxon signed-rank check; Fisher’s exact test tests task-level flips.Pairing controls for task difficulty, which dominates the spread across tasks; a comparison is resolved when its bootstrap interval excludes zero.Source: §5 · p. 5

6 Core-Suite Results

Paired comparisons distinguish models whose overall pass rates are close, while task-type splits reveal different capability profiles. The results also show task-level disagreement and uneven gains across model generations.

6 Core-Suite Results

  • A paired test separated 34 of 78 model pairs, with 15 surviving Holm correction, despite the five leaders falling within 1.1pp at 50.6–51.7%.Their marginal intervals overlap, which the paper notes is not evidence of equivalence.Source: §6 · p. 5
  • The five leaders’ solved-task sets had a union of 58 and intersection of 27, outside pooled-placebo intervals of 45–50 and 34–40.The observed divergence was not reached in any of 2,000 placebo draws.Source: §6 · p. 6
  • Across 78 model pairs, the median number of tasks separating them was 19; Claude Fable 5 and Gemini 3 Flash differed by 0.1pp overall but on 17 tasks.Those 17 task differences favored one model on eight tasks and the other on nine.Source: §6 · p. 6
  • For the top five, the spread was 5.0pp on script-authoring tasks, 12.5pp on scene-change tasks, and 13.1pp on debugging, versus 1.1pp overall.Gemini 3.1 Pro ranked first in script authoring and eighth in scene change; Claude Opus 4.8 ranked first in scene change and fifth in script authoring.Source: §6 · p. 6, Figure 2 · p. 7
  • Claude, GPT, and GLM each gained about 6pp in core pass@1 across generations on different kinds of work, while Gemini was flat in aggregate.Source: §6 · p. 6
  • A 2PL fit to the 13 × 84 response matrix placed 79% of ability variance between provider families rather than versions within families, with interval [0.71, 0.87].The authors characterize the fit as diagnostic because it has only 13 respondents.Source: §6 · p. 6
  • Seven high-discrimination tasks ranged from 14% to 90% pass rates and shared expectations implied but unstated in their prompts.Examples include a day–night lighting cycle, a bounce that avoids walls and ceilings, and a flashlight with an actual light.Source: §6 · pp. 6–7

7 Exploration Behavior as a Predictor of Task Success

The section tests whether task-specific investigation predicts run success, using dependencies annotated from reference solutions. Coverage is associated with higher pass probability across scene-only and script-only tasks, and remains informative under stricter controls.

7 Exploration Behavior as a Predictor of Task Success

  • The estimates compare runs of the same model on the same task, with coefficients expressed in percentage points for the binary pass outcome.The model includes task and model fixed effects and uses task-clustered confidence intervals.Source: §7 · p. 7
  • Coverage counts annotated dependencies reached through observation tools before the first action on each dependency, and ranges from none to all.It is measured on annotated tasks; the main analysis uses 37 tasks, split into 24 scene-only and 13 script-only tasks.Source: §7 · p. 7
  • 13.4pp on scene-only tasks and 9.8pp on script-only tasks higher pass probability is associated with inspecting all annotated dependencies before acting rather than none.The corresponding per-standard-deviation associations are +6.4pp and +4.3pp, respectively.Source: §7 · p. 7, Table 2 · p. 8
  • Reading reference-dependent scripts before editing is associated with +8.3pp pass probability per standard deviation and +17.8pp from none to full.The analysis covers 11 tasks; only two scene tasks have the annotation, so a scene-side counterpart cannot be tested.Source: §7 · p. 8
  • The estimated coverage association does not significantly differ between task types: the gap is −1.6pp per standard deviation (95% interval [−7.6, +4.3], p=0.59).Crediting observation at any point rather than only before acting raises the scene estimate from +6.4pp to +7.8pp and leaves the script estimate unchanged.Source: §7 · p. 8
  • Controlling for tool-call count moves the scene estimate from +13.4pp to +12.6pp, while generic pre-edit inspection predicts +6.2pp per standard deviation on scene tasks.The generic feature is not significant on script-only tasks; when both features are modeled, scene-only estimates cannot distinguish their contributions.Source: §7 · p. 8
  • On script-only tasks, coverage retains an association when generic pre-edit exploration is included, while the generic feature does not; scene-only effects cannot be separated.With both features in the model, script-task coverage is +3.9pp [+0.8, +6.9], while the generic feature is +1.2pp [−4.6, +7.0].Source: §7 · p. 8
  • Coverage remains associated with success when each task-model pair gets its own fixed effect: +5.7pp per standard deviation on scene tasks and +2.8pp on script tasks.This specification compares attempts by the same model on the same task.Source: §7 · p. 9

8 Limitations and Threats to Validity

The benchmark’s findings are bounded by its engine, language, text-only observations, annotated task dependencies, and scripted play-time checks. The authors also screen check validity and assess, but cannot rule out, contamination risk.

8 Limitations and Threats to Validity

  • The benchmark covers one engine and one language, so its trajectory signal may not transfer unchanged to other engines.Coverage also depends on task-specific dependency annotations; the tested annotation-free alternatives show no significant association on script-only work.Source: §8 · p. 9
  • Agents receive text descriptions rather than rendered frames, and the study does not test vision on tasks judged by layout or visible behavior.The reported results therefore describe text-only agents.Source: §8 · p. 9
  • Validity screens dropped four tasks, while the null arm passed five scored tasks whose checks also accept an untouched world.Those five include four core tasks and one debugging task; three core tasks still distinguish models, while one does not.Source: §8 · p. 9
  • Scripted play-time checks are deterministic but far from representative of real player behavior.Source: §8 · p. 9
  • The system prompt cannot be released, and the session-execution runtime is hosted rather than open source.Source: §8 · p. 9
  • The contamination comparison finds no consistent cohort-gap change, but cannot detect contamination shared by older and newer model generations.It assumes capability gains absent contamination would affect the withheld-solution and public-solution cohorts similarly.Source: §8 · pp. 9–10

9 Conclusion

OpenGameEval is a play-time-verified benchmark for agentic software development inside a deployed game engine. Its results show difficult tasks, differing task-level strengths among leading models, and an association between task-required investigation and run success.

9 Conclusion

  • The strongest of 13 frontier models fails 48% of the core suite, the five leaders reach similar aggregate rates on different tasks, and task-required investigation predicts run success.Source: §9 · p. 10

A Framework and Benchmark Details

OpenGameEval evaluates agents in reproducible Roblox Studio sessions with separate edit-mode and play-mode checks. The appendix details benchmark construction, reproducibility and exposure analyses, and contrasts the setting with related agentic benchmarks.

A Framework and Benchmark Details

  • The framework scores static scene properties and runtime behavior separately, using task hooks for setup, reference solutions, scene checks, and game or client checks.Runtime checks can test timing-dependent server-to-client behavior that static checks cannot capture.Source: A Framework and Benchmark Details · p. 13
  • The release includes task definitions, place files, per-task annotations, and an editor plugin, but the service that runs model sessions is not open source.The engine version is pinned, checks tolerate benign non-determinism, and the plugin runs setup, reference solutions, and checks without invoking an assistant.Source: A Framework and Benchmark Details · p. 13
  • The tasks were built with expert input, executable checks, and reference solutions, but their release dates mean exposure to training corpora varies across the suite.The suite grew through dated releases, and task exposure is therefore uneven.Source: A Framework and Benchmark Details · p. 14
  • The 84 core tasks combine releases with different public exposure; the cohort comparison finds no differential improvement, with a wide interval that includes zero.The estimated mean change is −1.0pp, with a 95% Student-t interval of [−6.9, +5.0].Source: A Framework and Benchmark Details · p. 14
  • The null arm passes five of 113 tasks; four core checks already pass in the untouched world, and one of those tasks separates no models.Three of the four core tasks still distinguish agents that break an already-correct world; all 13 models pass the fourth on every attempt.Source: A Framework and Benchmark Details · p. 14

B Extended Related Work

  • Unlike AppWorld, where agents operate existing applications, OpenGameEval agents write behavior that is later exercised in the game.Terminal-Bench and GameCraft-Bench describe agent behavior, but their scores depend on outcomes rather than trajectory.Source: B Extended Related Work · p. 15

C Per-Model Results

The core-suite reliability ladder shows success falling as repeated-attempt criteria tighten, while model rankings remain similar but not identical. The appendix also details timeout handling and task-paired statistical procedures.

C Per-Model Results

  • Dropping timeouts changes pass@1 by at most 1.5pp for GPT-5.4 and 1.2pp for every other model, changing some ranks but not the paper’s claims.The top five remain the same, while the order among them changes; GPT-5.4 rises from twelfth to tenth.Source: C Per-Model Results · p. 15
  • The reliability ladder drops by ∼31 points from pass@5 to all@5, and model rankings cross as the success criterion tightens.The pass@1 and all@5 ranks have Spearman correlation 0.90; crossings also persist within model families.Source: C Per-Model Results · p. 15, Figure 3 · p. 16
  • Table 4 reports a separate suite of 29 debugging tasks, scored on its own and never averaged into the core results; §6 reports how widely the top five models spread on it.Source: C Per-Model Results · p. 15, Table 4 · p. 16

D Statistical Procedures

  • Suite-level rates use a studentized bootstrap over tasks, while paired model comparisons use percentile bootstraps on mean per-task differences and a Wilcoxon check.The Wilcoxon check agrees with the bootstrap on 75 of 78 pairs; the three disagreements have p-values between 0.03 and 0.08.Source: D Statistical Procedures · p. 17
  • Resampling by place file rather than task leaves leaderboard conclusions unchanged, with clustered standard errors averaging 1.06 times task-level errors.The 84 core tasks share 11 place files; trajectory estimates use task-level clustering.Source: D Statistical Procedures · p. 17

E Item-Response Model Specification

The item-response analysis fits a hierarchical 2PL model to repeated model-by-task outcomes, with provider families represented in model abilities. Its parameter magnitudes are sensitive to priors, while task orderings are more stable.

E Item-Response Model Specification

  • The 2PL fit models each model-task cell’s 16 attempts using model ability, task difficulty, and task discrimination, with task and model parameters partially pooled.Model abilities are nested in four provider families, and the likelihood depends on α_e(θ_m − β_e).Source: E Item-Response Model Specification · pp. 17–18
  • The model assumes every task’s success probability increases with model ability, so a weaker model cannot be easier by construction.The authors identify this monotonicity as a modeling assumption, not a finding.Source: E Item-Response Model Specification · p. 17
  • About four fifths of ability variance lies between providers: 0.792 [0.707, 0.871], and a no-family refit still estimates 0.762 [0.670, 0.844].Randomly permuting family labels reduces the estimated share to 0.247 [0.026, 0.593] (p=0.002).Source: E Item-Response Model Specification · p. 18
  • Because 54.7% of model-task cells are all-pass or all-fail, difficulty and discrimination trade off, making their magnitudes prior-sensitive.The authors therefore do not interpret difficulty magnitudes.Source: E Item-Response Model Specification · p. 18
  • With diffuse priors, task difficulty β spans [−11.1, +16.0] logits instead of [−2.2, +3.1], while median discrimination α falls from 1.96 to 0.42.The same seven tasks are selected under both fits, and task rankings are preserved with Spearman 0.999 for each parameter.Source: E Item-Response Model Specification · p. 18
  • The six tasks no model solves have weakly identified difficulty and discrimination: their parameters move substantially across priors and their α intervals are about 4.6 times wider.Under both priors, all six rank between 6th and 12th of 84 by α, despite carrying no between-model information.Source: E Item-Response Model Specification · pp. 18–19

F Item Analysis Details

The item-analysis table identifies seven core tasks whose discrimination intervals exceed the suite median under both reported and diffuse priors.

F Item Analysis Details

  • The table identifies seven tasks whose 95% discrimination intervals lie above the suite median of α = 1.96.The same seven are selected under the diffuse-hyperprior refit.Source: Table 5 · p. 19, F Item Analysis Details · p. 19
  • The selection criterion is relative to each fit’s own α scale because α’s level is not identified independently of ability.The diffuse-prior fit has α values roughly five times smaller, yet selects the same tasks.Source: Table 5 · p. 19
  • The 84 tasks span difficulty β ∈[−2.2, +3.1] on the ability logit scale.Source: Table 5 · p. 19

G Trajectory Analysis Robustness

This section examines trajectory-analysis robustness across task partitions, dependency coding, controls, and inference choices, while documenting scored tasks and check phases and describing debugging-task construction and examples; the supplied excerpts do not report robustness findings.

G Trajectory Analysis Robustness

  • The 17 annotated tasks requiring both tools are excluded from the two disjoint work arms; their coverage estimate is −2.8pp none→full with an inconclusive interval [−21.6, +15.9].Including tasks requiring execute_luau and multi_edit in the script arm would make it half scene work.Source: G Trajectory Analysis Robustness · p. 19
  • On the scene arm, dependency-credit rules yield +14.2pp when actions count, +7.8pp for any observation, and +6.4pp when observation precedes the first action on that dependency.The stricter first-action-of-any-kind rule gives +3.9pp [−0.1, +7.9] on scene tasks and +3.0pp [+0.3, +5.8] on script tasks; it is not headlined because execute_luau also enables live-world probing, interleaving legitimate observations with actions.Source: G Trajectory Analysis Robustness · pp. 19–20
  • Controlling for distinct scripts leaves the scene none→full coefficient at +13.4pp; dropping two baseplate tasks gives +13.0pp, while excluding hard tasks leaves scene and script estimates at +6.4pp and +3.3pp.The baseplate-restricted interval is [−0.9, +26.9]; the hard-task-restricted script interval is [+0.4, +6.1]. Counting timeouts as failures moves every estimate by at most 0.1pp.Source: G Trajectory Analysis Robustness · p. 20
  • No inference alternative changes any row’s verdict; the headline interval is [+0.6, +12.3] as reported and remains positive under t(23), bootstrap, and two-way clustering.The alternatives give intervals of [+0.3, +12.6] under t(23), [+1.1, +11.7] under the bootstrap, and [+0.5, +12.3] under two-way clustering.Source: G Trajectory Analysis Robustness · p. 20
  • Figure 4 shows four outcome-flip examples where newer-model runs pass and trajectories differ, while noting that similar short paths can still produce different outcomes.The figure groups tool calls into observation and action bands and cautions that trajectories provide a signal but do not fully explain every flip.Source: Figure 4 · p. 21

H The Scored Tasks

  • The scored-task appendix lists all 84 tasks with prompts, place files, and check phases; a task passes only when every defined phase passes.check_scene evaluates the edited world and check_game evaluates a play session; a task with one phase is not weaker because its objective is observable in only one setting.Source: H The Scored Tasks · p. 22, Table 6 · p. 22
  • The benchmark includes 29 debugging tasks, each derived from a core task with an introduced defect and a developer-perspective prompt; each shares its core task’s place file.The supplied table caption describes the scope and construction of these tasks.Source: Table 7 · p. 25
  • Examples include prompts about scripts failing to change character height or remove legs, including a leg-removal defect affecting players who join after the first player.These examples frame failures from a developer’s perspective, such as characters remaining their default height or retaining their legs.Source: H The Scored Tasks · p. 26

Related papers

Loading 2610.02563v1…