PaperlayerView exact revision on arXiv

Source-linked AI summary

Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization

Joey Xiao, Haonan Huang

arXiv:2609.18996v1cs.AIcs.CL

TL;DR

Prior language agents have struggled to turn game knowledge into competent play, particularly in long-horizon games. Gauntlet has coding agents build and freeze standalone controllers from a bare interface; results include held-out game wins, alongside a sharp generation gap and limits on reliability and scope.

  • Problem

    Language agents have struggled to convert game knowledge into competent play, especially in long-horizon games such as Civilization [21].

    Source: §1 · p. 1

  • Method

    Gauntlet has coding agents build standalone game controllers from a description, interface, and empty file, then evaluates frozen artifacts on held-out instances.

    Source: §2.3 · p. 3, §9 · p. 11

  • Results

    Frontier agents produce full-game wins in StarCraft II and Civilization, while the private-game results show a sharp generational separation and the Civilization result remains a novice-tier foothold.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    Compiled programs expose the architecture agents build, and the distribution of frozen artifacts—not a champion alone—tracks capability and reliability.

    Source: §9 · p. 11, §10 · p. 12

    The Civilization study covers one model family at N=3, against the novice tier on small single-landmass maps, with modest held-out conquest rates.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

LLM agents have repeatedly struggled to convert knowledge of a game into competent play, even when researchers build the agent around the model - supplying perception, memory, skill libraries, planners, or executable-policy scaffolds. Rapid progress in coding agents raises two sharper questions: can frontier models now win games at all, and can they win them unaided, building the entire player themselves? We introduce Gauntlet, a develop-freeze-evaluate framework that ports games from small arcades to full commercial-scale titles, behind one deliberately bare contract: a general-purpose coding agent receives a game description, a raw observation/action interface, and an empty policy file - no strategy, no algorithm, no architecture. In a single autonomous session the agent experiments with the live game and engineers a standalone controller; we freeze the result and score it on held-out instances with zero model calls during play. On an unpublished procedural roguelike, held-out success spans 0-86 percent and exposes a sharp generational threshold: every observed session of a newest-generation system outperforms the best session of its predecessor. At full-game scale, a compiled raw-API controller defeats every fair StarCraft II built-in AI and two cheating variants, and single-session programs win complete games of Civilization (Freeciv) by total conquest on held-out seeds. Though at modest rates against novice AI, this is a first: no prior language-agent system had won full games of this genre standalone, without per-turn model calls and a hand-crafted tactical layer. Frontier coding agents begin to track long-horizon strategy. The frozen programs are inspectable. We call this capability compiled agency: development experience compiled into a persistent executable agent whose architecture is built by the model.

1 Introduction

Gauntlet tests whether frontier coding agents can build standalone game players from a bare interface, without researcher-supplied strategy or architecture. Results include a sharp generation gap and full-game wins, while leaving reliability as an open concern.

1 Introduction

  • Prior game agents often describe plausible strategies but fail at sustained competent play, especially over long horizons such as Civilization [7] [26].Heavily scaffolded agents also remain far from competent play in many environments [7] [21].Source: §1 · p. 1
  • Gauntlet asks whether frontier models can win games and build the entire player unaided.The coding agent receives a game description, raw observation/action interface, and empty policy file, then engineers a controller in one autonomous session; the frozen program is tested on inaccessible instances with no model calls during play.Source: §1 · pp. 1–2
  • Compiled agency is development experience embodied in an executable controller whose instrumentation, state, memory, strategy, and control structure are built by the agent.The claim is operational, based on persistent action-relevant code validated on unseen instances, not unrestricted understanding.Source: §1 · p. 2
  • Held-out success spans 0–86% on the private game; every observed Fable session beats the best observed Opus session, while frozen controllers also win full games of StarCraft II and Civilization.The authors caution that Civilization performance remains modest and fresh sessions can produce very different artifacts.Source: §1 · p. 2
  • The results shift the research question from only whether scaffolding helps to how much remains necessary.The authors do not claim human scaffolding or direct model play is obsolete; they argue that structures once supplied as inputs can now emerge as coding-agent outputs.Source: §1 · p. 2
  • The paper contributes a develop–freeze–evaluate measurement protocol and a generational threshold, with systems spanning 0–86% held-out success on a contamination-resistant private game.The qualitative shift is from brittle heuristics to simulation-based closed-loop control, with every session of one newest system outperforming its predecessor’s best session.Source: §1 · p. 2

2 Gauntlet: Measuring Autonomous Policy Engineering

Gauntlet evaluates the policy-producing process as well as each frozen controller: independent development sessions produce artifacts that are tested on held-out instances. The protocol withholds intelligence architecture while supplying the interface and executable infrastructure needed to act.

2.1 The evaluated object

  • The evaluated artifact is produced from a fresh task workspace and development randomness, then mapped from a standalone policy and inaccessible test instance to an outcome.A champion artifact can demonstrate that a policy works even when the process producing it is unreliable.Source: §2.1 · p. 2
  • A single score measures one frozen controller, while independent fresh sessions measure the reliability of the coding-agent system that produces controllers.Replaying one controller on more seeds estimates that policy’s competence, not what a fresh session will build.Source: §2.1 · p. 2, §2 · p. 2

2.2 The bare contract

  • The bare contract supplies an objective, user-facing information, a strategy-free interface, executable environment, and empty policy; the agent constructs its own instrumentation, state, memory, strategy, and controller.The protocol withholds intelligence architecture, but is not infrastructure-free; supplied environments may allow inspection of undocumented mechanics.Source: Figure 1 · p. 3, §2.2 · p. 3

2.3 Sessions, freezing, evaluation

  • Fresh isolated sessions deliver a frozen final policy, which is evaluated with zero language-model calls on inaccessible held-out instances.The exam uses 80 fixed private-game seeds, 16 StarCraft II scenarios, and 12 Civilization seeds; structured observations isolate experimentation, planning, control, and software construction from visual perception and online inference latency.Source: §2.3 · p. 3

3 A Generational Threshold on an Unpublished Game

On an unpublished long-horizon roguelike, held-out performance shows a sharp generational separation, alongside a qualitative shift toward closed-loop control. Agents also recover an undocumented game mechanic, but that discovery alone does not explain leaderboard performance.

3.1 A private, long-horizon control problem

  • The unpublished roguelike requires phase-coherent play across enemy farming, upgrades, three waves, and a boss sequence; greedy and handwritten baselines each clear 0/80 held-out seeds.The task’s observation exposes object kinematics, not a strategy, dynamics model, or controller.Source: §3.1 · p. 4

3.2 The generation gap

  • 86.25% for GPT-5.6-sol and 82.5% for Fable 5 are the strongest tested private-game configurations; every observed Fable session beats the best observed Opus session.The equal-weight session-mean difference is 41.25 points, with an exact two-sided permutation p=0.0286; the top cells remain small-N and should not be over-ranked against each other.Source: §3.2 · p. 4
  • The strongest GPT and Fable policies simulate observed projectiles over multiple horizons and replan each step, unlike lower-scoring reactive or short-rollout controllers.Their closed-loop control couples receding-horizon planning to an operational value function for farming, survival, and boss damage.Source: §3.2 · p. 4
  • Reasoning effort is not a universal monotone: GPT-5.6-sol rises from 50.4% to 57.9% to 86.25% across high/xhigh/max, while GPT-5.5 peaks at high.Effort labels are provider-specific, so the reported changes are within-system dose responses rather than a common compute scale.Source: §3.2 · p. 4

3.3 Recovering undocumented dynamics

  • The game’s full-health first-boss kill triggers an approximately 70M-HP void chain, and one policy avoids it by arranging a below-maximum-health killing blow.Mechanic recovery is evidence of investigation, not a causal explanation of leaderboard rank: a simpler sibling scored 80/80 while the most mechanic-aware controller scored 51/80.Source: §3.3 · p. 4

4 Breadth, Human Anchors, and Direct Play

Across classic ports, public GameStore games, and Battle Zone, results vary by task and interface; human scores serve as contextual rather than matched anchors. On the private roguelike, independent frozen sessions span 0–86% and show a separation between observed Fable and Opus sessions.

Classic ports

  • Classic ports produced distinct task profiles: Fable scored 99.4% on the Mario-style platformer, while lunar lander separated low-end from stronger systems.On lunar lander, Haiku scored 1.9% and stronger systems scored at least 91%; with the racer interface restored, every tier scored 96–100%.Source: §4 · p. 4

Public GameStore ports

  • 1.03–4.29×: the public-game post-hoc frontier exceeded the published two-minute human median on all nine retained games, but this was not an equal-budget comparison.The frontier is not one universal controller, and some games are weak capability tests because the greedy baseline itself exceeds the study median on two games.Source: §4 · pp. 4–5

Private roguelike

  • 0–86%: held-out roguelike success spanned this range, and every observed Fable 5 session exceeded the best Opus 4.8 session.The Fable sessions cleared 51–80 seeds, versus 47 for the best Opus session.Source: Figure 2 · p. 5

Direct play

  • On Battle Zone, Gemini 3.1 Pro’s direct-play mean of 3,520 exceeded the human-study median of 2,750, with an overlapping interval [2,596, 4,447].The pixel-protocol cross-check was exploratory and limited to one game; no interface dominated for every system.Source: §4 · p. 5

5 Full-Game Scale: StarCraft II and Civilization

The same bare contract is applied to full games with contrasting control demands: real-time StarCraft II and turn-based Civilization (Freeciv). Frozen programs demonstrate substantial StarCraft II competence and some held-out Freeciv conquests, while session variability, incomplete exams, and the challenge of finishing qualify the results.

5 Full-Game Scale: StarCraft II and Civilization

  • A frozen raw-API Terran controller won 14/16 held-out Elite scenarios; Freeciv held-out conquest results were modest, with two session exams still incomplete.The Elite exam spans four maps and 16 scenarios; at the manuscript freeze, g2 had two decided wins and g3 had seven decided losses, with games still in flight.Source: §5.1 · p. 5, §5.3 · p. 7

5.1 A complete architecture from a raw API

  • The strongest StarCraft II controller compiled worker, production, technology, scouting, threat-inference, attack, and cleanup routines not specified in the prompt.Its code filters changelings, retains enemy structures after they leave vision, and triggers a race-agnostic bio attack.Source: §5.1 · p. 5

5.2 Fresh sessions and the difficulty profile

  • StarCraft II Elite results varied across fresh sessions: Fable scored 14/16, 2/16, and 12/16, while two Opus sessions scored 0/16.GPT-5.6-sol interactive xhigh sessions scored 8/16 and 3/16; a distinct automated max-effort headless session scored 11/16 and was not pooled.Source: §5.2 · p. 6, Figure 4 · p. 7
  • The selected StarCraft II champion beat every fair built-in tier and won 5/6 against Cheater:Vision and 2/6 against Cheater:Resources, but not Cheater:Insane.This post-hoc, single-map replay describes one selected artifact, not expected fresh-session capability.Source: §5.2 · p. 6, Figure 4 · p. 7

5.3 Civilization by total conquest

  • On held-out Freeciv seeds, g1 won 4/12 games; g2 won both decided games and g3 lost seven, while both latter exams were still in progress.All fifteen decided held-out defeats ended at the deadline with the agent alive; nine ended with the agent ahead on in-game score, but only total conquest counted as a win.Source: §5.3 · p. 7
  • Three Freeciv programs independently implemented eight of nine conquest-doctrine elements in different program organizations, while civil-war management converged in goal but differed in mechanism.The programs used distinct architectures; the g2 program also computed combat win probabilities and inferred hidden City Walls from failed attacks.Source: §5.3 · p. 7, Figure 5 · p. 8

5.4 The agents evaluate their own generalization

  • Across both full-scale games, finishing was the binding constraint; agents also ran self-directed multi-seed checks whose held-out generalization claims could be adjudicated against replayable artifacts.A StarCraft II session’s self-reported 9/12 wins reproduced on re-execution, while that policy scored 7/16 on the independently fixed exam.Source: §5.4 · p. 8

6 What Gets Compiled

Audited frozen policies reveal distinct control structures and doctrines, with stronger examples using simulation and replanning rather than simple reactions. These artifact analyses identify operational patterns, not causal explanations.

6.1 From reflex to simulation

  • The strongest private-game policies simulate observed projectiles across multiple horizons and replan each step, coupling control to an operational value function.GPT and Fable families independently arrive at this closed-loop design; world-model functions are distributed across state updates, danger scoring, upgrade values, and fallbacks.Source: §6.1 · pp. 8–9

6 What Gets Compiled

  • Audited policies progress from reactive rules through one-step scoring and short commitment rollouts to multi-horizon receding-horizon control.The audit reconstructs 2,432 policy versions from 88 reconstruction-valid sessions and treats its atlas as qualitative, not a frequency sample.Source: §6.1 · p. 8, §6 · p. 8

6.2 Control structure and doctrine, not volume

  • A 353-line Fable policy clears 80/80, while a 663-line sibling clears 51/80; doctrine, not code volume, separates these outcomes.The longer policy pairs a deeper safety model with risk-seeking doctrine that accumulates chip damage.Source: §6.2 · p. 9

6.3 Simulation versus lookup

  • One artillery controller hardcodes training-seed solutions and clears 13/80 held-out instances, while controllers reading wind and ballistics clear 66–79/80.The held-out split distinguishes a general control law from a memorized table; the strongest private-game policies also propagate observed bullets without seed tables.Source: §6.3 · p. 9

6.4 Convergent doctrine, distinct programs

  • Three isolated Civilization sessions independently implement eight of nine elements of a shared conquest doctrine inside structurally different programs.All three learned the civil-war splinter rule from play, but handled it differently; convergence is within one system and weaker than the cross-family convergence on receding-horizon control.Source: §6.4 · pp. 9–10

7 Reliability and Self-Termination

Fresh-session outcomes vary, intermediate policies show uneven diminishing gains, and agents choose when to stop. The paper separates frozen-policy competence from session reliability and treats artifact evidence as real while diagnosis can remain uncertain.

7 Reliability and Self-Termination

  • A champion artifact establishes that a system can compile an agent, but does not establish how often a fresh session will succeed.The paper treats independent fresh-session distributions as necessary for system-level reliability claims.Source: §7 · p. 10

7.1 Observed spread at intermediate difficulty

  • StarCraft II Fable sessions span 7–13/16 at Very Hard and 2–14/16 at Elite, while the winning Elite session was the fastest and least expensive of three self-terminating runs.The small, unbalanced sample suggests spread is largest on difficult-yet-winnable tiers; effort alone did not ensure a stronger result.Source: §7.1 · p. 10

7.2 Diminishing marginal gain within sessions

  • On private-game win-speed, session endpoints are tier-ordered: Opus 0.737 (n=10), GPT-5.5 0.592 (n=9), Sonnet 0.442 (n=4), and Haiku 0.237 (n=4).Session-averaged curves rise with diminishing marginal gain, but individual trajectories are jagged; relative time and excluded trace formats make these descriptive, not a fitted law.Source: §7.2 · p. 10

7.3 Self-termination as a marginal-gain estimate

  • All 404 sessions in the primary JavaScript corpus self-terminated; the three Civilization sessions stopped after 5.0–7.3 hours and 392–616 assistant turns.Termination is the agent’s choice and an implicit marginal-gain estimate; the paper flags calibration as future work without establishing an optimal stopping rule.Source: §7.3 · p. 10

7.4 Grounded evidence, uncertain diagnosis

  • Audits of ten extreme reported 0/80 cases found no fabricated game: three wins reproduced, two used relaxed limits, and five used self-built proxies.The paper distinguishes fabricated evidence, real but non-comparable evidence, and causal diagnosis, which may remain underdetermined.Source: §7.4 · p. 10

8 Related Work

Gauntlet measures a complementary capability to direct-play and scaffolded agents: constructing persistent controllers without the model in the action loop. Its empty-policy, held-out evaluation also differs from prior policy-synthesis and code-editing setups.

Direct and online game agents

  • Direct-play agents measure perception, online adaptation, and reasoning with the model in the action loop, whereas Gauntlet tests construction of a persistent controller that runs without it.CivRealm language baselines use per-actor abstractions and fail to make substantial progress in full Civilization games.Source: §8 · p. 10

Scaffolded game agents

  • Prior successful game agents commonly supply memory, perception, skills, curricula, planning, or online model calls; Gauntlet asks whether coding agents reconstruct action-relevant components themselves.The paper does not argue those supplied components lack value.Source: §8 · p. 10

Programmatic policy synthesis

  • Gauntlet combines an empty policy and no prescribed strategy or architecture with autonomous repository work, fresh-session replication, frozen artifacts, and inaccessible held-out instances.Unlike programmatic policy synthesis constrained by hints, DSLs, demonstrations, or outer search, it evaluates standalone held-out competence from a fresh session.Source: §8 · p. 10

Competitive code editing

  • CodeClash studies iterative submission improvement under relative tournament feedback, while Gauntlet tests absolute held-out competence from an empty policy.That construct enables seed-level generalization tests, difficulty profiles, version replay, and evidence audits.Source: §8 · pp. 10–11

Agent evaluation and reliability

  • Gauntlet adds a deterministic, renewable domain whose product is an executable agent to broader work on deployed-system evaluation and reliability.Its emphasis on fresh-run variation, baselines, and artifact audits follows broader arguments about scaffolds, costs, randomness, and statistical units.Source: §8 · p. 11

9 Discussion

The discussion frames architecture construction as an emerging output of coding agents, while emphasizing modest Civilization results and complementary measurement from direct play and policy coding. It also identifies substantial scope and evaluation boundaries.

9 Discussion

  • A recent system generation constructs operational state, experimentation, strategy, and control machinery from a game description, interface, and empty file, including in a second full-scale game.The discussion shifts the question toward whether agents reconstruct withheld components, how reliably, and whether results generalize.Source: §9 · p. 11
  • Civilization yields engine-verified, single-session full-game wins, but the evidence is from one model family at modest held-out rates against the novice tier.The paper describes this as directional progress, not mastery, and notes finishing under a deadline as a failure mode.Source: §9 · p. 11
  • Direct play measures perception, one-shot adaptation, and online reasoning, while policy coding measures autonomous experimentation, software construction, and persistent control with the model removed.Readable artifacts support mechanistic hypotheses but not automatic causal identification because world model, strategy, and execution entangle.Source: §9 · p. 11
  • The Civilization study uses one model family at N=3, the novice tier, small single-landmass maps, and a 12-seed held-out exam with one non-deterministic run per seed.Other boundaries include a selected-policy, single-map StarCraft II profile and artifact taxonomies involving judgment.Source: §9 · pp. 11–12

10 Conclusion

The paper concludes that one-session frontier coding agents can compile complete game-playing programs, including StarCraft II opponents and Freeciv total-conquest runs. It frames this as a foothold and leaves the capability ceiling unmeasured.

10 Conclusion

  • Frontier coding agents compile complete game-playing programs that defeat every fair built-in StarCraft II opponent and carry Freeciv civilizations from a first settler to engine-declared total conquest.The programs are built from a game description, interface, executable environment, and one autonomous session.Source: §10 · p. 12
  • The frozen artifacts expose live simulation, persistent state, calibrated doctrine, and finishing logic, while their distribution—not a single champion—tracks the frontier's movement.The conclusion frames game evaluation as studying models that engineer agents rather than operate them.Source: §10 · p. 12
  • The capability is not yet reliable, and Civilization results remain a foothold at the novice tier; the paper leaves the ceiling unmeasured.Open questions include performance beyond novice opponents, multiple rivals, other genres, and better calibration of stopping and generalization.Source: §10 · p. 12
Loading 2609.18996v1…