PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu

arXiv:2609.01481v1cs.AI

TL;DR

Autonomous software development still requires agents to manage evolving requirements, dependencies, and verification over long horizons. HoH organizes existing coding-agent harnesses into iterative planning, coding, and independent-testing loops, consistently outperforming standalone harnesses across three benchmarks and three configurations.

  • Problem

    Autonomous software development requires agents to manage evolving requirements, dependencies, and heterogeneous behaviors while preserving and verifying a complete artifact.

    Source: §1 · p. 2, §3.1 · p. 5

  • Method

    HoH organizes existing coding-agent harnesses into repeated planning, development, and independent-testing cycles that produce bounded increments and carry artifacts and evidence forward.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across three benchmarks and three harness–model configurations, HoH@3 outperforms Vanilla in every reported task category, including ProgramBench average test pass rate gains of 6.09–16.85 points.

    Source: §4.2 · pp. 10–11

  • Takeaways & Limitations

    HoH improves final artifact quality across evaluated configurations and supports continued development over more than 70 loops in a multi-day game project.

    Source: §6 · p. 16

    The reported autonomy boundary did not extend to planning, implementation, debugging, testing, or acceptance.

    Source: §5.1 · p. 15

Abstract

from arXiv · show

This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/

1 Introduction

HoH targets autonomous software development by organizing coding agents into iterative planning, implementation, and independent evaluation loops that sustain coherent progress over long trajectories. Across controlled benchmarks and multi-day game development, it consistently outperforms standalone harnesses and produces a complete, human-playable game from high-level requirements alone.

Motivation

  • Autonomous software development requires agents to sustain coherent progress across long trajectories involving planning, coordination, integration, testing, debugging, and accumulated design decisions without human intervention.Long trajectories can cause forgotten requirements, conflicting local fixes, repeated repair cycles, redundant verification, or premature completion.Source: §1 · p. 2

Method

  • HoH organizes existing coding-agent harnesses into iterative planning–coding–testing loops, where each plan addresses outstanding problems while adding a small, verifiable capability and each result receives independent evaluation.Implementation uses focused local tests, followed by complementary white-box and black-box tests covering correctness, completeness, usability, and visual or audio quality when applicable.Source: §1 · p. 2
  • HoH constrains required artifacts and evidence rather than prescribing agent workflows, while progressive disclosure, role-specific resources, reuse, and versioned histories preserve continuity and support recovery from regressions.Structured outputs trigger retries when they violate the required schema, and persisted project records can restore verified states or inform diagnosis of recurring failures.Source: §1 · pp. 2–3

Evaluation

  • After three iterations, HoH consistently outperforms standalone harnesses, gaining 16.62–22.08 points on GameCraft-Bench, 19–29 points on FrontierSWE, and 6.09–16.85 points on ProgramBench.With Codex and GPT-5.5 (high) on FrontierSWE, performance continues improving from 22% to 72.67% over ten iterations.Source: §1 · p. 3
  • HoH autonomously transforms high-level product requirements into a complete, human-playable game with a coherent storyline, core mechanics, polished visuals, and integrated audio.The multi-day deployment uses role-specific tools and skills, and commits code changes and artifacts after each agent stage to make development traceable.Source: §1 · p. 3

2 Related Work

Prior work has advanced agent harnesses and software-development agents, but existing systems provide limited continuity across revisions. HoH builds on these foundations by iteratively improving an evolving software project through repeated implementation, evaluation, and refinement.

Agent Harnesses

  • Agent harness research has evolved from model-facing prompting, memory, and action mechanisms to optimizing the harness itself, whereas HoH improves an evolving project through repeated implementation, evaluation, and refinement [15] [17] [20] [30] [45].AutoHarness synthesizes harnesses from feedback, Meta-Harness searches harness code, and Self-Harness modifies its own harness [15] [20] [45].Source: §2 · p. 4

Agentic Systems for Software Development

  • Software-development agents span repository-level issue resolution, role-based and incremental workflows, test-feedback adaptation, and benchmarks for long-horizon evolution, degradation, and from-scratch construction [18] [21] [23] [28] [29] [40].FrontierSWE additionally evaluates from-scratch implementation, open-ended performance, and research objectives.Source: §2 · p. 4
  • Existing coding harnesses typically operate within bounded episodes, offering limited preservation of project decisions, verified functionality, and evaluation evidence across revisions; HoH extends them to iterative greenfield development with continuity across planning, implementation, and evaluation cycles.HoH maintains continuity across repeated development cycles rather than treating each episode as isolated.Source: §2 · p. 4

3 Harness-of-Harness

Harness-of-Harness (HoH) repeatedly applies fixed planning, development, and independent-testing roles to bounded software increments while carrying forward both the evolving artifact and execution evidence. Its Runtime constrains role authority and verifiable outputs without prescribing agent workflows, enabling iterative repair, capability growth, and preservation of validated behavior.

3 Harness-of-Harness

  • HoH repeatedly converts retained project state into bounded, independently assessed software increments through planning, development, and testing loops.Each loop carries the candidate and execution evidence into the next cycle, preserving validated behavior while supporting incremental growth.Source: §3 · p. 4, §3.1 · p. 5, §3.2 · p. 5

3.1 Problem Formulation and Challenges

  • HoH addresses long-horizon development challenges by retaining requirements, failures, design decisions, and validated behavior across repeated three-agent loops.These loops respond to underdetermined next changes, conflicting dependencies, repair stagnation, and heterogeneous behaviors that generic testing may miss.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.4 Implementation of a HoH Loop

  • The Runtime fixes role inputs, permissions, tools, write operations, and output schemas while leaving each agent free to choose its reasoning and implementation procedure.This constrains what roles may read, change, and deliver rather than prescribing their internal workflow.Source: Figure 3 · p. 5, §3.4 · p. 7

3.2 Harness-of-Harness Overview

  • HoH assigns objective selection, artifact modification, and acceptance to a Project Planner, Developer, and QA Tester with distinct context and authority.The same fixed harness–model configuration is invoked once per role in planning–development–testing order.Source: §3.2 · p. 6, §3.4 · p. 7

3.3 Cross-Loop State Management

  • HoH carries artifact state and evidence state across loops so implementation accumulates while validated behavior, unresolved gaps, and priorities guide subsequent objectives.The artifact records what exists; evidence records verified behaviors, unsupported claims, and failures requiring further work.Source: §3.3 · p. 6

3.4.1 Project Planning

  • The Project Planner reconciles the specification, evidence, and current artifact into a bounded, locally complete objective with observable validation conditions.The resulting development document includes related tasks and preservation requirements while excluding unrelated refactoring and opportunistic expansion.Source: §3.4.1 · pp. 7–8

3.4.2 Artifact Development

  • The Developer alone modifies the warm-started artifact, using implementation-time baseline–change–retest cycles while leaving acceptance to independent QA.Self-tests support local repair but do not establish that product requirements are satisfied.Source: §3.4.2 · p. 8

3.4.3 Independent Quality Assurance

  • Independent QA evaluates a frozen, read-only candidate with scenario-specific black-box and white-box observations, recording only candidate-bound evidence as verified behavior or gaps.Freezing preserves a single candidate identity, while complementary tests assess user-visible behavior, internal conditions, regressions, and unmet requirements.Source: §3.4.3 · p. 9

4 Experiments: Benchmark Evaluation

HoH is evaluated against matched Vanilla harness–model baselines across three software-development benchmarks, showing consistent gains after three iterative planning–coding–testing loops. Additional analyses examine component-level quality, longer runs, pass-controlled comparisons, qualitative artifacts, and cross-iteration ablations.

4.1 Experimental Setup

  • The evaluation covers GameCraft-Bench, FrontierSWE, and ProgramBench using Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3.The main experiments compare HoH@3 with one-pass Vanilla under identical initial states and harness–model configurations.Source: §4.1 · p. 10

4.2 Main Results

  • HoH@3 outperforms Vanilla across all three benchmarks and all three harness–model configurations, including every reported task category.GameCraft-Bench Overall rises from 49.58 to 71.52 for Codex, 26.90 to 48.98 for OpenCode, and 42.16 to 58.78 for Pi; FrontierSWE rewards rise from 0.31 to 0.54, 0.23 to 0.31, and 0.26 to 0.55; ProgramBench Pass Rate rises from 60.41 to 66.50, 45.27 to 57.56, and 35.83 to 52.68, respectively.Source: §4.2 · pp. 10–11, Table 1 · p. 11
  • HoH improvements accumulate across iterations: GameCraft-Bench scores rise monotonically, Codex FrontierSWE Dominance reaches 71% at HoH@3, and Codex and OpenCode peak on ProgramBench at HoH@3.For OpenCode, the GameCraft-Bench gain over Vanilla grows from 1.71 points at HoH@1 to 13.42 at HoH@2 and 22.08 at HoH@3.Source: §4.2 · pp. 11–12

4.3 Analysis and Ablation Study

  • HoH@3 improves all four GameCraft-Bench quality components across every configuration, spanning mechanics, content, visuals, and presentation.Component gains range from 20.00–25.56 points for Codex, 19.25–34.63 for OpenCode, and 11.32–25.38 for Pi.Source: Figure 4 · p. 12, §4.3 · p. 12
  • At matched development-pass budgets, HoH scores 59.71, 64.84, and 71.52 versus Vanilla’s 49.58, 54.99, and 58.24.HoH@2 reaches 64.84 with 5.67M tokens, exceeding three-pass Vanilla Continuation’s 58.24 with 6.33M tokens.Source: Table 2 · p. 13, §4.3 · p. 13
  • Removing plan updates, execution evidence, or warm-start lowers Full HoH@3 scores by 8.13, 6.28, and 7.85 points, respectively.Without warm-start, token usage increases from 8.41M to 11.12M per task because the artifact is repeatedly reconstructed.Source: Table 3 · p. 14, Qualitative Analysis · p. 14

5 Experiments: Multi-Day Autonomous FPS Game Development

HoH autonomously developed Fusepoint from an empty workspace across 70 loops, sustaining capability growth, repair, and preservation as implementation constraints, validated behavior, and failures accumulated. The trajectory moved from construction through expansion to stabilization, while versioned records kept regressions and unfinished work available for subsequent planning.

5 Experiments: Multi-Day Autonomous FPS Game Development

  • Fusepoint tested whether HoH could sustain coherent project evolution over more than 70 loops while accumulating implementation constraints, validated behavior, and observed failures.The case used a single-player narrative first-person shooter developed from an empty workspace containing only a product requirements document.Source: §5 · p. 14

5.1 Case Design and Autonomy Boundary

  • The product contract required a five-minute bomb-defusal mission with ordered objectives, three-stage defusal, 18 enemies across three regions, and distinct success and detonation branches.Satisfying the contract required integrating a 3D environment, external assets, mission logic, combat, narrative progression, interface feedback, and runtime reliability.Source: §5.1 · p. 14
  • Starting only from the PRD, HoH selected the engineering decomposition, implementation order, validation plan, and increments needed to construct the executable Godot project.Human involvement was limited to restoring network or API availability and excluded planning, implementation, debugging, testing, and acceptance.Source: §5.1 · pp. 14–15

5.2 Development Dynamics Across Loops

  • Across 70 loops, HoH progressed from initial construction, through capability expansion with diagnosis and repair, to stabilization as feature additions slowed and the backlog declined.Initial construction spanned Loops 1–27, capability expansion Loops 28–49, and later stabilization followed increasing integration across mission state, combat, interface, and runtime behavior.Source: §5.2 · p. 15
  • By Loop 70, 65 of 81 recorded issues were closed, 16 remained unresolved, and 17 had been reopened after later changes broke previously verified behavior.Reopened records preserved failed behavior and verification history, making regression repair explicit project work.Source: §5.2 · pp. 15–16
  • Versioned workspaces, commit and issue histories, development documents, and testing evidence preserved implementation state, verified behavior, regressions, and unfinished work for later planning.This continuity enabled development to alternate among capability growth, repair, and preservation as project state changed.Source: §5.1 · p. 15, §5.2 · p. 16

6 Conclusion and Future Work

Harness-of-Harness extends existing coding-agent harnesses into persistent, evidence-grounded orchestration for long-horizon autonomous software development. It improves benchmark artifact quality across configurations, supports continued gains with more iterations, and motivates broader real-world applications.

Conclusion

  • HoH extends existing coding-agent harnesses without modifying their implementations, organizing fixed harness–model configurations into continuous planning–coding–testing cycles that carry artifacts and execution evidence across iterations.It structures harness participation in development analogously to how coding harnesses structure model operation.Source: §6 · p. 16
  • HoH improves final artifact quality for all three evaluated configurations on benchmark tasks and continues benefiting from additional iterations.In the multi-day Fusepoint case, it developed a game project over 70 loops with versioned workspace, issue history, and evidence packets recording the trajectory.Source: §6 · p. 16

Future Work

  • Future work will extend HoH to broader real-world scenarios, including different game types and other software systems, toward a general framework for autonomous software development [42].Source: §6 · p. 16

Supplementary Material … B.4.1 GameCraft-Bench

This supplementary section details HoH’s executable interfaces, evidence flow, evaluation isolation, experimental controls, and GameCraft-Bench construction. It also presents representative game concepts across a broad range of genres used in the benchmark.

A.1 HoH Execution Protocol

  • Each HoH iteration uses three independent invocations of one fixed harness–model configuration: Planner, Developer, and QA Tester.The Developer modifies the active workspace, while Planner and QA Tester consume materialized artifacts and return structured records; artifact and evidence cross iteration boundaries.Source: §A.1 · p. 21

A.2 Role-Specific Prompt Construction

  • Role prompts combine reusable modules with runtime task, iteration, artifact, evidence, and output-contract values while enforcing bounded priorities, preservation, observable validation, and public-information restrictions.Planner prioritizes blockers and regressions, Developer preserves verified functionality and launchability, and QA records only publicly supported claims, failures, and insufficient evidence.Source: §A.2 · pp. 22–25

A.3 Development Tools and Workspace Operations

  • HoH reuses each underlying harness’s native tools and workspace, with benchmark adapters changing artifact launch and observation without changing the planning, coding, or testing roles.GameCraft-Bench operates on Godot projects, whereas FrontierSWE operates on task repositories and their public execution environments.Source: §A.3 · pp. 25–26

A.4 Evidence Collection and Representation

  • Evidence verification relies on execution outcomes, traces, screenshots, videos, replays, and asset inspection; source-code presence alone does not establish behavioral support.QA partitions claims into verified records and gaps, retaining evidence-linked statuses and converting them into preservation constraints, update targets, and validation requirements.Source: §A.4 · pp. 26–28

A.5 Cross-Iteration State Transfer and Evaluation Isolation

  • HoH maintains separate artifact and evidence channels across iterations, while hidden tests, scores, private rubrics, formulas, and evaluator rationales remain excluded from prompts and evidence.The public task specification remains the authoritative requirement source, and iteration records are retained for auditable transitions.Source: §A.5 · p. 29

B.1 Harness and Model Configurations

  • The experiments compare Vanilla and HoH within three fixed harness–model configurations, using three HoH iterations and additional two- and three-pass Vanilla conditions.Within each configuration, models, native harness settings, public materials, and benchmark tools are held constant.Source: §B.1 · p. 30, §B.2 · p. 30

B.2 Run Configuration and Repetition

  • Each task–condition score comes from one valid run, with infrastructure or transport failures replaced rather than counted as extra replicates; aggregates average across tasks.Runs start from separate benchmark workspaces, use provider defaults for generation, and apply fixed seeds only to task sampling and statistical resampling.Source: §B.2 · pp. 30–31

B.4.1 GameCraft-Bench

  • GameCraft-Bench uses 45 tasks—three from each of 15 public game families—sampled by seeded stratification and grouped for reporting into five nine-task categories.The coarse groups are Action, Timing, Strategy, Simulation, and Adventure; task scores retain the original family definitions.Source: §B.4.1 · pp. 31–32, Table 10 · p. 32
  • The benchmark includes strategy and card-game concepts such as Tower Defense, Chess Variant, Spell Tactics, Spire Descent, Poker Roguelike, and Autobattler.Source: §B.4.1 · p. 32
  • It also covers puzzle, tycoon, idle, and simulation concepts, including Sokoban Dungeon, Space Colony, Ant Empire, Kitchen Rush, and related variants.Source: §B.4.1 · p. 32
  • Additional concepts span horror, open-world, and visual-novel games, including Floor 13, Sky Islands, Detective Noir, Arcane Academy, and Time Paradox.Source: §B.4.1 · p. 33

B.4.2 FrontierSWE … C Complete Experimental Results

The paper evaluates HoH across a structured 15-task FrontierSWE subset, controlled baselines, ablations, benchmark-specific metrics, player-experience measures, and reproducibility artifacts. The protocols distinguish iterative planning, testing, evidence, and resource accounting while preserving task and evaluation constraints.

B.4.2 FrontierSWE

  • FrontierSWE evaluation covers 15 tasks: four Implementation, nine Performance, and two Research, with 10 end-to-end construction tasks and five optimization tasks.Two official tasks are excluded because of execution requirements rather than model outcomes.Source: §B.4.2 · pp. 33–34

B.5 Baseline and Budget-Controlled Protocols

  • Vanilla performs one standard pass, whereas Vanilla Continuation adds two same-harness passes from the latest artifact without development documents or independent QA evidence.This controls the initial task set and harness–model configuration while separating repeated coding from HoH’s planning–coding–testing structure.Source: §B.5 · pp. 34–35

B.6 Ablation Protocols

  • Ablations retain HoH’s three-iteration budget and alter one cross-iteration input: plan updates, QA evidence feedback, or warm-starting from the evolving artifact.The variants respectively reuse the first plan, withhold QA evidence from planning, or restart each Developer invocation from the benchmark artifact.Source: §B.6 · p. 35

B.7 Metrics and Resource Accounting

  • FrontierSWE reports official task rewards and their unweighted mean over 15 tasks, while dominance averages same-task pairwise comparisons across Implementation, Performance, and Research.The dominance pool contains 12 system–condition configurations, and domains receive equal macro-average weight.Source: §B.7 · pp. 36–37
  • Token accounting includes provider-reported input and output tokens from coding-harness calls, excludes benchmark evaluation, and compares usage within each harness–model configuration.The budget-controlled analysis also reports quality gained per additional million tokens relative to Vanilla.Source: §B.7 · p. 36

B.8 Player-Experience Evaluation and PXI Aggregation

  • Player experience uses the full PXI with ten separately interpreted core constructs, while Enjoyment is reported separately and no global PXI total is computed.Descriptive averages over Functional and Psychosocial constructs are presented only as compact summaries, not validated higher-order scales.Source: §B.8 · p. 38

B.9 Reproducibility Artifacts

  • The accompanying anonymous code package includes HoH implementation, role prompts, the GameCraft-Bench adapter, and wrappers, but omits benchmark data, raw artifacts, secrets, and hidden evaluators.The release therefore provides core implementation and evaluation wrappers without private or benchmark-hidden materials.Source: §B.9 · p. 38

C.1 GameCraft-Bench Per-Task Scores

This section reports GameCraft-Bench performance at both reporting-group and individual-task resolution. Figure 7 summarizes Vanilla and HoH@3 group means, while Tables 14–16 provide task-level scores and changes for each harness–model configuration.

C.1 GameCraft-Bench Per-Task Scores

  • Figure 7 summarizes Vanilla and HoH@3 scores across five GameCraft-Bench reporting groups, with each bar averaging nine tasks.The scores are unweighted means within each reporting group.Source: Figure 7 · p. 39, §C.1 · p. 39
  • Tables 14–16 report four observed conditions and task-specific HoH@3-minus-Vanilla changes for every sampled GameCraft-Bench task.Scores use the benchmark’s 0–100 presentation scale and five coarse categories defined in Table 10.Source: §C.1 · p. 39, Table 14 · p. 39, Table 15 · p. 40
  • The complete task-level results are separated by harness–model configuration for Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3.Tables 14–16 preserve the benchmark’s 0–100 score scale and define Δ as HoH@3 minus Vanilla.Source: Table 14 · p. 39, Table 15 · p. 40, Table 16 · p. 41

C.2 FrontierSWE Per-Task Rewards · C.3 Budget-Controlled Comparison

The section details FrontierSWE rewards by official category and task, then compares Vanilla, continued Vanilla, and HoH under matched three-pass and token-budget conditions on GameCraft-Bench. HoH achieves the highest score and score gain per additional million tokens in the budget-controlled comparison.

C.2 FrontierSWE Per-Task Rewards

  • FrontierSWE results are summarized as mean rewards by official taxonomy category for each harness–model configuration.Figure 8 explicitly shows unequal task counts on the horizontal axis.Source: §C.2 · p. 42, Figure 8 · p. 43
  • Tables 17–19 provide every FrontierSWE task and condition for Codex + GPT-5.5 (high), OpenCode + DeepSeek-V4-Pro, and Pi + MiniMax-M3.The tables complement the category-level means with complete task-level rewards.Source: §C.2 · p. 43, Table 17 · p. 43, Table 18 · p. 43

C.3 Budget-Controlled Comparison

  • HoH uses three complete planning–coding–testing iterations in the pass-controlled Codex with GPT-5.5 (high) experiment.The comparison uses the same 45 GameCraft-Bench tasks and tracks artifact quality alongside cumulative token use across development passes.Source: §C.3 · p. 44, Figure 9 · p. 44
  • 71.52 versus 58.24 and 49.58: HoH outperforms three-pass Vanilla Continuation and Vanilla on 45 GameCraft-Bench tasks.The corresponding token use is 8.41M, 6.33M, and 2.59M tokens per task, respectively.Source: §C.3 · p. 45, Table 20 · p. 45
  • 3.77 versus 2.32 score points per additional million tokens: HoH gains more budget-normalized performance than Vanilla Continuation.These rates are computed for the three-pass conditions using Eq. 12.Source: §C.3 · p. 45
  • 49.58, 54.99, 58.24, and 71.52 are the reported mean task-level scores in the budget-controlled comparison.The supplied table excerpt reports the complete mean row for the compared conditions.Source: Table 20 · p. 45, §C.3 · p. 46
  • 2.59M, 4.56M, 6.33M, and 8.41M are the reported mean cumulative token usages across the budget-controlled conditions.These values are reported per task for the corresponding task-level comparison.Source: Table 21 · p. 46, §C.3 · p. 47

C.4 Ablation Study · C.5 Resource Usage

The ablation study shows that removing plan updates, evidence feedback, or artifact warm-start lowers HoH’s mean GameCraft-Bench score. The resource analysis reports token usage for GameCraft-Bench and FrontierSWE, with provider accounting requiring within-panel comparisons.

C.4 Ablation Study

  • 71.52 mean score for full HoH falls to 63.39 without plan updates, 65.23 without evidence feedback, and 63.67 without artifact warm-start.These variants use T=3 across all 45 GameCraft-Bench tasks with Codex and GPT-5.5 (high).Source: §C.4 · pp. 47–48, Table 22 · p. 47
  • The ablation study evaluates three cross-iteration interventions: plan updating, evidence feedback, and artifact warm-start.Figure 10 places final-score changes beside cumulative token use.Source: §C.4 · p. 47, Figure 10 · p. 47
  • Table 22 reports complete task-level GameCraft-Bench scores for full HoH and the three ablation variants on the benchmark’s 0–100 scale.The evaluation uses Codex with GPT-5.5 (high) and T=3.Source: Table 22 · p. 47, §C.4 · p. 47
  • Table 23 separately reports mean cumulative coding-harness tokens per task for the GameCraft-Bench ablation variants.Figure 10 compares cumulative token use alongside final scores.Source: §C.4 · p. 47, Figure 10 · p. 47, Table 23 · p. 48

C.5 Resource Usage

  • Figure 11 shows per-invocation token distributions across the 45 GameCraft-Bench tasks, including medians and interquartile ranges.The panels retain native provider accounting for each harness–model configuration.Source: §C.5 · p. 48, Figure 11 · p. 49
  • Per-invocation token distributions should be compared within panels rather than across panels because provider accounting differs by harness–model configuration.Figure 11 represents tasks as points and summarizes distributions with boxes.Source: §C.5 · p. 48, Figure 11 · p. 49
  • Table 24 aggregates the recorded resource use for the FrontierSWE runs.The supplied passage identifies the table’s scope but does not provide its aggregate values.Source: §C.5 · p. 49, Table 24 · p. 49

D Qualitative Analysis

The qualitative analysis compares Vanilla with HoH@1–3 across representative games from all 15 GameCraft-Bench families, using Codex with GPT-5.5 (high) and four evaluation dimensions. Figures 12–16 visualize these comparisons, while Table 25 reports the exact underlying scores for all 60 artifacts.

D Qualitative Analysis

  • Representative games from all 15 GameCraft-Bench families compare Vanilla and HoH@1–3, selecting each family’s game by the highest HoH@3 Overall score under Codex with GPT-5.5 (high).Each artifact reports Overall, Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation scores.Source: D Qualitative Analysis · p. 50
  • Figures 12–16 organize the qualitative comparisons by Action, Timing, Strategy, Simulation, and Adventure game families.The figures cover platformer, shooter, roguelike, racing, rhythm, sports, strategy, card game, puzzle, tycoon, idle, simulation, horror, open world, and visual novel games.Source: Figure 12 · p. 50, Figure 13 · p. 50, Figure 14 · p. 51
  • Table 25 reports the exact Overall and dimension scores for all 60 artifacts shown in the qualitative comparison.The table covers the 15 selected games and their Vanilla and HoH@1–3 variants.Source: Table 25 · p. 52, D Qualitative Analysis · p. 52

Related papers

Loading 2609.01481v1…