Source-linked AI summary
Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu
TL;DR
Autonomous software development requires coding agents to build complete systems from high-level requirements without human intervention. HoH organizes existing harnesses into iterative planning–coding–testing loops that preserve project state and independently evaluate bounded increments. Across three benchmarks and configurations, HoH consistently improves final artifact quality, while a multi-day case demonstrates development over more than 70 loops.
Problem
Autonomous development requires long-horizon coordination, integration, testing, and debugging, while evolving trajectories can disconnect requirements, failures, and validated behavior from later changes.
Method
HoH extends existing coding-agent harnesses with iterative planning–coding–testing loops that scope work into bounded increments, preserve artifacts and evidence, and constrain verifiable outputs.
Results
HoH consistently outperforms corresponding standalone harnesses across GameCraft-Bench, FrontierSWE, and ProgramBench for all three evaluated harness–model configurations.
Takeaways & Limitations
HoH provides a practical path toward end-to-end software development by structuring harness participation in long-horizon development and preserving the development trajectory.
Takeaways & Limitations
The autonomy boundary in the case study did not extend to planning, implementation, debugging, testing, or acceptance.
Abstract
from arXiv · showhide
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
1 Introduction
Autonomous software development asks coding agents to transform high-level requirements into complete systems without human intervention, a longer-horizon challenge than conventional agentic coding. Harness-of-Harness addresses this challenge with iterative planning, coding, testing, persistent artifacts, and constrained outputs.
- Autonomous software development requires agents to independently transform high-level requirements into complete, functional, and deployable systems.
- Long-horizon development requires translating requirements into plans, coordinating interdependent tasks, integrating components, and continuously testing and debugging the evolving system.
- HoH organizes existing coding-agent harnesses into iterative planning–coding–testing loops that balance repairing problems with delivering small, concrete capabilities.
- Independent evaluation follows implementation-time testing, with each iteration producing a bounded increment whose outputs are constrained by required schemas rather than a rigid workflow.
- More than 70 iterations produced an autonomously developed game project in the open-ended multi-day setting.
2 Related Work
Related work spans agent harnesses that shape model interaction and software-development systems that organize coding into roles, workflows, or incremental changes. HoH builds on existing harnesses while repeatedly improving an evolving software project through implementation, evaluation, and refinement.
- Agent Harnesses: Agent harnesses determine the information an LLM receives, the actions it can execute, and how execution results enter later decisions.
- Agent Harnesses: Prior work optimizes harnesses through synthesized code, harness search, or iterative self-modification, whereas HoH uses existing harnesses to improve an evolving project.
- Agentic Systems for Software Development: Software-development systems have progressed from localized code generation to repository-level issue resolution, agent–computer interfaces, general software engineering, and refactoring.
- Agentic Systems for Software Development: Role-based workflows, sprint-based development, dependent-feature organization, and test-feedback adaptation represent established ways to structure software-generation agents.
3 Harness-of-Harness
HoH treats autonomous development as repeated construction and evaluation of bounded software increments, using both the evolving artifact and accumulated evidence to guide subsequent work. Separate planning, implementation, and independent assessment preserve validated behavior while exposing unmet requirements and failures.
- Overview: HoH repeatedly organizes planning, development, and independent testing around an evolving artifact, carrying the candidate and execution evidence into the next cycle.
- Challenges: Long development trajectories risk disconnecting requirements, design decisions, failures, and validated behavior from subsequent changes.
- Bounded Development Loop: A bounded loop selects one coherent objective, implements it, evaluates the resulting artifact, and uses evaluation results to inform the next objective.
- Cross-Loop State Management: The artifact state carries the current implementation, while the evidence state records verified behaviors, unsupported claims, and failures requiring further work.
- Project Planning: The Project Planner reconciles the specification, prior evidence, and current artifact to define a bounded objective and its validation conditions without modifying the artifact.
- Independent Evaluation: Acceptance is determined by a role that did not produce the candidate, separating implementation authority from independent behavioral assessment.
4 Experiments: Benchmark Evaluation
HoH is evaluated against corresponding Vanilla harness–model configurations on three software-development benchmarks. Across configurations and benchmarks, HoH@3 improves artifact quality, with further gains across iterations and quality dimensions.
- Experimental Setup: HoH is evaluated on GameCraft-Bench, FrontierSWE, and ProgramBench using Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3.GameCraft-Bench uses 45 stratified tasks from 15 game families; Vanilla is one development pass, while HoH@T uses T planning–coding–testing iterations with artifacts and execution evidence carried forward.
- Main Results: HoH@3 outperforms Vanilla across all three benchmarks under all three harness–model configurations and every reported task category.GameCraft-Bench Overall scores rise from 49.58 to 71.52 for Codex, 26.90 to 48.98 for OpenCode, and 42.16 to 58.78 for Pi.
- Main Results: 16.62–22.08 points are the reported GameCraft-Bench absolute gains after three iterations, while FrontierSWE and ProgramBench also improve across configurations.FrontierSWE rewards increase from 0.31 to 0.54, 0.23 to 0.31, and 0.26 to 0.55; ProgramBench Avg. Test Pass Rate increases from 60.41 to 66.50, 45.27 to 57.56, and 35.83 to 52.68.
- Iterative Improvement: HoH gains accumulate across loops: GameCraft-Bench Overall scores increase monotonically from HoH@1 to HoH@3 for all three harness–model pairs.For OpenCode, the gain over Vanilla grows from 1.71 points at HoH@1 to 13.42 at HoH@2 and 22.08 at HoH@3; FrontierSWE Codex Dominance rises from 44% under Vanilla to 71% at HoH@3.
- Analysis and Ablation Study: 20.00–25.56 points are the Codex gains across GameCraft-Bench’s four quality components, spanning mechanics, content, visuals, and presentation.For Codex, Functional Visuals rises from 48.67 to 74.23, while Art and Presentation rises from 45.28 to 65.28.
- Analysis and Ablation Study: At matched one-, two-, and three-pass budgets, HoH scores 59.71, 64.84, and 71.52 versus Vanilla’s 49.58, 54.99, and 58.24.HoH@2 reaches 64.84 with 5.67M tokens, exceeding three-pass Vanilla Continuation’s 58.24 with 6.33M tokens.
5 Experiments: Multi-Day Autonomous FPS Game Development
The multi-day case study evaluates whether HoH can sustain coherent autonomous development as requirements, failures, and validated behavior accumulate across many loops. Over 70 loops, development alternated between capability growth, repair, and preservation while maintaining traceable project state.
- Task and Autonomy Boundary: Fusepoint was developed from an empty workspace into a single-player narrative first-person shooter under a five-minute bomb-defusal product contract.The contract specified ordered objectives, a three-stage defusal, 18 enemies across three regions, and distinct success and detonation branches.
- Task and Autonomy Boundary: Human involvement was limited to restoring network or API availability, excluding planning, implementation, debugging, testing, and acceptance.The run used Codex CLI with GPT-5.6-Sol at high reasoning effort.
- Project State and Traceability: 70 development loops were tracked through versioned project state, GitHub histories, issue records, and testing evidence packets.These records preserved implementation changes, observed outcomes, unfinished work, verified behavior, and regressions for subsequent planning.
- Development Trajectory: During Loops 1–27, adding core capabilities increased the active issue backlog because the increasingly testable artifact exposed missing requirements and defects.This initial construction phase established the executable project and core interaction paths.
- Development Trajectory: During Loops 28–49, capability expansion continued alongside diagnosis and repair, while later stabilization prioritized issue resolution as feature additions slowed.Later changes affected previously integrated mission state, combat, interface feedback, and runtime behavior.
- Issue Resolution: By Loop 70, 65 of 81 issues were closed and 16 remained unresolved, with 17 issues reopened after later changes broke previously verified behavior.Reopened records retained failed behavior and earlier verification history, making regression repair explicit subsequent work.
- Project State and Traceability: Versioned workspace state, commit history, issue history, and evidence packets enabled development to alternate among capability growth, repair, and preservation.Together, these artifacts carried project continuity across changing software states.
6 Conclusion and Future Work
The paper presents HoH as an extension layer that organizes existing coding-agent harnesses for long-horizon software development. Benchmark gains and the 70-loop Fusepoint case support persistent, evidence-grounded orchestration, while broader real-world validation remains future work.
- Conclusion: HoH supports software development from scratch without modifying the implementations of existing coding-agent harnesses.It carries evolving artifacts and execution evidence across a continuous planning–coding–testing cycle.
- Conclusion: Across benchmark configurations, HoH improves final artifact quality and continues to benefit from additional iterations.The paper reports this pattern across all three evaluated harness–model configurations.
- Conclusion: In the multi-day Fusepoint case, HoH developed a game project over 70 loops while versioned workspace, issue, and evidence histories recorded its trajectory.These records supported traceability across implementation and testing progress.
- Future Work: The paper identifies broader real-world software scenarios, including different game types and other software systems, as future extensions.This work is positioned as a practical path toward end-to-end autonomous software development.
A Method and Implementation Details
HoH implements autonomous development as coordinated planner, developer, and QA harness invocations around a shared artifact and evidence bundle. Role-specific prompts, bounded outputs, progressive disclosure, and workspace continuity structure iterative development without replacing the underlying harness tools.
- Execution Protocol: Each HoH iteration uses three independent invocations of the same fixed harness–model configuration: Project Planner, Developer, and QA Tester.Role-specific instructions control what each invocation may read, modify, and return while all three coordinate around the same artifact.
- Execution Protocol: The Planner turns the public specification S and prior evidence E_t−1 into development document D_t; the Developer updates artifact A_t−1 into A_t; QA produces evidence E_t.The artifact and evidence bundle cross iteration boundaries, while D_t guides coding and testing within iteration t.
- Role-Specific Prompt Construction: The role prompts are assembled from reusable Markdown modules and runtime values containing task, iteration state, materialized documents, and execution records.This preserves role contracts and data dependencies while allowing conditional modules and runtime-specific content.
- Project Planner: The Planner prioritizes blockers and regressions before extensions, selects at most three achievable priorities, and pairs each with observable validation.For later iterations, it identifies verified functionality to preserve, visible gaps to repair, and insufficient evidence.
- Developer: The Developer repairs build and runtime blockers first, preserves verified functionality, and leaves a launchable project with valid replay traces and required runtime records.It uses the underlying harness’s native file, repository, shell, build, execution, and local-testing tools.
- QA Tester: The QA Tester independently evaluates the updated artifact without modifying production code, recording evidence, player impact, ownership, failures, regressions, and recommendations.Testing includes public mouse, keyboard, and wait-event replay checks, while private tests, benchmark scores, and hidden metadata are excluded.
- QA Tester: HoH’s testing interface includes QA evidence collection and deterministic normalization of claims, cited execution records, and statuses into E_t.The benchmark adapter converts the QA report into the evidence bundle used by the main protocol.
- Workspace Continuity: Warm-starting preserves the complete project workspace, including source, configuration, project resources, and public runtime artifacts required to continue from A_t−1.HoH extends the underlying harness rather than replacing its exposed tools.
B.4.2 FrontierSWE
FrontierSWE evaluates autonomous software development across official categories and construction scopes, using task-specific rewards and controlled comparisons between Vanilla and iterative HoH conditions.
- Evaluation scope: 15 tasks are evaluated under FrontierSWE’s official taxonomy: four Implementation, nine Performance, and two Research tasks.
- Evaluation scope: The evaluated subset distinguishes 10 end-to-end tasks that build independent deliverables from five optimization tasks that improve existing systems.
- Protocol: Vanilla performs one standard development pass, whereas HoH performs three planning–coding–testing iterations while carrying artifacts and execution evidence forward.
- Protocol: Vanilla Continuation repeats coding passes from the latest artifact but supplies neither development documents nor evidence from an independent QA invocation.
- Ablations: HoH ablations retain three iterations and the same harness–model configuration while removing plan updates, evidence feedback, or artifact warm-starts.
- Metrics: FrontierSWE reports task-specific official rewards averaged over 15 tasks, while dominance compares each configuration against the other 11 configurations on the same task.
C.1 GameCraft-Bench Per-Task Scores
GameCraft-Bench reports Overall scores for sampled game tasks, grouped into five reporting categories and presented with complete task-level results for three harness–model configurations.
- Reporting groups: GameCraft-Bench summarizes Vanilla and HoH@3 across five reporting groups, each containing nine tasks.
- Reporting groups: Each group score is the unweighted mean of its nine task scores.
- Per-task results: Tables 14–16 provide complete task-level scores for Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3.
C.2 FrontierSWE Per-Task Rewards
FrontierSWE results are organized by official category and harness–model configuration, with task-level rewards and a budget-controlled GameCraft-Bench comparison of repeated development passes.
- FrontierSWE results: Figure 8 reports FrontierSWE category means under the official taxonomy and displays unequal task counts on the horizontal axis.
- FrontierSWE results: Tables 17–19 report every evaluated FrontierSWE task and condition for the three harness–model configurations.
- Budget-controlled comparison: The budget-controlled GameCraft-Bench experiment compares Vanilla, three-pass Vanilla Continuation, and HoH on the same 45 tasks using Codex with GPT-5.5.
- Budget-controlled comparison: 71.52 is HoH’s mean score using 8.41M tokens per task, compared with 58.24 for Vanilla Continuation using 6.33M tokens and 49.58 for Vanilla using 2.59M tokens.
- Budget-controlled comparison: 3.77 score points per additional million tokens is HoH’s gain, versus 2.32 for three-pass Vanilla Continuation.
C.4 Ablation Study
The ablation study tests which cross-iteration inputs matter for HoH by comparing full HoH with variants that remove plan updates, evidence feedback, or artifact warm-starts.
- Ablation results: 71.52 is the full-HoH mean score, decreasing to 63.39 without plan updates, 65.23 without evidence feedback, and 63.67 without artifact warm-starts.
- Ablation design: All ablations use T=3 on the 45 GameCraft-Bench tasks with Codex and GPT-5.5 (high), preserving the same harness–model configuration and iteration budget.
- Resource analysis: Figure 10 places ablation score changes beside cumulative token use, while Figure 11 shows per-invocation token distributions within each provider’s native accounting.
D Qualitative Analysis
The qualitative analysis compares representative games across 15 GameCraft-Bench families, tracing Vanilla against HoH@1–3 across overall and four evaluation dimensions. The artifacts illustrate how game outputs vary across genres and development stages.
- 15 GameCraft-Bench families are represented by one game selected using the highest HoH@3 Overall score under Codex with GPT-5.5.
- Each qualitative comparison presents four stages: Vanilla and HoH@1–3.
- The comparisons evaluate Overall, Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation.
- Figures 12–16 organize representative games into Action, Timing, Strategy, Simulation, and Adventure categories.
- Table 25 reports the exact Overall and dimension scores for all 60 artifacts shown in the qualitative comparison.