HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

1 Introduction

• HarnessDev evaluates two stages: creating a complete harness from a weak runnable seed and development cases, then iteratively improving that persistent harness using downstream execution feedback.The benchmark evaluates runnable infrastructure that persists across tasks rather than one-off task outputs.Source: Figure 1 · p. 2, §1 · p. 2
• Unlike typical evaluations that treat the harness as experimental setup, HarnessDev measures whether models can develop persistent execution systems, a task that requires diagnosing behavioral and structural limits and making reusable improvements.The introduction identifies this as an underexplored capability and calls for separating harness developers from downstream executors while recording the development environment.Source: §1 · p. 2
• HarnessDev measures capability on held-out downstream tasks and efficiency by the executor-model tokens consumed by a frozen harness.The evaluation accounts for risks including overfitting to the creator, development examples, or feedback set, and regressions in other capabilities.Source: §1 · p. 2
• Model-built harnesses match references on short-form writing, exceed them on machine-learning experimentation, and lag most on search and research and substantially on code.Harnesses from different creator models also vary in task performance and executor-token cost; higher cost does not reliably yield better results.Source: Harness Creation · p. 3

Harness Creation.

• Evolution gains often fluctuate across revisions, shrink and become less consistent on unseen tasks, and depend strongly on the model running the harness.Changing the runtime model alters both starting performance and whether subsequent revisions help.Source: Harness Evolution · p. 3

Background

• A capable model is not yet a working system; forward-deployed engineers adapt it to customer data, workflows, and compliance constraints, then fold failures into fixes and feature requests.The growth of FDE hiring across frontier-model and data-platform companies reflects the continuing role of human engineers in closing this gap.Source: §2 · p. 3
• FDE work entails translating vague intent into objectives, constructing reliable feedback, and building and maintaining the tools, context, state, lifecycle logic, and verification needed for execution.These structures are normally presupposed by benchmark designers; they must be built and adapted as requirements shift.Source: §2 · p. 3
• HarnessDev asks whether language models can create an execution scaffold from a weak starting point and improve it through feedback while preserving constraint compliance and held-out performance.Unlike typical controlled evaluations, it treats the harness as the object of development; Aspire studies deployment-driven capability growth, while S3Gym studies judging and reusing interaction experience.Source: §2 · p. 3

3.1 Overview

• HarnessDev freezes each creator-built harness for reuse across downstream tasks, separating the creator that builds it from the executor that runs it.The harness includes the execution loop, tools, context management, persistent state, lifecycle control, and verification.Source: §3.1 · p. 4

3.2 Development settings

• Creation builds a harness from a weak runnable seed, while Evolution revises that harness using downstream execution feedback.Creation aims to generalize from a task-family specification and a few development cases; Evolution also examines edit trajectories, held-out generalization, and transfer across executors.Source: §3.2 · pp. 4–5, §3.1 · p. 4, Table 1 · p. 4
• Evolution starts from the creator’s frozen Creation harness and uses paired SWE-Pro and Terminal-Bench evaluations as feedback.The protocol allows ten post-H0 full-evaluation pairs and at most two fixed-subset probes; all official versions are also evaluated on 630 disjoint SWE-Pro instances withheld from the creator.Source: §3.2 · pp. 5–6

3.3 Domains and downstream benchmarks

• Creation spans four domains and five downstream benchmarks, while Evolution focuses on code harnesses; together, the suites contain 2,207 unique instances.Evolution feedback tasks come from the same suites and are not counted again.Source: §3.3 · p. 6, Table 2 · p. 6

3.4 Evaluation protocol

• Frozen harnesses are evaluated with fixed executors and scorers, while hidden scores remain unavailable during development.Creation compares generated harnesses with the common seed and, where available, mature references; Self-Eval uses the creator as executor, whereas Unified-Eval uses a shared fixed executor.Source: §3.4 · p. 6
• Harness constraints prohibit instance-specific solutions and access to hidden evaluation information, and the audit found no prohibited scoring routes.Scoring uses repository diffs or final environment state rather than harness self-reports; runs retain execution artifacts for post-hoc audit.Source: §3.4 · pp. 6–7

3.5 Metrics

• The benchmark reports downstream task performance and executor-model token cost for each frozen harness.Execution cost includes total and mean tokens per task, excludes creator tokens, and Evolution reports performance change from its initial harness under the same executor and scorer.Source: §3.5 · p. 7

4.1 Experimental setup

• The experiment evaluates six creator LLMs, averaging three independent harnesses per creator–benchmark pair and comparing self-execution with a fixed Gemini executor.Human-reference results are verified public system results, not paired controls under one executor.Source: §4.1 · p. 7

4.2 Harness Creation

• Creation artifacts differ in implementation strategy, and more code does not predict better performance: Gemini adds 1,006 lines yet scores 68.8 on Terminal-Bench.Across 18 Code harnesses, execution loops are explicit, but only 13/18 complete tool and lifecycle control; state persistence is rare, and 441 of 2,325 Data tasks yield undetected degenerate submissions.Source: §4.2 · p. 8, Figure 5 · p. 10
• Under Self-Eval, Opus 4.8 leads Creation at 67.8 yet trails the human-engineered reference (86.2); Writing nears the reference, while Search and Code lag.Opus 4.8 and Gemini 3.1 Pro lead MLE-bench with medal rates of 32.9 and 32.4; 77.8% of failed Data tasks are attributed to harness defects.Source: §4.2 · p. 7, Figure 4 · p. 9
• Many implemented mechanisms do not affect execution, and self-test count alone is a weak signal of downstream score.In Code, 18 of 108 component instances are never observed; self-test correlation is only 0.13–0.26 and nonsignificant, whereas revision calls reach 0.57 (p ≤.0005). Testing helps when failures prompt targeted changes that are re-verified.Source: §4.2 · p. 9
• Changing the executor substantially changes harness rankings: Qwen, Seed, and DeepSeek improve in some fixed-Gemini settings, while Opus and GPT-5.5 are often stronger with their own executors.Opus’s Self-Eval SWE-Pro score falls from 69.3 to 33.0 under Gemini, and its Search harness duplicate-query rate rises from 10.1% to 88.2%; MLE-bench token use varies about nineteen-fold without reliably higher scores at higher cost.Source: §4.2 · pp. 7, 8, 10, Figure 6 · p. 11

4.3 Harness Evolution

• All five self-runtime Evolution creators improve on visible feedback, but held-out gains shrink; Opus 4.8 has the largest held-out improvement at +4.44 points.With a fixed Gemini executor, only Opus improves on held-out tasks while the other three lineages regress.Source: §4.3 · p. 11, Table 6 · p. 12

Evaluation protocol.

• Evolution is nonmonotonic, and visible feedback is unreliable for selecting the best held-out version: only 2/9 declarations are held-out optimal.Only two of 64 comparable switches show clear gains beyond the repeated-run noise band; feedback and held-out scores move in the same direction 34 times (53.1%), and the same commit can vary by about ±4.75 pair-score points.Source: Table 6 · p. 12, Evaluation protocol · p. 13
• Evolution mostly changes execution and control flow, while creators often diagnose failures with limited evidence rather than the dedicated trajectory interface.Across 64 official switches, 58 change execution or control flow and only four change state; one GPT-5.5 candidate passes all five Terminal probes but scores 0.584 on the full set. Opus’s completion check addresses a concrete mismatch: 99 of 100 runs report success, but only 48 pass.Source: Evaluation protocol · pp. 12–13

Agent benchmarks and harness development

• Unlike task benchmarks and Harness-Bench [55], which assess execution under a selected harness or the effect of harness choice, HarnessDev also evaluates continued development, creator–executor separation, and execution cost.The Meta-Agent Challenge [27] is closest to Creation, evaluating sandboxed agent development on protected tests across five domains.Source: §5 · p. 14
• HarnessOpt-Bench [47] focuses on optimizing a provided harness, whereas HarnessDev connects Creation to Evolution and tests every frozen version on disjoint 630-task SWE-Pro held-out data.HarnessDev uses in-trajectory scores as feedback and evaluates the same frozen artifact under self and fixed runtime models, separating adaptation from held-out generalization.Source: §5 · p. 14
• Unlike Evo-Bench [16], which scores final revisions to a shared CodeAct seed under a fixed runtime model, HarnessDev evaluates trajectories, execution-token cost, and transfer across runtime models.Evo-Bench uses a disjoint, sensitivity-calibrated multidomain suite; HarnessDev also connects Creation and Evolution and scores every frozen version on its held-out split.Source: §5 · p. 15

Automated agent design and evolution

• Automated agent design spans searches over prompts, workflows, operators, and topologies [57] [51], as well as systems that construct more of an agent in natural language or code [22] [31] [39].Human-engineered systems including Claude Code, Codex, OpenHands [49], and SWE-agent combine execution loops, tools, context management, and recovery mechanisms [40].Source: §5 · p. 15
• Harness-improvement approaches include executable-code optimizers such as VeRO [46] and Meta-Harness [21], plus methods for repair, regression testing, demonstrations, and validated patches [56] [9] [8] [41].HarnessDev evaluates general-purpose frontier models in a shared Creation-and-Evolution protocol rather than proposing another improvement method.Source: §5 · p. 15

Evaluating harness evolution

• Prior work motivates separating diagnosis from search [17], update usefulness from executor benefit [24], and intermediate held-out performance from final gain [59]; HarnessDev evaluates trajectories, cost, and runtime-model transfer.Matched-budget studies report possible overfitting and no advantage over simpler search baselines [50]; matched-search evaluation remains future work for HarnessDev.Source: §5 · p. 15

6 Discussion, Limitations, and Conclusion

• Creation varies sharply by domain, while Evolution gains can be erased by later updates and depend strongly on the runtime model executing the harness.Under Self-Eval, current models match the human reference in writing and exceed it in machine-learning experimentation, but lag in search and research and still trail in code; changing only the runtime binding can substantially move H0 and alter which changes help.Source: §6 · p. 15

6.1 Limitations

• The evaluation has uneven human baselines, incomplete benchmark coverage, one trajectory per creator–runtime cell, and post-freeze held-out evaluation only on SWE-Pro.Unified-Eval reduces but cannot fully remove executor-model differences; the fixed development environment and model-external focus leave further evolution and replacement of parameter training unaddressed.Source: §6.1 · pp. 15–16

6.2 Conclusion

• HarnessDev makes creating and maintaining execution systems for future tasks a measurable target through human baselines, creation, feedback-driven evolution, and Self- and Unified-Eval.It compares agent-built harnesses with human-engineered references and treats harnesses as explicit, inspectable, testable, reusable systems that can be improved through feedback and engineering pressure.Source: §6.2 · p. 16
• The benchmark audits harness compliance after every run and reports no violations, but its reproducibility containers are not designed for containment.Generated harnesses should be treated as untrusted code and isolated more strictly when reused; audit artifacts are released for repeatable checks.Source: §6.2 · p. 16

A Candidate Harness Systems

• The pinned candidate set spans code, notebook/data-analysis, writing, and research/retrieval systems, with role assignments, commit hashes, and licenses recorded where applicable.Candidates include Claude Code, OpenCode, OpenHands, SWE-agent, and mini-SWE-agent; DataAgent and DB-GPT; AutoResearchClaw and webnovel-writer; and Alibaba-NLP/DeepResearch, dzhng/deep-research, modelscope/ms-agent, and gptresearcher.Source: A Candidate Harness Systems · p. 22

B Experimental Configuration

• Table 8 records creator LLMs, development environments, and decoding configurations; reported creators ran at high reasoning effort with streaming enabled and endpoint-maximum output length.Sampling followed provider defaults, and participation was stage-specific, so table presence does not imply coverage of every benchmark.Source: B Experimental Configuration · p. 22, Table 8 · p. 22

B.1 Downstream execution configuration

• MLE-bench downstream tasks ran in isolated containers with an A800 GPU, 14 vCPUs, 227 GiB RAM, a 500-step cap, and a 36,000-second limit.The harness received 34,200 seconds and the fixed grader 1,800 seconds; dataset preparation occurred before the task clock. RQ2 code benchmarks used the same step cap and a 7,200-second limit.Source: §B.1 · p. 22

B.2 Human-engineered reference systems

• Three starred Table 3 values are external reports from OpenAI’s GPT-5.6 release report: SWE-Pro 80.0 for Claude Fable 5, Terminal-Bench 2.1 88.8, and BrowseComp 92.2 for GPT-5.6 Sol.These values were not local reruns.Source: §B.2 · p. 23
• Human-engineered references use the highest publicly available system-level result verified from an official leaderboard or system report, pairing each harness with its executor rather than using one common executor.Table 9 records the exact harness–executor pairs used for the main-paper references.Source: Table 9 · p. 23

B.3 Creation performance and execution cost

• On MLE-bench, GPT-5.5 reaches a medal rate of 19.1 with 29.3M tokens, while DeepSeek V4 reaches 19.6 with 208.4M tokens.Figure 9 reports the performance–token trade-off directly rather than collapsing it into a cost-adjusted score.Source: Figure 9 · p. 23

B.4 Evolution execution cost by frozen version

• Figure 10 reports task-agent/runtime total_tokens for each frozen harness and benchmark leg, excluding creator, judge, and diagnostic-probe tokens.It covers self runtime and fixed Gemini runtime across SWE-Pro-100, Terminal-Bench 2.1, and the post-freeze SWE-Pro held-out-630 split.Source: Figure 10 · p. 24, §B.4 · p. 24

C Harness Interface Specification

• The six-module contract exposes execution run/step, tool registration/calls, and context build/compression interfaces.These are execution.py, tools.py, and context.py, respectively.Source: C Harness Interface Specification · p. 24
• State save/load, lifecycle hooks for actions and failures, evaluation, and trajectory recording complete the harness interface.The lifecycle also provides timeout handling; all methods return JSON-serializable objects.Source: C Harness Interface Specification · p. 24
• The reference seed implementation, audit script, and held-out task splits will be released with the benchmark.Source: C Harness Interface Specification · pp. 24–25

C.1 Seed Harness Skeleton

• The shared weak seed provides a stable CLI, runtime model configuration, audit writers, and policy-free primitives, with packaging adapted to the execution environment.Figures 2 and 3 distinguish the domain-general structure from domain-specific final artifacts.Source: §C.1 · p. 25
• The seed package supplies a runnable entry point and passive helpers for file, search, process, and artifact operations, but no policy paths.Source: §C.1 · p. 25
• The unmodified seed only parses inputs, optionally probes the LLM with a summary prompt, and exits partial; it lacks execution, context, state, recovery, and verification modules.Creators must implement these modules using public development-task feedback while hidden tasks, answers, and official scores remain withheld.Source: §C.1 · p. 25
• The authoritative artifact varies by domain, and the unmodified seed scores zero across all five downstream benchmarks.Required outputs include a code patch, scorer-readable analysis submission, final prose, or evidence-grounded research/search answer; common JSON and logs alone are incomplete.Source: §C.1 · p. 25

• The appendix restates the evaluation-setting and model-role definitions from Section 3.4.Source: D Evaluation Settings and Roles in Full · p. 25

D.1 Self and unified evaluation

• Unified-Eval holds the executor model fixed to compare harnesses, while Self-Eval pairs each harness with its creator to measure model–harness co-design.A high Unified-Eval score indicates transferability; human-engineered systems are external references, not common-executor controls or an absolute ceiling.Source: §D.1 · p. 25

D.2 Model roles

• The evaluation separates harness creation, development support, downstream execution, and scoring so support or executor ability is not attributed to harness quality.The experiments record development support for each creator configuration and hold the executor and evaluator fixed within comparisons.Source: §D.2 · p. 26

E Representative System Prompts

• The representative prompts cover shared RQ1 creation instructions and an RQ2 V6 evolution prompt, with only run-specific paths and evaluation identifiers replaced.The appendix omits task-specific prompts and auxiliary workspace documents to illustrate system-level contracts.Source: E Representative System Prompts · p. 26

E.1 RQ1: shared harness-creation system prompt

• The RQ1 contract requires executable harness code, not a design description, and a weak seed that lacks mature runtime capabilities.The harness must implement execution responsibilities such as tools, context, state, lifecycle, verification, and artifact construction; downstream task behavior determines the official score.Source: §E.1 · pp. 26–27

E.2 RQ2: harness-evolution system prompt

• The RQ2 contract directs the creator to improve an existing runnable harness so the harness-plus-runtime-model agent performs as well as possible on downstream benchmarks.The deliverable must be executable system code rather than an architecture description, README, or plan.Source: §E.2 · p. 29
• Evolution allowed 10 charged full-evaluation pairs, while each probe covered the same first five tasks per benchmark and provided directional rather than official scores.The controller-submitted H0 baseline pair was exempt; official version scores required completed full evaluations.Source: §E.2 · pp. 29–30
• Correctness came only from the fixed verifier’s score, while diagnostic and artifact signals helped locate problems; downstream benchmark performance alone determined official scores.The controller left choices about changes, validation, probes, submissions, and final-version selection to the agent, within physical limits.Source: §E.2 · pp. 30–31

F RQ1 Code: GPT-5.5

• The code trace records seed reads before the first edit and observed editing operations, but these excerpts provide no counts.Source: F RQ1 Code: GPT-5.5 · p. 31

Complete self-test ledger

• Only three of six official runs had benchmark scores; the other three were interrupted diagnostics, alongside five local harness executions.The official sequence included contract smoke, three interrupted diagnostics, then settled SWE and Terminal runs.Source: Complete self-test ledger · pp. 31–32

01 Confirm the delivery boundary before judging the seed

• The seed could call tools but could not genuinely solve tasks, and the packaged harness therefore needed a self-contained agent loop.The creator reviewed the build contract, CLI forms, required outputs, public development commands, code-benchmark documentation, and weak seed; only harness/ and runtime_llm/ would be packaged.Source: §01 · p. 32

02 Replace the one-shot runner with a JSON-action tool loop

• The new harness routes runtime-model JSON actions through read, write, search, shell, and patch tools, then verifies and emits benchmark-readable artifacts.A centralized agent module handles parsing, dispatch, state, verification, and finalization, making runtime-output reliability logic easier to repair from trajectories.Source: §02 · p. 32
• The implementation switched its import from `.seed_runner.run` to `.agent.run`, with the final `harness/agent.py` file containing 1,245 lines; overall, 12 paths changed, adding 1,251 lines and removing 4.Source: §02 · p. 32

03 Local smoke A executed the model but misreported the artifact

• The initial smoke made one LLM call and created smoke_out.txt, but reported empty patch.diff and changed_files because artifact accounting failed.The repair narrowed the initial environment summary, added artifact_paths, and registered files created by write, replacement, and patch tools independently of git diff.Source: §03 · p. 32

04 Local smoke B confirmed the accounting repair

• The repaired run returned status=success with result.txt containing exactly OK and artifact_paths=[result.txt], despite running outside a git repository.It made two LLM calls, two tool calls, and one file write; the repair addressed artifact bookkeeping rather than task execution.Source: §04 · p. 32

05 Validate the execution contract before spending a real benchmark budget; 08 The first Terminal attempt reached real compilation but did not settle

• The first official smoke verified the CLI, runtime model, batched actions, required files, and trajectory, establishing execution health rather than difficult-task capability.A later Terminal diagnostic created and compiled /app/gpt2.c but ended without a complete summary or verifier result, so it remained pending rather than scored.Source: §05 · p. 33, §08 · p. 33

06 SWE diagnostic A exposed multi-object JSON loss; 07 SWE diagnostic B exposed the shell mismatch

• SWE diagnostics exposed multi-object JSON loss and a /bin/sh versus Bash mismatch, prompting parser recovery and explicit Bash selection, but active containers did not load those fixes.The first parser rejected turns containing multiple complete JSON actions; the shell fix likewise was not hot-loaded into the incomplete diagnostic run.Source: §06 · p. 33, §07 · p. 33

09 End attempt 1 with a clean local contract check; 10 Resume the same workspace and verify retained behavior; 12 Verify the task-json CLI form and every required output artifact

• Local contract checks succeeded across alternate CLI forms and same-workspace continuation, while the three incomplete benchmark attempts remained unresolved.The checks created check.txt and out.txt containing OK, returned success, and verified artifact existence; later task-json validation produced ok.txt containing DONE and the required output files.Source: §09 · p. 33, §10 · p. 34, §12 · p. 37

11 Completed SWE revealed duplicate actions and a Codex-style patch envelope

• The completed SWE run scored 0.0, while its live trajectory prompted safe deduplication of identical read-like actions and support for Codex-style patch envelopes.Writes, patches, and finish actions were not deduplicated; the score validated the launch-time snapshot because the edits were made during the run.Source: §11 · p. 34

13 Validate the final snapshot locally, then stop; 12 Terminal passed, while live feedback prompted one final dedupe extension

• Contract smoke=1, SWE=0, and Terminal=1 were the final official scores; separate local checks verified the final snapshot.The Terminal 1.0 run validated the preceding snapshot because the command-deduplication extension was added after launch; the final local smoke passed an artifact check.Source: §13 · p. 34, §12 · p. 34

G RQ1 Code: Opus-4.8; 01 Read the seed, runtime client, and platform contract before rewriting; 02 Define module boundaries, then rewrite the execution system nearly all at once; 03 Encode budgets, context control, and honest termination upfront

• The creator replaced the seed’s limited tools with a native tool-calling ReAct harness, modular execution and verification, and explicit budgets and honest termination.The design used a single-threaded persistent-message loop, up to 120 steps, a 6,600-second budget, and a 150,000-token soft context limit.Source: §01 · p. 35, §02 · p. 35, §03 · p. 35

04 The first local invocation failed before the harness started; 05 The corrected terminal-path test ran end to end; 06 A local repository task exercised edit, verification, and patch generation; 07 The official contract smoke passed without prompting a code change; 08 Parallel submission created a separate infrastructure-invalid Terminal child; 09 SWE completed real work but failed hidden tests; 10 Serial Terminal produced the target artifact but still scored zero

• The local terminal and repository tests succeeded, and the official contract smoke passed, but the SWE and Terminal benchmark attempts scored zero.The first local invocation failed before harness startup because the package path was missing; a later parallel Terminal child was infrastructure-incomplete, so it was rerun serially.Source: §04 · p. 35, §06 · p. 36, §07 · p. 36

11 Remove unused seed remnants and immediately rerun the terminal path; 12 Verify the task-json CLI form and every required output artifact

• After pruning unused seed remnants, the harness still passed a local terminal smoke and the task-json CLI contract with recorded output artifacts.The post-prune smoke created out.txt containing 42 and returned success; the final task-json invocation also generated changed_files.json, patch.diff, response.md, result.json, stderr.log, stdout.log, and trajectory.jsonl.Source: §11 · p. 37, §12 · p. 37

13 Close with verified interfaces but no benchmark-driven harness revision

• Opus checked three CLI forms, target-file contents, seven artifact classes, imports, and package listing; its 1,876-line harness ended with smoke=1, SWE=0, and Terminal=0.The run demonstrated broad up-front construction and execution coverage, but neither settled zero prompted a core harness edit.Source: §13 · p. 37

01 Rewrite the architecture before complete H0 was available

• The round was architecture-first rather than feedback-driven, and H0 was later confirmed at SWE 51/100 and Terminal 60/89.The round added artifact-path discovery, JSON-action recovery, repeated-action unlocking, and a manifest entry point that prefers python3.Source: §01 · pp. 37–38

02 Replace weak completion checks with a final-review gate; 03 Extend final review to every successful Terminal task; 04 Remove verification-gap enforcement after probe regression

• After both T1 benchmarks regressed, a gate blocked immediate success under low evidence or failed verification, later covered all Terminal successes, and after a fixed probe fell from 4/5 to 3/5, removed gap enforcement but retained path filtering.Source: §02 · p. 38, §03 · p. 38, §04 · p. 38

05 Track files created by shell commands; 06 Preserve output head and tail, then review asynchronous cancellation

• Shell commands modified files without write_file, so artifact tracking missed real changes; the iteration added before/after run_command snapshots, opened no new SWE failure case, and focused on Terminal artifact handling.Source: §05 · p. 39

07 Try and revert large-file offsets, then constrain finish actions; 08 Select T2 instead of the latest version

• A probe for suspected large-file offset errors returned Terminal 1.0 but SWE 0.6, prompting offset-reading reversion and standalone explicit finish actions; high probe scores did not transfer to full evaluation.Source: §07 · p. 39

H RQ2 Code: GPT-5.5; 08 Select T2 instead of the latest version

• T2 was selected as final, scoring 56.0 on SWE and 74.157 on Terminal; three remaining pairs went unused, and the creator never repaired 54% tool-call rejection.The creator maintained a combined-score ledger and ended evolution rather than submitting another candidate after T7.Source: H RQ2 Code: GPT-5.5 · p. 37, §08 · pp. 39–40

I RQ2 Code: Opus-4.8; 01 Build failure categories from 189 cases before a structural repair

• The creator read cases.json, the feedback index, and raw results to confirm H0=68/100, then classified about 10 of 32 SWE failures as hidden-grader build, compile, or import errors; proposed changes added pre-finish verification and finalize safeguards.Thinking-block replay and prompt caching were checked first, but latency was rejected as the main lever; the hypothesis was that agent self-tests do not imply hidden-grader interface correctness.Source: §01 · p. 40

02 Give the model the real git diff for one self-review

• After T1 improved, the creator inspected 728 edits and found only 14 str_replace_- no_match events (1.8%), rejected the stronger-editor route, and added _current_diff() with mandatory diff-grounded self-review before repository tasks ended.The probe was explicitly treated as a noisy n=5 crash check.Source: §02 · p. 40

03 Extend reflection to non-git and Terminal tasks

• Reflection was extended to non-git and Terminal tasks by reviewing recorded write paths, command state, and recent working-directory files.The implementation added _written_paths, _ran_command, and _recent_workdir_files() for artifact self-review.Source: §03 · p. 41
• Because many Terminal artifacts were created through shell commands, git-diff reflection did not run for them.The earlier reflection applied only to repository tasks with a git diff.Source: §03 · p. 41
• All 25 T2 SWE failures had already triggered reflection, while R3’s interface-mismatch inspection still focused on general reflection.Source: §03 · p. 41

04 Attempt process-group termination, then revert it

• No formal T4 or full evaluation occurred; the final commit changed only LEDGER.md, with no H4 code commit.The reported final commit was 5faaa15.Source: §04 · p. 41
• Cross-version union/intersection/never-pass counts were SWE 81/65/19 and Terminal 71/59/18, while reflection prompted further edits about 4% of the time.Source: §04 · p. 41
• An attempted setsid/killpg change for Bash timeouts was reverted as too risky for 189 tasks over one edge case.Two concurrent sessions made opposite edits to tools.py.Source: §04 · p. 41

05 Stop sampling and select T3

• The creator estimated roughly ±3–4 tasks of noise per full evaluation and declined to spend seven remaining pairs chasing a random peak.It judged another trivial commit to be additional sampling rather than meaningful evolution.Source: §05 · p. 41
• T3 was selected as the balanced, validated, mechanism-complete tie-break instead of adding candidates or repeating samples.The decision used three of ten available pairs.Source: §05 · p. 41
• SWE 74.0 and Terminal 74.157 for selected T3; T1 was higher by about 0.06 pp, but T1–T3 passed the same total number of tasks.The final decision selected T3.Source: §05 · p. 41