PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, Yu-Gang Jiang

arXiv:2604.25850v4cs.CLcs.SE

TL;DR

Coding-agent harness engineering remains manual because full-harness edits are difficult to specify, trajectory evidence is voluminous, and edit effects are hard to attribute. AHE uses component, experience, and decision observability to create a closed-loop evolution process with falsifiable edits. Ten iterations raise Terminal-Bench 2 pass@1 from 69.7% to 77.0%, and the frozen harness transfers across tasks and model families.

  • Problem

    Jointly evolving all editable harness components remains open because long trajectories yield little actionable signal and tightly coupled frameworks make edits error-prone.

  • Method

    AHE exposes harness components as files, distills rollouts into layered evidence, and records each edit with a prediction verified against next-round outcomes.

  • Results

    Ten iterations raise pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing human-designed and self-evolving baselines, while the frozen harness transfers across benchmarks and model families.

  • Takeaways & Limitations

    AHE presents harness-level evolution as an externalized, auditable adaptation surface complementary to model-side training.

  • Takeaways & Limitations

    Evaluation covers Terminal-Bench 2 evolution and transfer to SWE-bench-verified and three alternate model families, leaving broader languages, repository-scale deployments, and human-in-the-loop workflows untested.

Abstract

from arXiv · show

Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term memory rather than the system prompt, suggesting factual harness structure transfers while prose-level strategy does not.

1 Introduction

Harness engineering strongly affects coding-agent performance, but jointly evolving prompts, tools, middleware, and other components remains difficult because trajectories are unwieldy and edits are hard to attribute. AHE addresses this through observability-driven evolution and improves Terminal-Bench 2 performance while transferring across tasks and model families.

  • Motivation: Harness design materially changes long-horizon task completion, and its optimal configuration is model-specific, making manual adaptation increasingly difficult.Developers currently inspect trajectories and hand-craft edits across prompts, tools, middleware, and skills.
  • Problem: Joint evolution remains open because few approaches edit the full harness, while long trajectories provide little actionable signal and tightly coupled frameworks make edits error-prone.Existing methods commonly target a single surface such as prompts, skills, or playbooks.
  • Approach: AHE makes observability the design pivot by exposing editable components as files, distilling trajectories into layered evidence, and pairing edits with verifiable predictions.These three pillars create a structured action space and falsifiable edit contracts.
  • Results: Ten AHE iterations raise pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing human-designed and self-evolving baselines.The result is reported for the Terminal-Bench 2 evaluation.
  • Results: The frozen harness transfers across SWE-bench-verified and three alternate model families, while tools, middleware, and long-term memory carry gains that the system prompt alone does not.The ablation indicates factual harness structure transfers more reliably than prose-level strategy.
  • Limitations: AHE’s self-attribution is reliable for fixes but largely blind to regressions, making regression foresight the clearest direction for future self-evolution loops.The reported regression precision and recall are only about twice their random baselines.

2 Related Work

Related work situates harness engineering as the design of the model-surrounding system and reviews increasingly realistic coding-agent evaluations and automated optimization approaches. AHE differs by optimizing the full harness as a combinatorial whole with minimal hand-specified methodology.

  • Harness Engineering: Harness engineering designs the tools, interfaces, memory, execution constraints, and feedback loops surrounding a coding model.These components mediate how agents perceive and act on repository environments during long-horizon tasks.
  • Evaluation: Coding-agent evaluation spans function-level tasks, repository-scale patch resolution, and multi-hour terminal workflows, alongside infrastructure for reproducible runtimes and verifiers.The evaluation landscape develops along task-horizon and environmental-realism axes.
  • Automated Optimization: Automated optimization methods vary in the evidence they observe and the surfaces they edit, including outputs, prompts, playbooks, demonstrations, and program structure.Many approaches optimize a restricted portion of the agent system rather than the complete harness.
  • AHE’s Position: AHE tunes the full harness as a combinatorial whole and leaves methodology for the optimizer to discover from rollouts rather than fixing it by hand.This makes cross-component trade-offs visible to the optimizer.

3 Method

AHE implements an unattended harness-evolution loop in which structured component, experience, and decision artifacts guide edits while keeping the base model and evaluation infrastructure fixed. Rollouts are distilled into drill-down evidence, edits are committed with predictions, and rejected changes are reverted after the next round.

  • Method: AHE records harness components, rollout trajectories, and edit decisions as structured artifacts that another agent can inspect and act on.This design makes every phase of the evolution loop observable.
  • Three Observability Layers: The three observability layers expose file-level components, layered rollout evidence, and change manifests whose predictions are verified in the next round.Together they compose an unattended iteration loop.
  • Component Observability: The substrate exposes seven loosely coupled harness component types as files, allowing edits to be localized, diffed, committed, and rolled back at file level.The components include prompts, tools, middleware, skills, sub-agents, and long-term memory.
  • Experience Observability: AHE generates multiple task traces, navigates them through a file-based Agent Debugger environment, and aggregates reports into a benchmark-level overview.Original raw and lightly processed traces remain available for verification through progressive disclosure.
  • Decision Observability: The Evolve Agent reads distilled evidence, edits only the harness workspace, and records each change as a versioned, falsifiable file-level claim.The runs directory, tracer, verifier, and model configuration are read-only, while the seed system prompt is non-deletable.
  • Decision Observability: Each manifest entry records failure evidence, root cause, targeted fix, and predicted fixes and regressions, which the next evaluation intersects with observed task-level deltas.Rejected edits are reverted, replacing rationale-based justification with measurable inter-round contracts.
  • Outer Loop: Algorithm 1 iterates rollout, trace cleaning, prior-edit attribution, rollback, evidence distillation, harness evolution, and commit operations from a seed harness.Using k ≥2 rollouts per task provides pass-rate signals for comparative diagnosis.

4 Experiments

The experiments evaluate AHE against human-designed and self-evolving harnesses, test transfer across tasks and model families, and analyze component contributions and self-attribution reliability. AHE improves benchmark performance and transfers broadly, but component interactions and regression blindness limit the loop.

  • The study asks where AHE stands against existing harnesses, whether it transfers beyond its optimization target, and what drives its gains.
  • Ten iterations start from the bash-only NexAU0 seed on all 89 Terminal-Bench 2 tasks, using GPT-5.4 high for the Code, Debugger, and Evolve Agents.The campaign runs for roughly 32 hours, and ACE and TF-GRPO share the same seed.
  • AHE outperforms every baseline overall, although Codex marginally leads on the Hard tier.Replacing the seed’s long-term memory alone with AHE’s memory surpasses Codex on Hard, implicating component interference rather than missing capability.
  • Prompt-only self-evolution leaves gains on tools, middleware, and long-term memory untouched, whereas AHE jointly evolves these components with the system prompt.The gain concentrates in the latter three components, which ACE and TF-GRPO do not edit.
  • On SWE-bench-verified, ACE and TF-GRPO regress below the seed while using 11% to 29% more tokens, whereas AHE achieves the highest aggregate and cuts tokens by 12% against the seed.AHE’s gains concentrate on the largest, most token-expensive repositories; marginal regressions occur only on the three smallest repositories.
  • Cross-family transfer gains are +10.1 pp for deepseek-v4-flash, +6.3 pp for qwen-3.6-plus, and +5.1 pp for gemini-3.1-flash-lite-preview, exceeding the +2.3 pp within-family gain.The evolved harness is evaluated without further evolution; the paper attributes stronger cross-family gains to coordination patterns encoded in tools, middleware, and memory.
  • Three positive single-component gains sum to +11.1 pp, exceeding full AHE’s +7.3 pp because components interact non-additively.The aggregate is dominated by 55 Medium tasks, producing a Medium-heavy trade-off that gives back part of the Hard-tier memory effect.
  • Fix self-predictions reach 33.7% precision and 51.4% recall, versus random baselines of 6.5% and 10.6%, but regression precision and recall are only 11.8% and 11.1%.The authors identify regression foresight as the clearest direction for future self-evolution loops.

5 Conclusion

AHE turns a coding agent’s harness into a learnable adaptation surface while keeping the base model fixed. Ten iterations improved Terminal-Bench 2 performance, and the frozen harness transferred to another benchmark and three alternate model families.

  • AHE exposes harness components as files, distills rollouts into layered evidence, and binds each edit to a falsifiable next-round prediction.
  • Ten iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%.
  • The frozen harness transfers to SWE-bench-verified and three alternate model families without further evolution.
  • Harness-level evolution complements model-side training by providing an externalized, auditable surface where coding-agent experience can accumulate.

Limitations

The evaluation scope, operating point, and governance mechanisms constrain how broadly AHE’s results should be interpreted. Broader deployment settings, multiple operating points, and stronger safeguards remain open.

  • The evaluation covers Terminal-Bench 2 evolution and SWE-bench-verified transfer, leaving broader languages, repository-scale deployments, and human-in-the-loop workflows untested.
  • AHE’s evolution budget and per-task timeout were fitted to GPT-5.4 high, coupling cross-model transfer results to the chosen operating point.
  • Re-running the loop under multiple operating points is required to disentangle harness portability from operating-point coupling.
  • AHE provides workspace bounds, versioned attribution, and file-level rollback but lacks a complete guardrail stack for long-horizon cleanup and misuse prevention.

A Experimental Setup: Full Details

The appendix specifies the evaluation metrics, runtime infrastructure, seed configuration, and reference operating points used for AHE experiments.

  • Setup: The appendix formalizes the metrics and runtime infrastructure underlying the reported AHE experiments.It expands the condensed setup with metric definitions and execution details.
  • Seed agent: NexAU0 is the seed agent: a bash-only code agent without skills, middleware, or long-term memory.All AHE iterations edit this workspace, and gains are measured against NexAU0.
  • Runtime infrastructure: Each rollout runs in a fresh E2B sandbox, while Harbor dispatches tasks and verifies pass/fail outcomes.InMemoryTracer records trajectories, and the Agent Debugger runs with concurrency 16 and a 600-second task timeout.
  • Metrics: pass@1 is the mean binary success rate over k rollouts per task.Infrastructure exceptions contribute zero rather than being discarded, and k varies by experiment.
  • Hyperparameters: The reference AHE run uses four agents and reports hyperparameters from the configuration snapshot that produced the Terminal-Bench 2 results.The table caption states that all four agents share GPT-5.4 and differ in reasoning tier, sampling, and per-component limits.
  • Metrics: Tokens k counts prompt plus completion tokens across every LLM call and is reported as a mean in thousands.Succ/Mtok combines success and token cost as expected successes per million tokens.

B Prompts and Configurations

The appendices document the literal prompts, workspace contracts, component taxonomy, loading rules, and source-code guidance that govern AHE evolution.

  • B Prompts and Configurations: The appendix reproduces the prompts and configurations used in the experiments, with runtime-filled Jinja placeholders.The files correspond to the public repository commit producing the Section 4 experiments.
  • B.1 Code Agent Seed System Prompt: The seed code-agent prompt is intentionally minimal, giving the model one shell tool, three behavioral rules, and runtime variables.The rules cover concise replies, warnings before destructive commands, and background execution for long-running processes.
  • B.2 Evolve Agent Prompt: The Evolve Agent prompt requires workspace-only control, evidence-driven changes, and a versioned change-manifest deliverable.Each change records failure evidence, root cause, targeted fix, predicted impact, and possible task risks.
  • B.2 Evolve Agent Prompt: The loop convention distinguishes previous outputs in input/ from current edits in evolve/, which the next iteration evaluates.An evaluation completed in iteration N concerns iteration N-1’s changes.
  • B.2 Evolve Agent Prompt: The workspace exposes seven editable component types, including prompts, tools, middleware, skills, sub-agents, and long-term memory.Short-term memory is session-scoped and read-only for the Evolve Agent; component characteristics determine when each level is appropriate.
  • B.2 Evolve Agent Prompt: The Evolve Agent should consider every component level before selecting a fix and revisit the level after two or more persistent failures.Creating a file alone is insufficient for new components; registration in code_agent.yaml is also required.
  • B.2 Evolve Agent Prompt: Component code is loaded through workspace-relative bindings and imports, while runtime environment variables specify the current LLM provider and model.Examples map tool and middleware imports to files under the workspace.
  • B.3.1 Source-code Exploration Agent: The source-code exploration agent produces a practical development guide for creating and modifying NexAU components.NexAU supplies tools, middleware, configuration loading, and an execution loop for coding agents.

C.1.1 Trajectory 1: db-wal-recovery

The db-wal-recovery case shows how evidence-linked harness changes replace guessed reconstruction and proxy checks with specification- and evaluator-aligned behavior.

  • C.1.1 Trajectory 1: db-wal-recovery: db-wal-recovery requires reconstructing a SQLite database from a corrupted WAL and writing an exact recovered.json output.The verifier checks every row and updated value against ground truth.
  • C.1.1 Trajectory 1: db-wal-recovery: The seed passed 1 of 2 rollouts, while the failing rollout used stale shell output, guessed missing rows, missed existing-row mutations, and relied on row-count validation.The failure exemplifies proxy validation instead of evaluator-isomorphic validation.
  • C.1.1 Trajectory 1: db-wal-recovery: Figure 5 illustrates three Evolve Agent edits: a middleware new file, a system-prompt append, and a shell-tool edit.These examples span the controllability levels used for harness evolution.
  • C.1.1 Trajectory 1: db-wal-recovery: Four of eight iteration-2 rules fired on the failing trajectory, each linking a failure step to a corrective passing step.Figure 6 presents the shared prefix, failure divergences, triggered rules, and corresponding successful steps.
  • C.1.1 Trajectory 1: db-wal-recovery: The contract-first rule forces rereading the specification, the no-overfit rule blocks extrapolation, and the mirror-the-evaluator rule aligns self-checks with hidden assertions.Together, the rules address cached output, guessed values, and the inadequate json-length check.
  • C.1.1 Trajectory 1: db-wal-recovery: db-wal-recovery passed 2/2 on the next evaluation after the iteration-2 changes.The result demonstrates one task flipping through targeted harness edits.

C.1.2 Trajectory 2: path-tracing

The path-tracing case shows an execution-time shell guard repairing a state-destruction failure that prompt advice alone did not prevent.

  • C.1.2 Trajectory 2: path-tracing: The iteration-5 round targeted cross-task post-validation state destruction and improved tasks beyond those explicitly named by the Evolve Agent.Path-tracing was among the tasks benefiting from this targeted change.
  • C.1.2 Trajectory 2: path-tracing: path-tracing renders a scene into reconstructed.ppm, which the verifier compares pixel-for-pixel with a reference image.Only that output file is read by the verifier.
  • C.1.2 Trajectory 2: path-tracing: At iteration 4, both rollouts deleted a correct reconstructed.ppm during cleanup and submitted on the cleanup command’s exit code.The verifier then rejected the rollout because the required file was absent.
  • C.1.2 Trajectory 2: path-tracing: At iteration 5, path-tracing improved from 0/2 to 2/2 after a shell guard blocked deletion of the protected output.The agent acknowledged the guard and completed without rerunning the destructive cleanup.
  • C.1.2 Trajectory 2: path-tracing: The same iteration recovered polyglot-rust-c and large-scale-text-editing, while configure-git-webserver improved only partially until iteration 8.The later gap involved a reset path that the iteration-5 guard still allowed.

C.1.3 Trajectory 3: mcmc-sampling-stan

Trajectory 3 shows complementary tool- and middleware-level changes converting a five-iteration proxy-then-skip failure into completed, cross-checked execution of the required analysis.

  • Iteration-6 changes: The iteration-6 harness combined a tool-level publish-state guard with step-spanning execution-risk middleware.The pair covered script entrypoints and risk patterns including inline-proxy and shallow validation, matching the failure sequence.
  • Task: The task required installing rstan 2.32.7, fitting a hierarchical betabinomial model to 30 observations, and writing posterior means to two text files.The verifier reran analysis.R end-to-end and checked alpha in [2.84, 2.91] and beta in [16.1, 16.7].
  • Failure before iteration 6: From iterations 1 through 5, the agent used grid integration as a proxy, killed background MCMC, and submitted files that the verifier rejected.The unconverged sampler produced values around 1e19, outside the expected range; earlier safeguards protected deliverables but not analysis.R.
  • Outcome: After the changes, both rollouts ran analysis.R at iter = 100000 to completion, cross-checked an independent full run, and published the corrected results.The changes addressed both the unprotected script entrypoint and the trajectory’s proxy-validation and premature-termination risks.

C.1.4 Trajectory 4: configure-git-webserver

Trajectory 4 shows the harness preserving a successful deployment by hard-blocking destructive cleanup and making relevant middleware warnings salient on the next model turn.

  • Task: The task required an externally reachable webserver serving a deployed page from a git working tree.The verifier issued an HTTP request from outside the agent’s shell and checked the response body.
  • Failure at iteration 7: At iteration 7, a successful deployment was destroyed by overrideable cleanup commands that deleted the page and reset the repository state.The verifier received a 404; git-multibranch failed for the same structural reason.
  • Outcome: After installation, configure-git-webserver improved from 0/2 to 2/2 because the agent dropped cleanup after hard-block messages and submitted the live state.The same successful deployment was reached, but protected web-root and ref modifications were refused at the shell layer.
  • Iteration-8 changes: The iteration-8 tool change upgraded protected-output deletion and repository resets from soft reasons to hard shell blocks.The change preserved the prior architecture while preventing the destructive command itself.
  • Iteration-8 changes: The iteration-8 middleware added contract-aware warnings for clean-layout builds and unofficial wrapper or revision usage.These warnings were designed to appear on the next model turn, addressing the prior salience gap.
  • Attribution: AHE carries each edit with predicted fixes, regressions, and constraint level into the next iteration’s attribution check for retention or rollback.The change manifest makes edits binding contracts whose predicted effects are evaluated against subsequent task outcomes.

D Per-round Self-attribution Breakdown

The per-round breakdown evaluates whether predicted fixes and regressions match observed outcomes, using precision and recall alongside contemporaneous pass@1.

  • Panel design: Figures 12 and 13 decompose fix and regression predictions into true and false positives or negatives across precision and recall panels.Dashed lines show the metric on a 0 to 100% axis, while solid lines show contemporaneous pass@1.
  • Fix attribution: Fix precision and fix recall swing from near-zero to near-saturation across rounds.This pattern supports the paper’s reported self-attribution analysis of evolving harness improvements.
  • Interpretation: The breakdown pairs attribution metrics with contemporaneous pass@1 to assess prediction quality round by round.The figure captions distinguish precision denominators based on predictions from recall denominators based on actual outcomes.
  • Practical implication: AHE is presented as lowering human harness-engineering cost while obtaining higher pass@1 from the same base model at lower per-trial token cost than prompt-only self-evolution baselines.The claimed practical consequence is broader access to capable coding agents without dedicated harness teams.
Loading 2604.25850v4…