Source-linked AI summary
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, Wenxuan Zhang
TL;DR
Agent capability depends partly on model-external harnesses, yet evaluations usually treat those systems as fixed configuration rather than development targets. HarnessDev evaluates Creation and feedback-driven Evolution of runnable harnesses, measuring held-out capability and execution cost. Creation shows strong domain variation, while Evolution gains are unstable, transfer partially to held-out tasks, and depend on the executing model.
Problem
Most agent evaluations report task performance under a selected harness, leaving models’ ability to create and continually improve persistent runnable harnesses comparatively underexplored.
Method
HarnessDev has Creation from a weak runnable seed and Evolution from an existing harness using downstream execution feedback, evaluating frozen harnesses on capability and executor-token efficiency.
Results
Creation matches or exceeds selected references in writing and machine-learning experimentation but trails them in code, search, and research; Evolution gains are unstable, partially transfer, and runtime-model dependent.
Takeaways & Limitations
HarnessDev makes model-developed execution systems an explicit, inspectable, reusable, and testable object of agent evaluation.
Takeaways & Limitations
The study has uneven human baselines, incomplete benchmark coverage, limited Evolution trajectories, and held-out Evolution evaluation only on SWE-Pro.
Abstract
from arXiv · showhide
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.
1 Introduction
HarnessDev evaluates whether LLMs can build and iteratively improve persistent agent harnesses, rather than merely produce task outputs. It measures both downstream capability and execution efficiency, finding strong domain dependence in Creation and unstable, model-dependent gains in Evolution.
- HarnessDev: HarnessDev shifts evaluation from one-off task outputs to runnable infrastructure that persists across downstream tasks.The benchmark covers Creation from a weak runnable seed and Evolution through continued feedback-driven maintenance.
- Evaluation: Capability measures held-out task success, while efficiency measures executor-model tokens consumed by the frozen harness.Creator tokens used to build or modify the harness are excluded from execution-cost measurement.
- Harness Creation: Creation performance varies sharply by domain: generated harnesses match or exceed references in writing and machine-learning experimentation but lag in code, search, and research.Search and research require long-horizon information seeking, while code requires repository inspection, editing, and verification across many turns.
- Harness Creation: Execution cost varies substantially across creator models, and higher token use does not reliably produce better downstream results.Harness quality therefore requires joint assessment of capability and efficiency.
- Harness Evolution: Evolution can produce useful local improvements, but revisions often fluctuate, transfer weakly to unseen tasks, and depend strongly on the runtime model.Changing the model that executes the harness can alter both starting performance and whether later revisions help.
2 Background
HarnessDev targets the execution layer that conventional benchmarks usually fix in advance. It asks whether models can build and maintain that scaffold as deployment requirements, feedback, and task execution evolve.
- Background: Most agent benchmarks prespecify the task, reward or judge, and execution scaffold, leaving deployment-specific engineering outside evaluation.Real deployments instead require systems to work with customer data formats, workflows, and compliance constraints.
- Background: The model-side challenge includes translating vague intent into objectives, constructing reliable feedback, and building usable execution systems.These systems must manage tools, context, state, lifecycle logic, and verification as requirements change.
- Background: HarnessDev isolates the execution layer by evaluating whether models can create a scaffold from a weak starting point and improve it while preserving compliance and held-out performance.This distinguishes harness development from broader deployment-capability growth and interaction-experience reuse.
3 Benchmark
HarnessDev evaluates runnable execution systems rather than single-task answers, separating harness Creation from Evolution and measuring both downstream capability and execution efficiency. Creation builds from a weak seed under constrained development feedback, while Evolution revises a creator-built harness using designated benchmark feedback before held-out evaluation.
- 3.1 Overview: HarnessDev evaluates a frozen runnable harness reused across downstream tasks, with an executor model producing outputs that an evaluator scores.The harness contains execution-loop, tool, context, state, lifecycle, and verification components.
- 3.2 Development settings: Creation asks whether a model can build an effective harness from a weak runnable seed, task specification, constraints, tutorial, and one to three development cases.The resulting harness is frozen before evaluation, and the creator never sees the human implementation or hidden evaluation set.
- 3.2 Development settings: Evolution asks whether a model can improve its own Creation harness from downstream execution feedback while preserving already-working behavior.The protocol starts from the creator’s frozen code harness, uses feedback from 100 SWE-Pro tasks and all 89 Terminal-Bench tasks, and permits ten post-H0 full-evaluation pairs plus fixed-subset probes.
- 3.2 Development settings: The common seed supplies compatibility, configuration, permitted low-level tools, and artifact logging, but no agent loop, planning, context, state, verification, recovery, or stopping logic.Unmodified, it does not attempt tasks and scores zero; nonzero Creation performance must come from creator-added execution logic.
- 3.3 Domains and downstream benchmarks: Creation covers four domains and five downstream benchmarks totaling 2,207 unique instances, while Evolution focuses on code harnesses and reuses benchmark-suite tasks as feedback.Creation uses all 731 SWE-bench Pro public instances; Evolution uses a 100-task feedback subset, a disjoint 630-instance held-out split, and all 89 Terminal-Bench tasks as feedback.
- 3.4 Evaluation: Capability is measured by task-level success on held-out downstream tasks, while efficiency is measured by executor-model tokens consumed by the frozen harness.Every score uses a standardized runtime with fixed executor and evaluator within each comparison; hidden scores remain unavailable during development.
- 3.4 Evaluation: Creation compares generated harnesses with the common seed and, where available, mature human-engineered references under both creator-matched and fixed-executor evaluation.Self-Eval sets the executor equal to the creator, whereas Unified-Eval runs every generated harness with the same fixed executor for direct comparison.
- 3.4 Evaluation: Official Evolution versions require complete two-benchmark evaluation pairs and are additionally tested on disjoint held-out tasks after trajectories end.This separates adaptation to observed feedback from held-out generalization; constraint rules prohibit instance-specific solutions, hidden evaluation access, and replacement of the provider-neutral runtime interface.
4 Experiments
HarnessDev evaluates harness creation and evolution across downstream performance, execution cost, executor transfer, and implementation behavior. Results show substantial variation across task families and creators, unstable evolution gains, and strong dependence on executor compatibility and feedback quality.
- Harness Creation: Creation quality varies substantially across task families: writing approaches mature references, while Search has the largest gap and Code also remains behind.Under Self-Eval, Opus 4.8 has the highest overall score at 67.8 versus the human-engineered reference at 86.2.
- Harness Creation: 77.8% of failed Data tasks are attributed to harness defects, indicating that failures are not explained only by executor capability.
- Harness Creation: Independent harnesses from the same creator can differ sharply, and executor changes can reverse performance because limits and stopping rules are model-specific.An Opus Code harness nearly collapses under Gemini after hard-coding a 120-step limit around its original executor.
- Harness Creation: Implementation size is not predictive of performance: Gemini adds 1,006 Code lines yet achieves the best Terminal-Bench score of 68.8.Across Code artifacts, state and memory remain weak, with only one artifact exposing state saving and one implementing periodic checkpointing.
- Harness Creation: Executor transfer is harness-dependent: Qwen gains 17.6 points on BrowseComp and 12.9 on MLE-bench under Gemini, while Opus’s SWE-Pro score falls from 69.3 to 33.0.Compatibility depends on prompts, tool protocols, budgets, and stopping rules remaining aligned with the runtime model.
- Harness Evolution: Evolution improves visible feedback performance, but held-out gains are small and unstable: Opus achieves the largest held-out improvement at +4.44 points, while fixed-Gemini transfer benefits only Opus.Across 64 switches, feedback and held-out scores move together only 34 times, or 53.1%.
5 Related Work
HarnessDev builds on work evaluating agent benchmarks, automated agent construction, and executable harness evolution. It distinguishes optimizing a provided harness from connecting creation, evolution, runtime transfer, cost, and held-out evaluation.
- Standard agent benchmarks evaluate task execution under a selected harness, while Harness-Bench measures how harness choice changes model performance.
- HarnessOpt-Bench optimizes a provided harness using graded feedback and inaccessible test evaluation, whereas HarnessDev connects from-scratch Creation to Evolution.
- HarnessDev evaluates frozen artifacts under self and fixed runtime models, measures execution-token cost, and tests every version on a disjoint 630-task SWE-Pro split.
- Evo-Bench evolves a shared CodeAct seed with a fixed runtime model, whereas HarnessDev adds Creation, self- and fixed-runtime views, cost, transfer, and every-version held-out scoring.
- Related systems search prompts, workflows, operators, topologies, memory, tools, or executable harness code, including versioned snapshots and trace-informed proposals.
6 Discussion, Limitations, and Conclusion
HarnessDev shows that harness quality depends on domain, runtime, and evolution stability, motivating direct evaluation of persistent execution systems. Its conclusions are bounded by uneven references, limited trajectories, benchmark coverage, and deployment-safety considerations.
- Discussion: Creation matches the human reference in writing and exceeds it in machine-learning experimentation, but remains far behind in search and research and trails it in code.
- Discussion: Cross-executor comparisons show stronger executors can improve some harnesses, while others exhibit creator co-adaptation.
- Discussion: Evolution is unstable: useful intermediate updates can be erased, and more updates do not guarantee a positive final gain.
- Discussion: Changing only runtime binding can substantially move H0 and alter which harness changes are useful.
- Limitations: The study uses one trajectory per creator–runtime cell, one unfinished main-runtime cell, and post-freeze held-out evaluation only on SWE-Pro.
- Limitations: Generated harnesses should be treated as untrusted code because the container boundary was provisioned for reproducibility rather than containment.
7 Contributions
The benchmark draws candidate systems from four categories and assigns them roles such as human-engineered counterpart, update-history source, or development environment. The categories span code, data analysis, writing, and research or retrieval.
- The benchmark’s four categories are code agent, notebook/data-analysis, writing agent, and research/retrieval.
- Code agent: Code-agent candidates include Claude Code, OpenCode, OpenHands, SWE-agent, and mini-SWE-agent.
- Notebook / data-analysis: Notebook and data-analysis candidates include DataAgent and DB-GPT.
- Writing agent: Writing-agent candidates include AutoResearchClaw and webnovel-writer.
- Research / retrieval: Research and retrieval candidates include Alibaba-NLP/DeepResearch, dzhng/deep-research, modelscope/ms-agent, and gptresearcher.
B Experimental Configuration
The experiments specify creator models, development environments, runtime references, downstream resources, and execution-cost accounting. Configuration differs by stage and benchmark, and reported references pair human-engineered harnesses with their associated executors.
- Configuration: Table 8 records creator LLMs, development environments, decoding configurations, and stage-specific participation.
- Configuration: MLE-bench tasks run in isolated containers with one NVIDIA A800-SXM4-80GB GPU, 14 vCPUs, 227 GiB RAM, a 36,000-second clock, and a 500-step cap.
- References: Human-engineered references use the highest publicly verifiable system-level result, pairing each harness with the executor model used by that system.
- References: Three Table 3 starred values are external reports: SWE-Pro 80.0 for Claude Fable 5, Terminal-Bench 2.1 88.8 for GPT-5.6 Sol, and BrowseComp 92.2 for GPT-5.6 Sol.
- Cost and performance: Figure 9 plots downstream score against execution tokens on a log scale under Self-Eval and Unified-Eval, with dashed human-reference lines where measured.
- Evolution execution cost: Figure 10 reports task-agent/runtime tokens across frozen harness versions, excluding creator, judge, and probe tokens.
C Harness Interface Specification
The harness contract defines required execution responsibilities, artifacts, and evaluation boundaries for a runnable system. A weak seed provides only the runnable floor, so creators must implement task-solving behavior and domain-specific outputs.
- Functional modules: Each admissible harness implements six functional modules covering execution, tools, context, state, lifecycle, and evaluation.These responsibilities include action execution, tool registration and calls, prompt construction and compression, checkpointing, recovery, verification, and trajectory recording.
- Seed implementation: The shared weak seed supplies a stable CLI, runtime-model configuration, audit writers, and policy-free primitives, but no task-solving policy.Its unmodified pass parses inputs, may issue a summary-only LLM probe, writes the audit envelope, and terminates with partial status.
- Development boundary: Creators must implement the execution modules using only public development-task feedback while hidden tasks, answers, and official scores remain withheld.The reference seed, audit script, and related contract are shared, while hidden evaluation tasks are reserved for evaluation.
- Domain outputs: The authoritative final artifact is domain-specific: patches for code, scorer-readable submissions for data analysis, prose for writing, and evidence-grounded answers for research.Common JSON and log files alone do not constitute complete execution.
D Evaluation Settings and Roles in Full
This appendix defines the evaluation setting and model roles summarized in Section 3.4.
- The appendix provides definitions for the evaluation setting and model roles summarized in Section 3.4.
D.1 Self and unified evaluation
HarnessDev separates harness development from downstream execution to measure the generated software system rather than a single answer. Self-Eval measures creator–harness co-design, while Unified-Eval compares harnesses under a common executor.
- Self-Eval: Under Self-Eval, the creator LLM also executes hidden downstream tasks through its own frozen harness.This regime measures whether a model builds a harness suited to its own capability boundary.
- Unified-Eval: Under Unified-Eval, one fixed executor runs every generated harness, reducing executor-LLM differences in harness comparisons.A high unified score indicates transferability as a software asset rather than fit to one model.
- Roles: The creator develops or modifies the harness in environment D, while executor LE runs downstream tasks only after the harness is frozen and evaluator J scores the output.Separating these roles prevents development tools or executor ability from being attributed to harness quality.
- Seed boundary: The shared seed is deliberately incomplete, lacking mature tool-loop, state, context-compression, verification, recovery, memory, and benchmark-policy components.Creators may retain, modify, replace, or delete it while preserving the invocation contract.
- Run contract: Every run must produce machine-readable results, structured trajectories, human-readable responses, logs, and task-specific final artifacts with honest status values.Success requires reasonable evidence of completion; partial indicates progress without sufficient verification or with a blocked dependency.
- Harness artifact: The generated harness must be executable system code that performs the required responsibilities, not merely an architecture description or README.It combines with the runtime LLM to form the generated agent.
05 Validate the execution contract before spending a real benchmark budget
The development record emphasizes validating execution contracts and interpreting diagnostics before relying on benchmark outcomes. Smoke tests established runnable paths, while interrupted or stale runs were retained as incomplete evidence rather than treated as scores.
- Validate the execution contract: 1.0 contract smoke verified the CLI, runtime LLM, batched actions, required files, and trajectory capture, but not difficult-task capability.The run reported a started harness and found trajectory.
- Diagnose parser failures: Parser diagnostics showed that accepting multiple JSON actions per assistant turn prevented invalidation and wasted steps and context.The proposed fix used JSONDecoder.raw_decode to extract up to eight objects and convert them into a batch action.
- Diagnose environment failures: Shell diagnostics required selecting /bin/bash for shell execution when available because /bin/sh rejected otherwise executable Bash fallbacks.The corresponding run remained incomplete because its old container did not hot-load the fix.
- Separate activity from scores: The created agent demonstrated real repository and Terminal activity, including compiling /app/gpt2.c, but incomplete summaries or verifier results kept those runs from being scores.The record explicitly preserved these as pending diagnostics and continued from the same workspace.
- Interpret settled outcomes: Terminal 1.0 passed its verifier, while SWE-Pro scored 0.0 despite producing a non-empty patch; later edits were not hot-loaded into the active runs.The final record distinguishes official scores from local final-snapshot verification.
- Construct the harness: The resulting harness used a single-threaded native tool-calling ReAct loop, structured tools, context handling, verification, artifact management, and explicit limits on steps, time, and context.Repeated no-tool turns or an empty repository diff were downgraded to partial rather than reported as success.
13 Close with verified interfaces but no benchmark-driven harness revision
The creator built and repeatedly revised a harness, ending with verified interfaces and artifact checks, but the revisions were largely probe- and architecture-driven rather than benchmark-driven.
- Close with verified interfaces: The initial harness passed interface and artifact checks, yet its broad construction did not reflect a core edit caused by settled zero results.Opus checked CLI forms, target files, seven artifact classes, imports, and package listing; the harness contained 1,876 lines.
- No benchmark-driven revision: H0 was confirmed as SWE 51/100 and Terminal 60/89, prompting architecture-first edits rather than feedback-driven revision.The first revision added filesystem, permission, planning, artifact-path, JSON-action, and Python-entry-point mechanisms.
- No benchmark-driven revision: Subsequent revisions added success gates, final review, path filtering, and artifact tracking after weak self-checks and unrecorded shell edits caused evaluation concerns.The creator extended review to all Terminal successes and compared files before and after shell commands.
- No benchmark-driven revision: Probe outcomes repeatedly guided reversions and additions, but high probe scores did not reliably transfer to full evaluation.A fixed Terminal probe fell from 4/5 to 3/5, later reached 4/5 and 5/5, and still failed to predict full results.
- Close with verified interfaces: The selected final version scored SWE 56.0 and Terminal 74.157, while leaving three evolution pairs unused.The creator selected T2 because it remained the true combined-score argmax rather than sampling another candidate.
- Close with verified interfaces: The final harness never repaired its own 54% tool-call rejection, underscoring the gap between interface validation and complete harness improvement.The final decision retained a candidate that improved benchmark scores but did not address this rejection rate.
03 Extend reflection to non-git and Terminal tasks
Reflection was extended from git-based repository tasks to non-git and Terminal artifacts, but evolution remained constrained by limited gains and cautious sampling decisions.
- Extend reflection to non-git and Terminal tasks: Reflection was extended to inspect written paths, command state, and recent workdir files for Terminal and non-git tasks.This addressed the failure of git-diff reflection to fire when shell commands created artifacts.
- Extend reflection to non-git and Terminal tasks: All 25 T2 SWE failures had already triggered reflection, but the revision still focused on general reflection rather than concrete interface mismatches.The evidence suggests the remaining SWE failures were not simply uncovered by adding reflection.
- Extend reflection to non-git and Terminal tasks: A formal T4 evaluation was absent: the final 5faaa15 commit changed only LEDGER.md after concurrent sessions made opposite edits.One session attempted timeout handling while another reverted the tools change.
- Extend reflection to non-git and Terminal tasks: The creator estimated roughly ±3–4 tasks of full-evaluation noise and declined seven remaining pairs, selecting T3 as a validated tie-break.T3 scored SWE 74.0 and Terminal 74.157; T1 was higher by about 0.06 pp but passed the same total number of tasks.