Source-linked AI summary
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, Yuan Xue
TL;DR
Harness optimization demands reasoning under expensive, stochastic evaluation, but lacks a common way to measure frontier models’ ability to improve agents. HarnessOpt-Bench introduces a controlled benchmark and finds that optimizer models differ more than the coding harnesses they use, with gains varying substantially across tasks and seed regimes.
Problem
Harness optimization is a demanding capability because agents must be improved from incomplete, costly, and noisy evaluation evidence rather than through ordinary code testing.
Method
HarnessOpt-Bench has LLM optimizers edit seed harnesses under fixed budgets and scores nominated candidates by normalized held-out gain within a trusted evaluation protocol.
Results
Optimizer-model differences were larger on average than shared-versus-native harness differences, with no consistent native-harness advantage and substantial variation across tasks and seed regimes.
Takeaways & Limitations
HarnessOpt-Bench makes harness engineering a reproducibly measurable capability, while current frontier models improve agents unevenly and remain difficult to rank finely.
Takeaways & Limitations
The benchmark is hack-resistant rather than hackproof because repeated development and validation feedback may reward strategies tailored to stable evaluator artifacts.
Abstract
from arXiv · showhide
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.
1 Introduction
Harness optimization is a demanding, stochastic, long-horizon capability that requires more than coding, motivating HARNESSOPT-BENCH as a controlled evaluation target. The benchmark fixes the target and evaluation protocol while holding out test scores, enabling comparison of optimizer models and coding harnesses.
- Motivation: Harness optimization requires diagnosing failures from incomplete evidence, making system-level changes, managing limited evaluations, distinguishing improvement from noise, and selecting what to deploy.Its difficulty arises because harness changes must be assessed by running a stochastic agent across many cases at substantial cost.
- Motivation: Harness optimization combines long-horizon reasoning, interaction with a diverse tool ecosystem, and interpretation of stochastic systems against measured metrics.Together, these properties make it a demanding test of frontier capability beyond producing a single correct action.
- Benchmark: HARNESSOPT-BENCH gives an LLM optimizer a seed harness, graded feedback, and a fixed target-evaluation budget, then scores its nominated candidate by normalized gain on an inaccessible held-out test partition.The benchmark is agnostic to optimizer design so long as a system operates within budget, edits the seed harness, and nominates a final candidate.
- Evaluation protocol: The protocol spans 4 downstream tasks with pinned seeds and non-overlapping splits, while trusted execution enforces access and resource limits, isolates held-out state, and versions candidates for audit.It also records baselines for seed and off-the-shelf harnesses and uses graded disclosure.
- Results: 111 scored optimizer runs compare five frontier models under shared and native harnesses, with optimizer-model differences under the shared harness larger on average than shared-versus-native harness differences.Release-level experiments resolve variation among successive optimizer-model releases on OfficeQA; broader intervention correlates with greater held-out gain, while detailed failure-trace inspection is rarely used and not positively associated with gain.
2 Related Work
Prior work spans prompt, code, and end-to-end harness optimization, but differing modification scopes, search roles, seeds, budgets, and scoring protocols limit comparability. HarnessOpt-Bench addresses this gap by fixing the optimization and evaluation setup while positioning harness optimization alongside coding-agent benchmarks.
- Automated code and harness optimization: Existing optimization systems differ in which harness components they modify and what role the LLM plays during search.Prompt optimization methods such as DSPy modify prompts while holding the surrounding program fixed, whereas other approaches use larger search scaffolds to discover or optimize programs.
- Automated code and harness optimization: End-to-end optimizers let coding agents inspect target source, prior scores, and execution traces while choosing what evidence to gather.This differs from orchestrating mutations within a fixed search algorithm.
- Automated code and harness optimization: Method-specific seeds, budgets, search spaces, and scoring protocols make reported outcomes across existing harness-optimization works difficult to compare.HarnessOpt-Bench instead fixes the optimization problem and evaluation protocol to compare optimizer models, harnesses, and search algorithms on common ground.
- Coding agent benchmarks: HarnessOpt-Bench extends coding-agent evaluation toward iterative optimization while targeting the surrounding harness rather than function synthesis or repository-scale software tasks.It also involves navigating the target system codebase and iterating on environmental feedback.
3 HARNESSOPT-BENCH: Harness Optimization as a Task
HarnessOpt-Bench formalizes harness optimization as a constrained, stochastic program-optimization problem in which an optimizer edits executable harnesses under fixed interfaces, disclosure rules, and evaluation budgets. It selects a candidate using development and validation feedback, while a trusted server evaluates the nominated harness on an inaccessible test partition and reports normalized gain over a pinned seed.
- Candidates and invariants: Harness optimization treats the candidate as an executable codebase whose prompts, tools, memory, and control flow may all be edited under fixed interfaces and immutable paths.The feasible set is H, with H0 ∈ H as the pinned seed and task invariants θ = (M, E, V).
- Evaluation and disclosure: Search uses disjoint development, validation, and test partitions, exposing traces and per-case outcomes on development, aggregate scores on validation, and no test information before nomination.The trusted server evaluates the nominated candidate on Dtest only after search.
- Budget: The optimizer must satisfy a componentwise budget, including 100 evaluation calls per partition, four full development and validation passes, and a cap on expendable target-model tokens.Its own inference is metered but uncapped in this work.
- Optimizer and objective: The optimizer is an LLM operating through a coding harness that produces candidate versions and nominates a final candidate to maximize expected improvement over the pinned seed on held-out test data.Because test disclosure is empty, the optimized quantity is never observed during search; cross-task comparisons use normalized gain, with negative values indicating a candidate worse than the seed.
- Seeds and reproducibility: Pinned seeds are deliberately untuned and leave substantial headroom, while stochastic scoring averages K=3 independent rounds for seeds and candidates and reports task-specific resolution bands.GAIA’s seed is a non-functional stub, and the OfficeQA seed is a roughly 130-line agent with three tools and a 24-turn loop.
4 Experimental Setup
The experiments compare five frontier optimizer models across four tasks under shared and native coding harnesses, with additional harness and release comparisons. Evaluation uses repeated held-out scoring, normalized gain, task-specific resolution bands, and a task-adjusted shared-harness model score.
- Task grid: Five optimizer models are evaluated on all four tasks with both a fixed shared harness, opencode, and each model family’s native harness.The models are claude-opus-5, claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, and kimi-k3; native harnesses are claude-code, codex, and kimi-cli.
- Task grid: Additional comparisons run goose and mini-swe-agent across all models on GAIA, and earlier Claude Opus and GPT releases with native harnesses on OfficeQA.These runs support comparisons of optimizer-harness sensitivity and the capability ladder in Section 5.2.
- Scoring protocol: Each held-out evaluation averages three attempts per test case, each optimizer configuration is run twice, and inference is metered without an optimizer-model cap.Reported ranges in Table 2 reflect the two configuration runs, while uncapped inference estimates performance when optimizer reasoning is not scarce.
- Analysis protocol: Normalized gain g measures the fraction of headroom above the pinned baseline captured by an optimizer.Task normalization places results in common headroom units but does not remove systematic task differences.
- Analysis protocol: The task-specific resolution band is derived from repeated candidate scoring and treats smaller differences as unresolved rather than formally significant.The discrepancy is carried to the K=3 normalized-gain scale used for held-out scoring, producing a descriptive threshold for each task.
- Analysis protocol: LSS-λ is the task-adjusted mean performance relative to the evaluated models’ grand mean, using shared-harness runs on tasks with competent seeds.Native-harness runs are excluded to hold the scaffold fixed, and higher LSS-λ is better.
5 Results
Results show that optimizer-model choice separates more than coding-harness choice, while performance varies across tasks, releases, and search regimes. Search breadth predicts gain, but case budgets and held-out evaluation constrain reliable measurement, and native harnesses have no consistent advantage.
- Model and harness effects: Changing the optimizer model moves normalized gain by 0.142 on average, versus 0.079 for changing the coding harness, a roughly 1.8× larger contrast.Both contrasts exceed task resolution bands, although the harness contrast does so narrowly.
- Model and harness effects: The strongest configuration captures roughly two thirds of OfficeQA headroom and half of BrowseComp-Plus headroom, while the weakest is unresolved from zero on two tasks.Intermediate configurations often differ less than round-to-round variation, supporting tiers rather than a complete ranking.
- Model progress: Across 5 GPT releases, gain rises monotonically from +0.03 to +0.49, whereas across 5 Claude Opus releases it ranges from +0.37 to +0.59 non-monotonically.The GPT series has three of four steps exceeding the task resolution band, and the Claude first-to-last spread also exceeds it.
- Search behavior: Lever coverage is positively associated with gain on every task, with Spearman ρ ranging from +0.34 to +0.88, but it measures exploration rather than retained final changes.Breadth correlates with total modification volume, so the data do not isolate exploration breadth from search effort.
- Search behavior: The median optimizer uses 8 evaluation calls, or 4%, but 82% of its case allowance, and 55 of 100 cells exhaust at least one case budget.Trace-reading share is negatively associated with gain from −0.31 to −0.64, while detailed traces were requested only 16 times by 7 of 111 cells.
- Shared versus native harnesses: Across 20 model–task pairs, the shared harness wins 11 and the native harness wins 9, with 0 ties, showing no consistent native-harness advantage.In 11 pairs, differences exceed task resolution bands, and GAIA harness rankings cross across models, indicating heterogeneous effects.
6 Conclusion
Harness engineering is an empirical model capability: frontier models can diagnose, modify, and improve agents using held-out reward, but performance is uneven and task-dependent. HARNESSOPT-BENCH makes this capability reproducible to measure and concrete to optimize.
- Capability: Frontier models can diagnose, modify, and improve agents as measured by held-out reward.Harness engineering is presented as a model capability rather than merely infrastructure.
- Capability: The strongest models search broadly, but gains remain task-dependent and often too close for fine-grained ranking.Current performance separates models unevenly across tasks.
- Benchmark: HARNESSOPT-BENCH makes harness optimization reproducible to measure and concrete to optimize.The benchmark turns harness engineering into an empirical object.
Limitations
HARNESSOPT-BENCH is hack-resistant but not hackproof, and its conclusions are bounded by fixed evaluation artifacts, task-specific seed priors, and limited coverage of languages, runtimes, architectures, and target models.
- Evaluation robustness: The benchmark blocks access to test data and changes to the target model, environment, or verifier, but stable evaluation feedback may still reward evaluator-specific strategies.Future versions should add per-run jitter to cases, tool behavior, and verifier implementation.
- Seed regimes: Seed harnesses impose task-specific priors, while the suite mixes mature-agent refinement and stub-based construction without systematically varying seed complexity.A controlled ladder of harness completeness and architectural complexity would test how optimizer performance changes with prior strength.
- Coverage: Candidates are limited to Python, and each task uses one pinned target model, leaving generalization to other languages, runtimes, agent architectures, and models untested.Broader coverage, matched compute conditions, and additional replication are needed before treating the benchmark as comprehensive.
Ethics Statement
The work acknowledges the dual-use risks of automated agent improvement while limiting optimization to bounded, benign downstream tasks. It uses public, licensed data without personal or sensitive information and reports resource use transparently.
- Risk mitigation: The benchmark addresses dual-use concerns by restricting optimization targets to bounded, benign tasks such as document QA, deep research, multi-step reasoning, and terminal use.The score is based on the task’s own verifier on a held-out split.
- Data governance: The study reuses existing public datasets and benchmarks under their respective licenses, releases seed agents and split definitions, and introduces no personal or sensitive data.The downstream corpora consist of public documents.
- Resource reporting: The benchmark reports token usage as a first-class metric and denominates its search budget in evaluation calls and case-runs.This makes the computational cost of the work visible.
A Full optimizer results
Table 2 expands the compact optimizer-gain comparison by reporting mean performance, observed ranges across contestant rounds, and two properties of resulting harnesses. It covers the core contestants and two additional GAIA harnesses, while generational-ladder runs remain in Figure 3.
- A Full optimizer results: Generational-ladder runs are excluded from Table 2 and presented separately in Figure 3, where release order is the comparison of interest.The table’s task blocks can differ in length because two coding harnesses ran on only one task.
- A Full optimizer results: Table 2 reports optimizer performance by task as means with observed ranges across each contestant’s rounds.The parenthetical range is not a confidence interval, and two rounds do not estimate dispersion.
- A Full optimizer results: The table repeats the Gain column from Table 1 and includes two additional harnesses evaluated on GAIA.Rows are ordered by model name, so each contestant occupies the same position across task blocks.
B The suite · C Model effects · D Supporting figures
The suite defines task-specific targets, splits, seed baselines, and controlled off-the-shelf harness comparisons, while model effects are reported in normalized-gain units. Supporting figures examine edits, actions, costs, modification volume, harness choice, validation transfer, GAIA sweeps, and case-pass budgets.
- B The suite: Scores average K=3 rounds with standard errors across round means, count timeouts as zero, and include GAIA, OfficeQA Pro, BrowseComp-Plus, and Terminal-Bench 2.0.GAIA begins from a non-functional stub.
- B The suite: Table 3 specifies each task’s pinned target model, data split, held-out seed baseline, and off-the-shelf harness scores under otherwise identical conditions.Off-the-shelf harnesses provide context about available headroom rather than the gain reference.
- C Model effects: LSS-λ reports the model effect after accounting for task, in normalized-gain units, and matches the ordering plotted in Figure 2.Two additional estimators agree on the ordering, so the ranking does not depend on the additive assumption.
- C Model effects: Table 4 also measures resolution as the estimator’s split-round swing and uses it to define tiers for comparing model effects.A shared tier means models are closer together than the movement caused by rerunning the grid.
- D Supporting figures: Supporting figures quantify edit prevalence, shipped edit breadth, optimizer action profiles, and search cost against normalized gain across tasks and optimizer models.Polling is separated from classified non-polling actions, and Figure 6 annotates run counts.
- D Supporting figures: Additional figures relate shipped-agent cost and modification volume to gain, compare native with common optimizer harnesses, and contrast validation with held-out performance.Task-level rank correlations, non-dominated frontiers, significant sign tests, and the equality diagonal provide the figure-specific reference structures.
- D Supporting figures: The remaining analyses sweep GAIA optimizer-model–harness pairings and show that case-pass budgets, rather than evaluation calls, bind resource use.Dashes mark unrun GAIA combinations, while budget annotations count exhausted case-pass caps.
E Reproducibility
The benchmark enforces reproducibility through immutable dataset references, exact non-overlapping splits, re-pinned baselines, and versioned candidate environments. Evaluation inputs and artifacts are fixed and auditable across runs.
- Dataset and split control: Datasets use immutable references and committed manifests to define exact 20/40/40 splits with no overlap.The split generator verifies the committed tree byte-for-byte.
- Baseline control: Baselines are seed agents’ held-out scores pooled over K=3 rounds and re-pinned whenever a seed commit changes.A script reuses the original evaluation path when recomputing baselines.
- Candidate versioning: Every optimizer-produced candidate is stored as an immutable Git commit, with dependencies pinned by lockfile.These controls preserve candidate and environment versions for reproducible evaluation.