Source-linked AI summary
Evo-Bench: Can Language Models Improve Agent Harness?
Lisheng Huang, Chen Yang, Hao Zhou, Huatong Song, Zongchao Chen, Ran Le, Yang Song, Wayne Xin Zhao, Tao Zhang
TL;DR
Static agent evaluations do not isolate models’ intrinsic ability to evolve their own harnesses across tasks and over long horizons. Evo-Bench addresses this gap with controlled fixed-policy evaluation, harness-guided task construction, and sensitivity-aware splitting. Across nine models, top evolvers gain up to 16.6 points and approach human-engineered performance, while results vary by domain and expose early saturation and current limitations.
Problem
Existing evaluations do not isolate harness improvements from base-model strength, prevent task-specific overfitting, or capture long-horizon iterative harness research.
Method
Evo-Bench fixes the policy model and protocol, evolves executable harnesses over long horizons, and constructs tasks through auxiliary harness evolution followed by sensitivity-aware stratified splitting.
Results
Across nine frontier and open-weight models, top evolvers achieve gains reaching 16.6 points over the seed harness and closely approach the human-engineered baseline of 47.5.
Takeaways & Limitations
Evolved harnesses transfer across diverse policy models, but performance is domain-dependent: evolution excels in Search and General tasks while struggling with workflow-specific Office tasks.
Takeaways & Limitations
Evolved harnesses remain limited by superficial aggregate-score reactions, naive domain routing, underused research budgets, and primitive planner, context, and verifier components.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.
1 INTRODUCTION
Evo-Bench is introduced to measure intrinsic harness-evolving capability rather than static task-solving or base-model strength. Its design addresses harness sensitivity, cross-split generalization, and long-horizon iterative refinement, while evaluations reveal substantial but domain-dependent gains and early saturation.
- Benchmark motivation and contribution: Three challenges motivate the benchmark: harness sensitivity, cross-split generalization, and sustained multi-round evolution.These requirements target responsiveness to framework improvements, protection against task-specific overfitting, and iterative diagnosis-driven code updates.
- Benchmark motivation and contribution: Evo-Bench evaluates models’ intrinsic ability to autonomously refine executable agent harnesses across Search, Office, and General domains.The benchmark uses a fixed policy model, five established benchmarks, and controlled long-horizon evolution.
- Benchmark motivation and contribution: The benchmark construction framework selects harness-sensitive tasks and applies sensitivity-aware stratified splitting to produce aligned, disjoint validation and evaluation suites.Auxiliary-task evolution first identifies diverse harnesses, after which task selection prioritizes responsiveness while preserving difficulty diversity.
- Main findings: Across nine frontier and open-weight models, top evolvers achieve gains reaching 16.6 points over the seed harness and approach the human-engineered baseline.GPT-5.6 Sol and Claude Opus 4.8 are reported as leading evolvers, with gains of 16.6 and 16.1, respectively, against a 47.5 baseline.
- Main findings: Evolution shows early saturation, strong Search performance, superiority over artificial harnesses in General tasks, and difficulty with workflow-specific Office tasks.Later iterations can introduce detrimental modifications after high-quality structures are discovered.
- Main findings: Synthesized harnesses transfer across diverse policy models, consistently improving policies from Qwen, DeepSeek, and GLM.The paper characterizes evolved harnesses as transferable reasoning structures rather than narrowly task-specific modifications.
2 RELATED WORK
Related work spans automated harness engineering and agent benchmarking, while Evo-Bench adopts an orthogonal evaluation perspective by fixing the policy model and research protocol. The benchmark targets intrinsic long-horizon harness-engineering capacity rather than static task execution or model-weight optimization.
- Automated Harness Engineering: Automated agent improvement has progressed from prompt tuning to workflow optimization and executable harness refinement.Harness-level methods modify code orchestrating reasoning, tool use, and memory, expanding beyond constrained prompt- or workflow-level design spaces.
- Agent Benchmarks: Figure 1 organizes Evo-Bench across three domains and compares its validation and evaluation splits.The figure summarizes domain hierarchy, source-benchmark allocation, and split composition.
- Automated Harness Engineering: Evo-Bench fixes the policy model and research protocol to benchmark frontier models’ intrinsic capacity to act as long-horizon harness engineers.This is presented as complementary to proposing a new optimization algorithm.
- Agent Benchmarks: Existing agent benchmarks predominantly evaluate task execution within static harnesses, while AI-R&D benchmarks assess model-weight or training-pipeline optimization.The related work distinguishes these settings from harness evolution.
3 EVO-BENCH
Evo-Bench frames harness evolution as long-horizon, code-centric improvement of an executable agent harness under a fixed policy model. Runs iterate on validation evidence before freezing a revision for disjoint evaluation, with complementary metrics for final generalization and evolutionary progress.
- Definition and task formulation: Harness-evolving capability is defined as long-horizon, code-centric iterative improvement of an executable agent harness.Models diagnose failure patterns, formulate hypotheses, and revise implementation to achieve sustained gains.
- Definition and task formulation: Each run pairs a fixed policy model with an evolver that edits the policy harness using accumulated validation-side evidence under a bounded resource budget.The final revision is frozen and evaluated on a disjoint held-out suite.
- Task domains and composition: Evo-Bench covers Search, Office, and General agent domains using five established benchmarks and a 160-task visible validation suite.The source benchmarks are BrowseComp and HLE for Search, GDPval and APEX-Agents for Office, and Claw-Eval for General tasks.
- Harnesses and evaluation: The evolving policy harness starts as a minimal CodeAct loop and is optimized to improve performance across all three domains.Its initial tools are shell execution and final-answer completion, while policy execution occurs in a separate sandbox.
- Evaluation metrics: Aggregate score measures policy performance with a harness across target domains, while Overall Score evaluates the final evolved harness on unseen tasks.These metrics use native source-benchmark scoring aggregated across target domains.
- Evaluation metrics: Anytime Validation Score averages the best-so-far validation performance over the evolution budget, carrying forward the final best score after early termination.This metric captures progress throughout evolution rather than only the final revision.
- Task selection: Table 1 summarizes harness-guided task selection, filtering non-positive sensitivity before stratified selection by difficulty.Sens(x) is Pearson correlation with harness quality, while Perf(x) is average performance across auxiliary harnesses.
4 BENCHMARK CONSTRUCTION
Evo-Bench constructs benchmark suites by evolving diverse auxiliary harnesses, measuring task responsiveness to harness quality, and using sensitivity-aware stratification for suite selection.
- Two-Stage Framework: The two-stage framework first evolves auxiliary harnesses, then uses them to characterize task sensitivity and difficulty for constructing benchmark suites.The stages are designed to identify tasks that respond reliably to harness improvements.
- Stage 1: Auxiliary Harness Generation: Four frontier models produce 73 evaluated harness variants, from which 12 representative harnesses are selected for diversity across capabilities, tool orchestration, structure, and performance.The selected set is denoted Haux = {h1, . . . , hK}, with K = 12.
- Stage 2: Harness-Guided Task Selection: The framework evaluates 2,329 candidate tasks with the 12 selected harnesses, excluding multimodal tasks and downsampling two benchmarks to control cost.The candidate tasks come from APEX-Agents, BrowseComp, Claw-Eval, GDPval, and HLE.
- Stage 2: Harness-Guided Task Selection: Task selection combines correlation with overall harness quality and average task score, rather than relying only on score variance across harnesses.This distinguishes consistent responsiveness to harness quality from large but potentially uninformative performance differences.
- Stage 2: Harness-Guided Task Selection: Harness sensitivity is the Pearson correlation between task-level scores and leave-one-task-out overall harness quality, while task difficulty is defined as 1−Perf(x).Lower average performance indicates greater performance headroom.
- Stage 2: Harness-Guided Task Selection: The final construction removes tasks with non-positive sensitivity, selects highly sensitive tasks within difficulty strata, and randomly splits tasks within each stratum.This preserves difficulty diversity while prioritizing tasks responsive to harness evolution.
5 EXPERIMENTS
Evo-Bench evaluates autonomous harness evolution under controlled budgets against CodeAct and human-engineered baselines. Results show strong but uneven gains: evolution is most effective for Search and General tasks, while Office workflows remain difficult.
- Experimental Setup: The experiments evaluate seven frontier and two open-weight models using a fixed policy model, shared seed harness, and controlled evolution protocol.The main setup uses DeepSeek-V4-Flash as policy model, CodeAct as the seed harness, and common budgets of 20 iterations, 1,000 steps, and 48 hours.
- Overall Performance: GPT-5.6-Sol and Claude Opus-4.8 lead with scores of 46.3 and 45.8, while the domain-specific Artificial harness scores 47.5.The strongest evolved harnesses closely approach, but do not exceed, the human-engineered composite baseline overall.
- Per-Domain Performance: Claude Opus-4.8 gains +34.8 on Search, whereas Office models show marginal improvements or regressions and General evolvers surpass the Artificial harness.The domain pattern indicates that web-navigation logic is easier to synthesize than specialized Office workflows.
- Budget Use: GPT-5.6-Sol and Kimi-K2.7-Code exhaust the 20-iteration budget, but Qwen3.7-Max achieves a highly competitive score after stopping at 15 iterations and about 200 steps.Budget exhaustion does not strictly determine success; models exhibit both exhaustive exploration and early saturation.
- Cost Analysis: The cost-performance frontier places GPT-5.6-Sol above $500 per run, while GLM-5.2 and Qwen3.7-Max deliver strong performance for under $40.DeepSeek-V4-Pro synthesizes functional improvements for less than one dollar, illustrating a broad cost-performance trade-off.
- Case Study: GPT-5.6-Sol evolves a hierarchical domain router, specialized Search and Office tools, failure recovery, and safeguards for General tasks.Its Search changes include web-search and fetch tools plus an HTML cleaner that removes noise while preserving links.
- Limitations: The evolved harness remains limited by superficial score-based diagnosis, naive domain routing, underused research budgets, and primitive planner, context, and verifier components.These limitations leave the overall harness behind Artificial baselines on Search and Office.
6 ABLATION STUDIES
The ablations test whether more evolution resources improve performance and whether evolved harnesses transfer across policy models. Both analyses support positive scaling and strong cross-policy robustness.
- Resource Budget Scaling: Expanding the budget from 24 to 48 hours produces consistent, monotonic improvements in Overall Score and Anytime Validation for Qwen3.7-Max and GLM-5.2.The compared settings range from 24 hours, 10 iterations, and 500 steps to 48 hours, 20 iterations, and 1,000 steps.
- Resource Budget Scaling: GLM-5.2 improves sharply through the 36-hour setting before plateauing, while Qwen3.7-Max grows more steadily across all tested constraints.The two evolvers therefore exhibit different performance trajectories under expanded computational budgets.
- Policy Model Robustness: Harness evolution remains robust when the policy model changes from DeepSeek-V4-Flash to Qwen3.6-35B-A3B or GLM-5.2.The re-evaluation uses Qwen3.7-Max and GLM-5.2 as evolvers.
- Policy Model Robustness: With Qwen3.6-35B-A3B as policy, evolvers raise Overall Score from 13.9 to 27.9 and 29.2; with GLM-5.2 as policy, GLM-5.2 raises 38.0 to 48.4.These gains support transfer of evolved reasoning and tool-use structures across policy models rather than dependence on one model’s flaws.
7 CONCLUSION
Evo-Bench evaluates whether language models can autonomously improve agent harnesses while separating genuine evolution from prompt tuning and runtime noise. The benchmark is positioned as a foundation for future self-evolving agent research.
- Conclusion: Evo-Bench uses controlled attribution, long-horizon iteration, and transfer alignment to evaluate intrinsic harness-evolving capability.These design principles aim to disentangle genuine harness improvement from trivial prompt tuning and stochastic runtime noise.
- Conclusion: The benchmark’s evaluation framework reveals both the potential and current limitations of models acting as autonomous research engineers.
- Conclusion: Evo-Bench is intended as a foundational testbed for self-evolving agent architectures and execution-grounded reasoning.
- Conclusion: The effect-of-evolution-budget analysis is presented as part of the benchmark’s evaluation evidence.
8 FUTURE WORK
Future development includes maintaining Evo-Bench as a living benchmark while supporting broader tasks, agent frameworks, and model populations. Its implementation provides a fixed evolution harness, controlled policy execution, research tools, evaluation interfaces, and sandboxed snapshots.
- Future Work: Evo-Bench is planned as a living benchmark for measuring harness evolution as an initial, practically measurable form of AI self-evolution.
- Future Work: Future extensions will add coding and scientific research tasks, support more agent frameworks, and evaluate a wider range of models.
- Implementation: The fixed evolve harness gives every evolver the same prompt, tools, state management, and resource accounting while allowing policy-harness revision.
- Implementation: The orchestrator enforces a diagnose–hypothesize–edit–evaluate cycle with live budget tracking and experiment history.
- Implementation: Engineering and research tools support code inspection, external research, rollout filtering, task comparison, multi-evaluation analysis, and hypothesis tracking.
- Implementation: Evaluation runs consume one iteration on an immutable snapshot and return headline, comparative, domain-level, task-level, and failure-class results.
- Implementation: Policy rollouts use read-only snapshots and isolated workspaces, while held-out evaluation begins only after the selected revision is frozen.
- Implementation: All runs begin with the CodeAct seed harness, whose shell-command loop leaves planning, memory, domain tools, and verification open to evolution.
A.3 BENCHMARK DIFFICULTY AND HARNESS SENSITIVITY
Benchmark selection combines harness sensitivity with task difficulty and preserves aligned validation and evaluation splits. The construction process is organized around a selected auxiliary suite and subsequent harness-guided selection details.
- Benchmark Difficulty and Harness Sensitivity: Harness sensitivity is defined as Sens(x), while task difficulty is defined as 1 −Perf(x), with Perf(x) averaged across twelve auxiliary harnesses.
- Benchmark Difficulty and Harness Sensitivity: Selection favors non-negative sensitivity while preserving difficulty coverage and alignment between the two benchmark splits.
- Benchmark Difficulty and Harness Sensitivity: BrowseComp, HLE, and APEX-Agents skew hard, GDPval is broader, and Claw-Eval is easier; eight slightly negative Claw-Eval tasks backfill sparse strata.
- Benchmark Difficulty and Harness Sensitivity: Figure 9 encodes task selection with colors for unselected, validation, and evaluation tasks, marker size for coordinate multiplicity, and shading for negative sensitivity.
- Benchmark Difficulty and Harness Sensitivity: Stage 1 first selects a 320-task auxiliary suite, then uses it to generate and select representative auxiliary harnesses.
B.1 SELECTION OF AUXILIARY TASKS
Auxiliary-task selection builds a balanced suite across Search, Office, and General domains by combining rollout-based score and interaction-horizon estimates with stratified quotas. The selected tasks then support diverse harness generation and representative revision selection.
- Selection of Auxiliary Tasks: 11,322 candidates remain after validity, rollout-coverage, deduplication, and leakage checks: 2,988 Search, 7,564 Office, and 770 General.
- Selection of Auxiliary Tasks: Rollout characterization estimates each task’s mean score s(x) and turn count ℓ(x) using domain-native workflows, independently of the CodeAct seed.
- Selection of Auxiliary Tasks: The headroom–horizon procedure selects 128 Search, 128 Office, and 64 General tasks across six score bands, favoring lower scores and longer interactions.
- Harness Generation: Four evolvers independently revise the same seed harness with a fixed policy model and shared 20-iteration, 1,000-step, 48-hour budget, producing 73 evaluated harnesses.
- Harness Selection: Content hashing reduces the 73 evaluated harnesses to 65 unique revisions before capability-based comparison.
- Harness Selection: A deterministic k-medoids procedure selects three harnesses per evolution while retaining final revisions and covering distinct capability levels and designs.
C EXPERIMENT DETAILS
The experiments specify model settings, integrity controls, trajectory audits, and revision-selection procedures for evaluating harness evolution. They also document a severe Qwen regression caused by a malformed verifier and weak recovery practices.
- Model configurations: Seven frontier and two open-weight evolvers were evaluated with fixed policy and judge settings, while model configurations were documented in Table 5.Requests allowed 65,536 output tokens with 600-second timeouts; endpoint-specific settings varied.
- Integrity controls: The protocol audited sandbox leakage, benchmark-answer retrieval, and attempts to evade retrieval safeguards.Controls included strict write separation, scanning tool outputs, and semantic audits of complete trajectories.
- Trajectory auditing: Figure 11 tracks validation Overall Score trajectories, intervention surfaces, best and frozen revisions, catastrophic revisions, and Kimi’s restoration of I13 after I20.Table 6 similarly records research-budget use and revision selection.
- Trajectory findings: Qwen rose from 33.0 to 49.7 at I10, but a malformed I15 verifier crashed 159/160 tasks and reduced the score to 0.3.Repairs recovered only to 45.4 at I18.
- Trajectory findings: A 2.2-point Overall Score range across byte-identical Qwen revisions exposed weak replication, missing smoke checks, bundled edits, and poor best-snapshot management.Qwen froze I18 while leaving two iterations and 651 steps unused.
D.2 DEEPSEEK-V4-PRO: PREMATURE PLATEAU AND PROCEDURAL EVALUATIONS
DeepSeek reached an early high score after adding broad capabilities, then relied on procedural evaluations and stopped without causal localization. The broader prompt and protocol required falsifiable, evidence-driven iteration, but later revisions did not follow that standard.
- DeepSeek trajectory: DeepSeek reached 46.5 at I3 after adding search, fetch, file/Python tools, URL filters, and domain guidance.Later scores fluctuated from 35.9 to 43.0 while raw rollouts were accessed only four times.
- DeepSeek trajectory: Requiring three searches at I11 reduced the score to 42.5 and tripled runtime, while I12 reflection reached 40.6.These changes illustrate costly procedural modifications without sustained gains.
- Failure analysis: DeepSeek labeled I3 an outlier without reevaluating it, froze I15, and left five iterations and 798 steps unused.The analysis characterizes this as aggregate hill climbing followed by premature plateau declaration.
- Evolver protocol: The evolver prompt required inspecting failures, stating a falsifiable mechanism, locally testing edits, evaluating immutable snapshots, and retaining or reverting from evidence.It also instructed evolvers to use validation diagnostically while preserving general mechanisms.
- Evaluation setup: The policy began as a minimal CodeAct interaction with workspace boundaries, while evolved prompts and code were frozen together and evaluation-side judges remained fixed.Policies did not receive references, rubrics, or judge instructions.