Source-linked AI summary
EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, Xuan Phi Nguyen, Semih Yavuz, Mohit Bansal, Shafiq Joty
TL;DR
EVOHARNESSBENCH addresses the limited evaluation of agents when their externally supplied harness evolves rather than remaining fixed. It constructs controlled multi-axis harness streams and separates deployment retention from self-evolving adaptation, finding forgetting, inconsistent adaptation gains, and a retention–adaptation tension.
Problem
Existing agent benchmarks typically keep the harness fixed or omit outer harness evolution, leaving performance responses to changing externally supplied capabilities unevaluated.
Method
EVOHARNESSBENCH evaluates fixed underlying models across cumulative tool, skill, and agent harness streams using deployment and self-evolving adaptation modes.
Results
Harness expansion can cause forgetting, adaptation gains vary across stages, axes, and environments, and retention improvements can conflict with adaptation to newly introduced capabilities.
Takeaways & Limitations
Harness evolution is a distinct challenge requiring agents to preserve earlier competence while keeping pace with newly introduced capabilities.
Takeaways & Limitations
The benchmark’s skill-task association uses a keyword-based heuristic that does not guarantee matched skills are necessary, sufficient, or unique for solving tasks.
Abstract
from arXiv · showhide
Modern LLM-based agents operate through a harness of tools, reusable skills, and specialist agents that shapes what they observe and what they can do. In practice, this harness continually evolves as new capabilities are added. We introduce EVOHARNESSBENCH, a benchmark for evaluating agents under controlled harness evolution across three axes (tools, skills, and agents). Unlike existing continual-learning benchmarks for agents, which typically place non-stationarity (i.e., what changes over time) in the task stream while keeping the harness fixed, EVOHARNESSBENCH places non-stationarity in the externally supplied harness itself. It contains 17 multi-stage harness streams constructed deterministically from verifier-based benchmarks, comprising 802 tasks, 520 tools, 42 skills, and 62 agents. We evaluate two complementary settings corresponding to the central challenges of harness evolution: deployment evaluation, which isolates retention of previously accessible competence as the harness expands, and self-evolving adaptation evaluation, which tests whether accumulated experience remains useful as new capabilities are introduced. Our results reveal three persistent gaps. First, harness expansion alone can degrade performance on previously solved tasks, producing harness-induced forgetting. Second, gains from self-evolving adaptation remain inconsistent across stages of harness evolution, capability axes, and environments. Third, retention and adaptation can pull in different directions: preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and vice versa. These results establish harness evolution as a distinct challenge for building agents that can keep pace with an evolving harness while preserving previously effective behavior.
1 Introduction
EVOHARNESSBENCH frames harness evolution as a distinct source of non-stationarity, testing whether agents retain prior competence and adapt as tools, skills, and agents expand. Its controlled evaluation reveals forgetting, inconsistent adaptation, and tension between retention and adaptation.
- The agent harness includes callable tools, reusable skills, and coordinating agents that reshape what models observe and can do.
- Harness evolution creates retention and adaptation challenges because previously effective behavior can become harder to recover or stale as capabilities expand.The underlying model can remain unchanged while competence degrades through an expanded harness, and earlier adaptive artifacts may misdirect execution.
- EVOHARNESSBENCH constructs nested cumulative harness streams across tools, skills, and agents, assigning tasks to the earliest stage where newly introduced capabilities make them solvable.The benchmark varies the externally supplied harness rather than placing non-stationarity only in the task stream.
- The benchmark separates deployment evaluation from self-evolving adaptation evaluation to isolate retention under expansion and the usefulness of persistent experience across stages.Deployment disables inner adaptation and evaluates each harness version independently, while self-evolving adaptation carries persistent components across stages.
- 17 controlled harness streams span 3 to 6 cumulative stages, with 802 unique tasks evaluated across 520 tools, 42 skills, and 62 specialist agents.The experiments include representative single-agent and multi-agent systems under both evaluation modes.
2 Related Work
Existing agent benchmarks generally study fixed harnesses or self-evolution without outer harness evolution. Consequently, they do not evaluate how task performance responds as the supplied harness changes.
- Existing benchmarks study static harnesses, inner self-evolution without outer evolution, or capability growth generated by the agent itself.
- These benchmark groups do not evaluate how task performance responds to evolution of the externally supplied harness.
3 EVOHARNESSBENCH
EVOHARNESSBENCH constructs deterministic, multi-stage harness streams in which tools, skills, or specialist agents accumulate over time while tasks remain verifier-checkable. It evaluates both direct effects of the expanding harness and whether persistent adaptation helps agents operate within it.
- 3.1 Overview: A harness stream exposes a cumulative capability set H1 ⊆ H2 ⊆ ··· ⊆ HT across discrete stages, with tasks assigned to the earliest stage containing all required capabilities.Capabilities remain available after introduction, and each task satisfies feasibility and new-capability pressure by construction.
- 3.4 Evaluation Protocol: Evaluation exposes the full cumulative harness rather than an oracle-pruned subset, leaving earlier capabilities available as potential affordances or distractors.Deployment uses fresh system instances without persistent state, whereas self-evolving adaptation carries persistent state across stages and may update it using adaptation data.
- 3.3 Axis-Specific Construction: The dataset combines EnterpriseOps-Gym and Agentic Last Exam into 17 harness streams spanning 802 tasks, 520 tools, 42 latent skills, and 62 specialist agents.The streams contain 3–6 stages each and 1,510 axis-specific evaluation examples.
- 3.2 Construction Pipeline: The benchmark converts verifier-based static tasks into evolving tool, skill, and agent streams through deterministic capability annotation, frequency-ranked release, cumulative harness construction, and earliest-stage task assignment.High-frequency capabilities are released earlier than low-frequency capabilities, and the pipeline requires neither LLM-generated tasks nor human verification.
- 3.3 Axis-Specific Construction: Tools use source-benchmark annotations, skills use rule-based procedure mining and keyword matching, and agents group tools by shared entities into specialist pools.Skill annotations are explicitly treated as a heuristic proxy rather than a guaranteed necessary, sufficient, or unique correspondence.
4 Main Results and Analysis
Harness evolution produces axis- and environment-dependent changes in deployment retention, adaptation gains, and transfer. Across tools, skills, and agents, broader exposure and persistent adaptation can improve some outcomes while worsening cost, retention, or new-task performance.
- Evolving Tools: Broader tool exposure improves deployment accuracy but substantially increases token usage, indicating that added affordances require more search and execution.The larger catalog is not merely distracting; agents can benefit from additional tools at higher execution cost.
- Evolving Tools: Self-evolving adaptation improves tool-harness performance in some environments but remains inconsistent across environments and methods.On EOG, MemToolAgent reaches 38.6% pass rate versus 30.2% without adaptation, while most ALE methods remain near or below deployment performance.
- Evolving Tools: Tool-harness adaptation can improve retention while harming newly introduced-task performance, with GEPA and Meta-Harness showing positive BWT but negative ALE FWT.On ALE, their FWT values fall to −28.5% and −11.1%, respectively.
- Evolving Skills: Skill-pool expansion has little aggregate deployment effect and only modest, method-dependent adaptation gains.On EOG, Codex remains at 18.9% pass rate under cumulative skills, while GEPA increases pass rate from 18.9% to 24.1%.
- Evolving Agents: Agent-pool expansion and adaptation are highly environment-dependent, producing both large gains and severe forgetting.On EOG, Meta-Harness reaches 18.5% from an 8.8% deployment baseline, while ALE deployment Codex reaches −34.7% BWT; stable exposure improves Meta-Harness to 20.3%.
5 Conclusion
EVOHARNESSBENCH evaluates agents as externally supplied tools, skills, and agents evolve, separating deployment retention from self-evolving adaptation. Its trajectory-level evaluation reveals retention and adaptation failures that aggregate final-stage evaluation can obscure, motivating methods that manage stale persistent artifacts and future capability replacement.
- EVOHARNESSBENCH evaluates agents under controlled growth of externally supplied tools, skills, and agents while keeping the underlying model fixed.
- Evaluating the full harness trajectory exposes adaptation and retention failures that final-stage aggregate evaluation can obscure.
- The benchmark’s findings motivate bi-level adaptation that balances learning newly introduced capabilities with preserving earlier competence.
- Future work should address stale or harmful persistent artifacts and extend evaluation beyond monotonic growth to capability replacement and retirement.
B EVOHARNESSBENCH Statistics
EVOHARNESSBENCH combines controlled multi-stage harness trajectories with deterministic construction checks across tools, skills, and specialist agents. Its statistics characterize both capability growth and the distractors, coverage, and coordination requirements exposed to evaluated systems.
- Evolving Tools: Tool statistics track newly released and cumulative tools, task demands, distractor exposure, release schedules, and full-catalog evaluation.Evaluated systems receive the cumulative catalog rather than an oracle-pruned tool list.
- Evolving Skills: Skill task coverage is the fraction of seed tasks assigned at least one verifier-grounded essential-skill tag, with reuse frequency driving core-to-long-tail releases.Hidden reference skills are withheld from evaluated systems so the axis tests adaptive skill learning rather than direct curriculum retrieval.
- Evolving Agents: Agent statistics measure releases, cumulative pools, task requirements, multi-agent coordination, and distractor agents across stages.The lead agent receives the full cumulative specialist pool without direct tool access, requiring selection and delegation.
- Construction checks: Construction checks verify valid ownership, nonempty disjoint specialist bundles, annotated-tool coverage, and exposure of the intended cumulative harness.
- Benchmark overview: 17 independent evolution trajectories contain 802 unique tasks, 1,510 axis-specific examples, 520 tools, 42 latent skills, and 62 specialist agents.The trajectories contain 3–6 cumulative versions each and 70 harness snapshots.
C Metrics
The benchmark represents performance as a lower-triangular matrix across harness stages and task cohorts, allowing separate analysis of new-task adaptation, old-task retention, transfer, efficiency, and failure mechanisms.
- Performance matrix: The lower-triangular performance matrix records accuracy for each task cohort under every later harness stage.Rows represent harness stages, columns represent task cohorts, and cohort τ exists only from stage τ onward.
- Adaptation and retention: Diagonal entries measure adaptation to newly introduced tasks under their intended harness, while off-diagonal entries measure retention under an expanded harness.
- Backward transfer: Negative BWT indicates forgetting when final-stage cohort performance falls below performance at introduction.
- Forward transfer: Negative FWT indicates that self-evolution harms performance on newly introduced capabilities.
- Aggregate diagnostics: Operating cost tracks tokens, tool calls, and latency at each stage, while the matrix diagnoses when expansion helps or hurts and whether distraction or stale state is implicated.
D.2 Associating Skills with Tasks
The benchmark deterministically derives task-linked skills and specialist agents from verifier-grounded capability structure, then evaluates memory configurations under cumulative harness exposure. This construction separates reusable procedural content from visible prompts and tests whether different forms of persistent experience remain useful as tools evolve.
- Task-skill association: EOG associates skills with tasks through verifier-checked table, column, and value constraints, while ALE uses fixed lexical patterns and software-anchor overlap.
- Task-skill association: Reference skills with no associated tasks and tasks without skill annotations are filtered before frequency-ranked releases are constructed.
- Specialist-agent construction: Specialist agents are deterministically formed by grouping each capability owner’s tools or software, with the lead agent accessing the environment only through selected specialists.
- Specialist-agent construction: Required agent sets come from capability owners, and cumulative agent pools expose both current-task specialists and previously released agents.
- Memory configurations: Memory variants compare complete trajectories, success-only or failure-only traces, oracle-tool controls, and compact reasoning artifacts with or without tool information.
G Claude Code Results
Claude Code shows axis-dependent effects of cumulative harness exposure, with little aggregate change under evolving skills and modest declines under evolving agents.
- These within-system trends provide a frontier-system reference for axis-dependent harness-expansion effects without supporting direct absolute comparisons with GPT-5 or Codex.
- Claude Code’s EOG pass rate changes from 29.3% to 30.2% under evolving skills, while ALE remains at 13.2%.
- Under evolving agents, cumulative exposure lowers Claude Code’s pass rate from 11.0% to 10.6% on EOG and from 16.9% to 15.3% on ALE.
H.1 Single-Agent Systems
Single-agent analyses show that structured, task-relevant memory can improve adaptation under evolving tools, while additional tool information may instead create interference.
- The analysis varies memory representation, content, and retained experience to test which properties of prior experience support adaptation as the tool catalog expands.
- Memory-based self-evolution accounts for several of the strongest gains under evolving tools, motivating detailed study of experience content and representation.
- Structured memory outperforms raw replay: Raw Trajectory reaches 33.0% pass rate, Reasoning Bank 36.9%, and MemToolAgent 38.6%.
- Adding explicit tool information to Reasoning Bank reduces pass rate from 36.9% to 30.3%, indicating that more interface detail can interfere with adaptation.
- Task-specific adaptation tools raise Raw Memory’s EOG pass rate from 33.0% to 35.3% and reduce adaptation time from 25.7 to 20.5 hours under identical evaluation harnesses.
H.2 Multi-Agent Systems
Multi-agent results show little forgetting under evolving tools, no clear retention advantage from persistent memory, and architecture-dependent effects on search and adaptation.
- MAS backward transfer remains within ±2.3 percentage points across methods, indicating that earlier task performance is largely preserved as tools evolve.
- No memory method exceeds the no-memory AutoGen baseline’s BWT of +2.3%, so persistent memory provides no clear retention advantage for MAS.
- Retention should be assessed jointly with adaptation to newly introduced tasks because small BWT differences do not identify the strongest memory mechanism.
- AutoGen memory sharpens tool selection, whereas G-Memory broadens exploration and lowers precision; LegoMem instead gains through broader execution and fewer failed verifications.
- Memory benefits are less predictable in MAS than SAS: G-Memory reaches 30.1%, while LegoMem reaches 20.6%, compared with MemToolAgent’s 35.1% in SAS.
I.2 Skill generation methods
Skill-generation results indicate that execution experience and iterative feedback improve skill quality, while skill invocation depends strongly on the underlying model and agent.
- Raw Traj updates a shared skill library from compact trajectories and verifier outcomes after each training stage, then carries it forward through the curriculum.
- Skills extracted from raw trajectories reach 20.0% pass rate, exceeding empty skills at 16.9% and zero-shot generation at 17.6%.
- Batch Teacher Feedback reaches 22.1% and Skill Creator 20.9%, making iterative refinement the strongest-performing skill-generation pattern.
- Skill overlap with task-specific libraries is evaluated stage by stage, and downstream performance is related to final-stage overlap using Pass and Score.
- At the final stage, GPT-5.5 Codex invokes 82% of task-specific skills, while memory-based methods invoke only 0.4–2.4% of their cumulative libraries.
J Additional Analysis on Evolving Agents
Additional analysis shows that evolving-agent performance depends on both reaching the right specialists and coordinating their work afterward. Self-evolution improves delegation completeness, but routing quality involves a trade-off between discovering specialists and avoiding distractors.
- 54% of failures involve missed required agents without adaptation, versus 47% for GEPA and 43% for Meta-Harness.Self-evolution reduces routing failures, but does not eliminate them.
- 78–86% of trials still fail their verifier even after every required specialist is reached.This separates delegation from the downstream coordination and execution bottleneck.
- GEPA raises newly required-specialist recall from 71% to 81%, whereas Meta-Harness remains near 81% before and after adaptation.The methods therefore appear to obtain gains through different mechanisms.
- Self-evolution raises required-specialist recall from 79% to 83–89% and exact match from 27% to 33–38%, while precision stays near 90%.Adaptation improves delegation completeness more than selectivity.
- Across MAS methods, selection precision ranges from 88.0% to 94.2% and F1 from 84.9% to 90.2%, yet stronger selection does not consistently improve end-task performance.Selection accuracy alone is not the primary determinant of task success.
- G-Memory combines strongest specialist recall with an 11.8% distractor rate, while LegoMem has the lowest distractor rate at 5.8% but weaker recall.These results expose an accuracy–caution trade-off as the agent pool expands.