Source-linked AI summary
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang, Hanqing Lu
TL;DR
It remains unclear whether base task-solving capability predicts which models can improve external harnesses and which can benefit from those updates. Across seven LLMs and three benchmarks, the paper finds harness-updating largely flat across capability tiers, while harness-benefit is non-monotonic, peaking in the middle tier.
Problem
End-to-end evaluations do not disentangle evolver update quality from task-solving agents’ use of updated harnesses, leaving which models update or benefit best unclear.
Method
The study analyzes harness-updating and harness-benefit across seven LLMs and three benchmarks, pairing models as evolvers and task-solving agents.
Results
Harness-updating is flat across capability tiers, while harness-benefit is non-monotonic, with mid-tier models benefiting most and weak-tier models least.
Takeaways & Limitations
The findings support allocating capability budget to task-solving agents rather than evolers and training agents to invoke and follow harness guidance.
Takeaways & Limitations
The study evaluates fixed-weight harness adaptation with a representative but non-exhaustive model set, excluding parametric and hybrid adaptation methods.
Abstract
from arXiv · showhide
LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.
1 Introduction
The paper separates harness-updating from harness-benefit to explain how models contribute to self-evolving agent systems. Across three benchmarks and seven LLMs, harness-updating is flat across capability tiers, whereas harness-benefit is non-monotonic, peaking in the mid tier.
- Motivation: External harnesses—including prompts, skills, memories, and tools—shape agent behavior in environments involving tool use, software interfaces, and long-horizon tasks.Harnesses are commonly designed manually but can be updated automatically from failures, feedback, trajectories, and successful procedures.
- Findings: Harness-benefit is non-monotonic: mid-tier models benefit most, strong-tier models benefit less after reaching the performance ceiling, and weak-tier models benefit least.GPT-OSS-120B exemplifies the mid-tier peak, while Claude Opus 4.6 and Qwen3-32B exemplify the strong- and weak-tier patterns.
- Problem and approach: End-to-end evaluations cannot distinguish gains from higher-quality evolver updates from gains produced by better task-solving-agent use of updated harnesses.The study therefore defines harness-updating as producing useful updates and harness-benefit as benefiting from them during task solving.
- Findings: Harness-updating is flat across capability tiers: different evolvers produce similarly beneficial updates, with no evolver dominating across all substrates.Qwen3.5-9B can produce updates whose downstream gains match Claude Opus 4.6 despite a large base-capability gap.
- Design guidance: The harness-updating gap across evolvers is at most 3.1 percentage points on any benchmark, while weak models often fail to invoke harnesses or adhere to them over long trajectories.Weak-tier adherence decays over four times more steeply than strong models, motivating investment in the task-solving agent, harness invocation, and long-horizon instruction following.
2 Related Work
Prior harness-engineering evaluations typically combine one update procedure with one target agent, conflating base capability, harness-updating, and harness-benefit. This work separates these factors by varying task-solving agents and evolvers independently and measuring the two evolution capabilities separately.
- Harness engineering: LLM agents combine a frozen backbone with an external harness mediating reasoning, tool use, memory access, and environment interaction.
- Harness engineering: Prior evaluations typically report the end-to-end gain of one update procedure paired with one target agent on one substrate.
- Harness engineering: Such scores conflate improvement from the agent’s base capability, harness-updating, and harness-benefit.
- Controlled analysis: This work varies task-solving agents and evolvers independently, measures harness-updating and harness-benefit separately, and tests whether either tracks base capability.
3 Harness-Evolution Capabilities
This section formalizes harness self-evolution as iterative updates to an agent’s external harness while keeping the model backbone fixed. It defines harness-updating and harness-benefit as distinct capabilities measured through evolution gains.
- Protocol: Harness self-evolution updates editable external harness components around a fixed model using execution evidence collected during task solving.Editable components include prompts, skills, and memories, while tool interfaces and execution policies remain fixed.
- Protocol: The protocol iterates for T steps: the agent solves task batches, collects execution trajectories and outputs, and an evolver produces the next harness.Starting from H0, each update yields agent At = (f, Ht), with the final harness denoted HT.
- Metrics: Base capability is the initial agent’s task-solving performance on the task set before harness evolution.The initial agent is A0 = (f, H0), evaluated by the scoring function JX(f, H).
- Metrics: Harness-updating capability is an evolver’s mean pairwise evolution gain across an anchor agent set.Evolution gain measures improvement over the paired agent’s task-solving performance before evolution.
- Metrics: Harness-benefit capability is a model’s maximum pairwise evolution gain across a fixed anchor evolver set.This metric captures how much the model benefits from harness self-evolution.
4 Experiments
Experiments show that harness-updating varies little across evolvers, whereas harness-benefit is non-monotonic across task-solving models. Post-evolution performance is therefore driven more by the agent’s base capability, with weak-tier failures arising from harness activation and adherence problems.
- Harness-updating: On SkillsBench, an Opus 4.6 agent scores 1.0 with either a Qwen3.5-9B- or Opus 4.6-evolved skill, versus 0.67 without evolution.Both evolved skills encode procedurally similar guidance and enable the same agent to solve the task.
- Harness-updating: ∆update varies by at most 3.1 pp across evolvers on any benchmark, indicating relatively flat harness-updating capability.Qwen3.5-9B achieves 3.8 pp on SkillsBench, exceeding Opus 4.6 at 2.3 pp and Qwen3-235B at 1.5 pp.
- Agent-side performance: Strong agents lead weak agents by 18.6 to 35.2 pp even under the strongest-versus-weakest evolver pairing, so post-evolution performance is bottlenecked on the agent side.The same pattern persists on SWE and SkillsBench.
- Harness-benefit: Harness-benefit is non-monotonic: SWE gains peak at Qwen3-235B with 19.3 pp, while Qwen3-32B gains 4.4 pp and Opus 4.6 gains 2.6 pp.On MCP, the peak shifts to GPT-OSS-120B at 7.0 pp, with lower gains at both ends.
- Failure modes and training implications: Weak-tier models often fail either to activate relevant harness artifacts or to follow loaded guidance faithfully.Skill-load rate is 0.251 for Qwen3-32B versus 0.957–0.961 for stronger models; HFR is 0.142 for Qwen3-32B versus 0.757 for Opus 4.6, and Qwen3-235B has SLR 0.961 but HFR 0.350.
- Failure modes and training implications: Agent training should target harness invocation and sustained long-horizon instruction following, since Qwen3-32B’s adherence drifts from 0.52 to 0.13 after harness loading.The recommended invocation target is reflected in its 25.1% skill-load rate versus approximately 96% for strong-tier models.
5 Conclusion
The conclusion distinguishes harness-updating from harness-benefit as two capabilities separate from base capability. Across seven LLMs and three benchmarks, harness-updating is flat across capability tiers, while harness-benefit is non-monotonic in base capability.
- Capability decomposition: Harness self-evolution is decomposed into harness-updating and harness-benefit, both distinct from base capability.Harness-updating produces harness updates; harness-benefit concerns benefiting from updated harnesses during task solving.
- Harness-updating: Across seven LLMs and three benchmarks, harness-updating is flat across base-capability tiers.Models from different capability tiers produce updates yielding similar gains.
- Harness-updating: Even the Qwen3.5-9B evolver induces gains comparable to Claude Opus 4.6.This comparison illustrates the flatness of harness-updating across capability tiers.
- Harness-benefit: In contrast, harness-benefit is non-monotonic in base capability.The conclusion contrasts this pattern with harness-updating’s flat relationship to base capability.
6 Limitations
The study examines harness self-evolution with fixed model weights, excluding parametric and hybrid adaptation methods. Its representative but non-exhaustive model set leaves broader capability-grid analysis for future work.
- Scope of adaptation methods: The study evaluates external harness updates while keeping model weights fixed, rather than parametric fine-tuning or reinforcement learning of model weights.Hybrid methods combining weight updates with harness updates are also not evaluated.
- Model coverage: The model set spans open-source and closed-source models across multiple capability tiers but is not exhaustive.A broader model grid could further clarify harness self-evolution across capabilities.
7 Ethics Statement · A Full Details of Related Works · A.1 Harness Engineering
The paper examines ethical risks of persistent harness self-evolution and situates its analysis within prior work on editable agent harnesses. It complements existing harness representations by studying models’ abilities to update harness artifacts and benefit from those updates.
- 7 Ethics Statement: Experiments use benchmark tasks and do not collect or process private user data.
- 7 Ethics Statement: Persistent harness updates may carry incorrect lessons, unsafe tool-use rules, biased instructions, or sensitive information into future tasks.
- A Full Details of Related Works: The related-works appendix provides the full version of the related works discussed in Section 2.
- A.1 Harness Engineering: LLM agents are framed as compound systems in which a frozen model is surrounded by external artifacts that shape reasoning, tool use, memory access, skill invocation, and environment interaction.
- A.1 Harness Engineering: The paper calls this external layer the agent harness.
- A.1 Harness Engineering: Prior work studies harness artifacts including prompts that encode behavioral rules, task policies, and reasoning procedures.
- A.1 Harness Engineering: Existing studies establish harnesses as editable agent state rather than passive context.
- A.1 Harness Engineering: This work is complementary because it analyzes capabilities for updating harness artifacts and benefiting from the resulting updates instead of proposing a new harness representation.
A.2 Self Evolution of LLM agents … B.3 Metrics
The paper distinguishes harness-updating from harness-benefit and evaluates self-evolving agents across complementary benchmarks, model families, and pass-rate-based metrics in an in-situ setting. This setup separates persistent harness evolution from task-solving performance while preventing a task’s own update from influencing its score.
- A.2 Self Evolution of LLM agents: Early self-evolution methods store textual reflections or lessons, whereas newer methods evolve persistent harness components such as prompts, playbooks, and memories.These approaches show that writing execution experience into persistent harness components can improve downstream task performance, but single-agent evaluations conflate base capability with harness-updating and harness-benefit.
- B.1 Dataset Details: The evaluation covers long-horizon code repair, multi-server tool orchestration, and skill-based execution using SWE-bench Verified, MCP-Atlas, and SkillsBench.SWE-bench Verified contains 500 tasks across 12 Python repositories; MCP-Atlas requires 3–6 tool calls across 36 servers exposing 220 tools; SkillsBench contains 86 tasks across 11 domains.
- B.1 Dataset Details: SkillsBench uses an empty skill set for the no-evolution baseline, while evolved cells use only skills produced by the evolver from earlier in-situ tasks.The native benchmark includes curated skills, but those skills are not used in the no-evolution baseline.
- B.2 Models: The study uses seven open- and closed-source LLM backbones spanning capability tiers, with six models used as task-solving agent backbones.The models include Claude Opus 4.6, Sonnet 4.6, and Haiku 4.5; Qwen3-235B-A22B, Qwen3-32B, and Qwen3.5-9B; and GPT-OSS-120B.
- B.3 Metrics: All four metrics use pass rate as the scoring function, with pass rates and average scores reported in percent and gains reported in percentage points.The benchmark-specific grading procedures determine the per-task scores averaged by JX.
- B.3 Metrics: SWE-bench Verified uses a binary resolved score, MCP-Atlas reports strict pass rate and average claim-fulfillment score, and SkillsBench reports an average score across tasks and trials.MCP-Atlas claim fulfillment is continuous in [0, 1], while SkillsBench averages binary per-task scores over 5 trials.
- B.3 Metrics: In-situ evaluation uses the same task stream for evolution and evaluation, scoring each task under the pre-update harness before its execution evidence produces the next harness.Thus, an individual task’s pass rate cannot be influenced by harness updates derived from that task itself.
B.4 Implementation Details · C Evolver-side Analysis Details in Sec. 4.2 · C.1 Additional Results for Observation 1
The implementation fixes the evolver prompt format across benchmarks and model backbones while varying writable scope and execution evidence. Additional results report per-cell pass rates for three anchor agents under each evolver alongside the resulting harness-updating scores.
- B.4 Implementation Details: B.4 limits edits to skills directories for SWE-bench Verified and SkillsBench, while MCP-Atlas also permits prompt and append-only memory updates.The tools/ directory and evaluation files remain read-only across all benchmarks.
- B.4 Implementation Details: B.4 passes benchmark-specific writable permissions to the evolver at every cycle, while keeping the evolver system prompt fixed.This applies across benchmarks and model backbones.
- B.4 Implementation Details: B.4 keeps the prompt format fixed across benchmarks and model backbones, varying only task evidence and benchmark-specific writable scope.The execution-evidence payload is canonicalized before being supplied to the evolver.
C.2 More Details of the Case Study
In the flink-query case study, both Qwen3.5-9B and Opus 4.6 evolvers transform the same Opus 4.6 agent from a 0.67 score to 1.0 by producing skills with the same five problem-solving steps. Their implementation details differ, but both achieve identical downstream pass rates when injected at turn 0.
- Case-study comparison: Without an evolver, the Opus 4.6 agent omits the FINISH-event filter and scores 0.67; either evolver raises its score to 1.0.The comparison uses no evolver, Qwen3.5-9B as evolver, and Opus 4.6 as evolver, with the evolved skill injected at turn 0.
- Shared skill content: Both evolved skills encode five steps: filtering SUBMIT and FINISH events, counting SUBMITs separately, emitting (jobId, count), and applying a 10-minute session window.These steps specify the shared problem-solving procedure learned from the trajectories.
- Implementation differences: The skills differ only in implementation: Qwen3.5-9B uses manual batch sessionization, whereas Opus 4.6 uses a KeyedProcessFunction for the same 10-minute window.The differing implementation surfaces do not change the encoded procedure.
- Downstream outcome: Both skills yield identical downstream pass rates of 1.0 when injected into the same Opus 4.6 agent.This result holds despite the evolvers’ different implementation choices.
C.3 Additional Results for Observation 2 · D Agent-side Analysis Details in Sec. 4.3 · D.1 Case Studies for the Two Agent-Side Failure Modes
Additional benchmarks show that differences between task-solving agents dominate variation across evolvers, even under unfavorable pairings. Agent-side failures arise when Qwen3-32B cannot express a correct skill-loading intent in the required action format or cannot adaptively follow a loaded fallback procedure after an error.
- C.3 Additional Results for Observation 2: Across SWE-bench Verified and SkillsBench, within-agent variation across evolvers remains smaller than between-agent differences in base capability.The same pattern extends Observation 2 beyond the original benchmark, including extreme agent–evolver pairings that still favor stronger agents.
- C.3 Additional Results for Observation 2: 53.5 pp and 20.9 pp are the base-capability gaps between Opus 4.6 and Qwen3-235B on SWE and SB, respectively.These gaps exceed the largest within-agent spreads of 5.0 pp on SWE and 9.3 pp on SB by factors of 11 and 2.2.
- C.3 Additional Results for Observation 2: 5.0 pp and 9.3 pp are the largest within-agent spreads across seven evolvers on SWE and SB, attained by Qwen3-235B and Sonnet 4.6, respectively.Sonnet 4.6’s evolved SB scores range from 22.1% to 31.4%.
- C.3 Additional Results for Observation 2: 18.6 to 35.2 pp is the strong agent’s advantage over the weak agent across every benchmark, even when the strong agent receives its worst-performing evolver.The comparison pairs the weakest anchor agent with its best evolver against the strongest anchor agent with its worst evolver.
- D.1 Case Studies for the Two Agent-Side Failure Modes: Qwen3-32B’s activation failure occurred because it identified the relevant threejs skill but bundled load_skill with analysis and plan in a multi-key JSON action.The SkillsBench format gate accepts only single-key actions, so the agent’s intent could not be translated into the runner’s expected action protocol.
- D.1 Case Studies for the Two Agent-Side Failure Modes: Qwen3-32B’s adherence failure occurred after it loaded the pg-essay-to-audiobook skill but treated its TTS fallback chain as a literal script rather than a contingent procedure.The first prescribed step produced a FileNotFoundError on turn 1, after which the agent continued through subsequent steps.
D.2 More results of ∆benefit in Sec. 4.3 … E Information about AI Assistants
Additional results show that harness benefit remains non-monotonic on MCP but is noisier on SkillsBench, while supplementary sections define blinded judge-based adherence measures and document limited AI-assistant use for writing and formatting.
- D.2 More results of ∆benefit in Sec. 4.3: The full pass-rate matrix reports no-evolution baselines, pass rates under Opus 4.6, Sonnet 4.6, and Qwen3-235B, and ∆benefit as the maximum gain over NONE.These values cover each benchmark and task-solving model.
- D.2 More results of ∆benefit in Sec. 4.3: 7.0 pp is the MCP-Atlas ∆benefit peak for GPT-OSS-120B at a 28.0% base pass rate, with gains decreasing toward weaker and stronger models.The MCP pattern mirrors SWE but has a smaller gain range.
- D.2 More results of ∆benefit in Sec. 4.3: 15.1 pp is Haiku 4.5’s largest SkillsBench gain, whereas Qwen3-235B gains only 1.1 pp despite a similarly low base rate.Qwen3-32B and GPT-OSS-120B start at 0.0%, Qwen3-235B at 4.7%, and Haiku 4.5 at 5.8%; the low-base regime is more variable across domains.
- D.3 Judge Details for Harness-Following Rate: Harness-Following Rate judges whether an agent follows a loaded skill by converting the skill body into atomic procedural instructions and checking the blinded trajectory against them.Claude Sonnet 4.6 serves as judge, and model identifiers are replaced with <MODEL>.
- D.3 Judge Details for Harness-Following Rate: HFR is computed from skill-loaded trajectories and the subset judged as following the loaded skill.The notation distinguishes Nload_f, the number of skill-loaded trajectories for model f, from Nfollow_f, the judged-following subset.
- D.4 Judge Details for Phase-Level Adherence Score: Phase-level adherence analysis separately scores three execution phases—harness loaded, mid turn, and final turn—on a 0–1 scale using a fixed rubric and blinded trajectory.The analysis is intended to measure adherence drift over long-horizon execution and is reported separately from HFR.
- E Information about AI Assistants: Supplementary tables provide task-solving seed prompts for SWE-bench Verified and MCP-Atlas, fixed evolver prompts and user-message templates, and judge prompts for HFR and phase-level adherence.The evolver system prompt is constant across backbones and benchmarks, while writable artifacts depend on benchmark-specific permissions.
- E Information about AI Assistants: The paper used GPT-5.5 as a writing and formatting assistant for grammar, clarity, captions, and layout, not for research ideation, experimental design, implementation, data analysis, or technical content.The authors reviewed and edited all outputs and take responsibility for the final text and visuals.