PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolving to Match Capabilities: Improving Harness Self-Evolution via Skill Adaptation

Zhihao Yang, Haojin Wang, Juncheng Wu, Kuan Zhang, Yukun Chen, Run Luo, Yaojie Lu, Yuyin Zhou, Lu Wang, Xianpei Han

arXiv:2610.33024v1cs.LGcs.AIcs.CL

TL;DR

Harness self-evolution refines agent guidance from execution experience without updating model parameters, but skills that help stronger models can provide little benefit or even hurt weaker ones. The paper diagnoses weak-executor failures and uses a fixed meta-skill to adapt skills, including within a five-round self-evolution loop. Across seven executors, that loop improves final scores over the same loop without the meta-skill.

  • Problem

    Skills can help stronger models while providing little benefit or even degrading performance for weaker models, despite relevant requirements already appearing in the skills.

    Source: §1 · pp. 1–2

  • Method

    The authors diagnose weak-model failures and distill them into a fixed, task-independent meta-skill for adapting guidance during harness self-evolution.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    After five rounds, all seven executors score 2.0–5.1 points higher with the meta-skill than under the same self-evolution loop without it.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Harness evolution should evaluate benefits by capability tier and provide diagnostic guidance on weak-executor failures, since evidence alone can lead weak rewriters to drift.

    Source: §6 · p. 11

    The evaluation excludes skill retrieval and progressive disclosure, bundles multiple skills, and uses a fixed five-round budget without updating model parameters.

    Source: §6 · p. 11

Abstract

from arXiv · show

Harness self-evolution offers a practical route for LLM agents to improve from execution experience without updating model parameters. Its effectiveness depends on whether models can successfully act on the guidance provided by their harnesses. However, we find that harness benefits are not uniform across model capabilities: skills that improve stronger models can provide little benefit or even degrade performance for weaker ones. Across seven models and three benchmarks, we identify 103 task-skill pairs where skills improve GPT-5.6-luna by 31.1 points but degrade Gemma-4-E4B by 8.8 points. In 86% of failed weak-model runs, the violated requirement is already stated in the skill. This motivates skill adaptation: reformulating existing skills to match the executor's capabilities and make their guidance easier to execute. To achieve this, we diagnose failures in tool execution, context management, instruction following, and verification, and distill these diagnoses into a fixed meta-skill that guides adaptation during harness self-evolution. Guided by the meta-skill, a capable external rewriter, Kimi-K3, adapts skills without execution evidence, improving the three weak executors by 5.1-14.1 points over the original skills. We then incorporate this adaptation into a five-round self-evolution loop, where each model revises its own skill from execution trajectories and verifier feedback. Across seven executors, from Gemma-4-E4B to Kimi-K2.5, the meta-skill consistently improves final scores by 2.0-5.1 points over the same loop without the meta-skill. Our findings provide practical design guidance for developing more effective harness self-evolution methods.

1. Introduction

Harness self-evolution can refine agent guidance without changing model parameters, but the same skills do not benefit models uniformly. The paper diagnoses weak-model failures and introduces skill adaptation to make harness guidance more executable.

1. Introduction

  • On 103 skill-sensitive pairs, skills improve frontier models by 29.2 points on average but weak models by only 1.2 points, and 86% of failed weak-model runs violate requirements already stated in the skill.The evaluation covers seven models and three independently constructed benchmarks.Source: §1 · pp. 1–2
  • Weak-model failures recur in tool execution, context management, instruction following, and verification.Agents may invoke unavailable tools or fail to recover from errors, lose track of the task, leave stated requirements unmet, or submit unchecked incorrect results.Source: §1 · p. 2
  • Skill adaptation reformulates guidance to match executor capabilities, using a fixed, task-independent meta-skill to target diagnosed failure patterns.The meta-skill is intended to facilitate harness self-evolution and more robust skill use.Source: §1 · p. 2
  • With no execution trajectories or verifier feedback, Kimi-K3’s meta-skill-guided revisions improve three weak executors by 5.1–14.1 points over the original skills.The meta-skill adds 5.1–6.0 points over a generic rewriting instruction, accounting for more than half of the average gain.Source: §1 · pp. 2–3
  • After five rounds, the meta-skill raises scores by 2.0–5.1 points across all seven executors versus the same self-evolution loop without it.Both settings improve tasks associated with tool-execution and instruction-following failures; verification-related gains are larger in self-evolution, where revisions use concrete failure evidence.Source: §1 · p. 3

2. Framework

The framework compares skill effects across capability tiers using task-skill pairs drawn from three benchmarks. It selects pairs where stronger tiers benefit more than weak models, then uses those pairs to study skill sensitivity.

2. Framework

  • Figure 2 depicts five rounds of self-evolution, with weak executors improving steadily under skill adaptation but barely moving without it.Horizontal lines mark tier averages.Source: Figure 2 · p. 3

2.1. Skill-Task Curation

  • The study draws from SkillsBench, DocOps, and Tessl, selecting seven models and grouping them into weak, moderate, and frontier tiers by performance without skills.The included reliable tasks total 607: 87 in SkillsBench, 186 in DocOps, and 334 in Tessl.Source: §2.1 · p. 3

2.2. Task Execution

  • Each task runs with and without its human-crafted skill while other inputs and the verifier remain identical; benchmark-native judges score success on a normalized 0–1 scale.The benchmarks use a deterministic task verifier, realized document state, and an LLM judge, respectively; retrieval and progressive disclosure are not evaluated.Source: §2.2 · p. 4

2.3. Pair Selection

  • A task-skill pair is the observation unit; a task with several skills counts as one pair carrying the whole bundle because individual skill contributions cannot be estimated.The same skill in two tasks counts twice.Source: §2.3 · p. 4
  • Skill-sensitive pairs are selected when frontier benefit exceeds a threshold relative to weak-tier benefit, or when frontier performance is saturated and the moderate tier benefits; pairs with weak-tier skill scores of at least 0.80 are excluded.The selection rule retains 103 pairs for the remainder of the paper.Source: §2.3 · p. 4
  • On 103 selected pairs, skills lift frontier and moderate models by up to 31.1 points while leaving weak models nearly unchanged and sometimes hurting the weakest model.The selected testbed contains 32 SkillsBench pairs, 32 DocOps pairs, and 39 Tessl pairs.Source: §2.3 · p. 4

3. Diagnosing Weak Executors

Weak executors often fail to act on guidance already present in skills, motivating a fixed meta-skill that adapts skill formulations to executor capabilities. The authors classify trajectory failures into four modes and turn those diagnoses into rewriting instructions.

3.1. Identifying Failure Modes

  • Failure labels use each weak model’s first observable failure in its trajectory, with verifier output also informing labels.A task can receive multiple labels when weak models fail at different points.Source: §3.1 · p. 4
  • The four modes distinguish tool errors or absent deliverables, losing task context, unmet stated requirements, and outputs that fail verification without an effective check.Context failures include following examples instead of the task or saving to the wrong path; instruction-following failures may still produce an output.Source: §3.1 · pp. 4–5
  • 86% of weak-model runs that still fail with a skill violate a requirement already stated in the injected skill, motivating adaptation to executor capabilities.Across 103 skill-sensitive task-skill pairs, the analysis assigns 131 failure labels; tool execution is most common and verification least common.Source: §3.1 · p. 5

3.2. From Failure Modes to Meta-Skill

  • The fixed, task-independent meta-skill translates the four failure modes into concrete rewriting guidance for adapting skills to executor capabilities.It specifies callable-tool checks and error handling, task scope and delivery path, actionable requirements with completion conditions, and independent verification; with trajectories, address the first observed failure first.Source: §3.2 · pp. 5–6

4. Experiments

The experiments test whether a fixed meta-skill helps adapt skills both through capable external rewriting and through executors’ own five-round self-evolution. They also compare gains across failure types and examine where adaptation still fails.

4.1. Skill adaptation with meta-skill

  • With a capable rewriter, Gemma-4-E4B rises from 0.029 to 0.170 under Kimi-K3, while weak executors rewriting their own skills score below the original.Kimi-K3 and GLM-5.3 improve all three weak executors, lifting the average by about +0.09.Source: §4.1 · p. 7, Table 1 · p. 7
  • With Kimi-K3, the meta-skill improves all three executors over a one-sentence rewriting instruction, whereas that instruction alone leaves Qwen3.5-9B below the original skill.The same rewriter is used in both setups.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2. Self-evolution with meta-skill

  • At round 5, all seven executors score higher with the meta-skill than without it; four larger executors’ gaps range from +0.020 to +0.046.The loop revises skills from prior execution trajectories and verifier output; each round runs once on all 103 pairs.Source: Figure 6 · p. 8, §4.2 · pp. 8–9

4.3. Analysis

  • One-shot adaptation gains are +0.056 for Execution, +0.055 for Instruction, and +0.103 for Context; self-evolution’s Verification gain is 0.101 larger.The analysis groups the 103 pairs by failure labels; labels can overlap. In the illustrated task, the meta-skill revision adds checks including rereading answer.json and recomputing the distance.Source: §4.3 · pp. 9–10, Table 2 · p. 9, Figure 7 · p. 10
  • Adaptation can specify the right procedure without ensuring execution: failures include editing a script without rerunning it, stopping before artifact delivery, or dropping constraints.In the illustrated case, Qwen3.6-flash passes four of five rounds with the meta-skill but fails a round by not rerunning a fixed script.Source: §4.3 · p. 10, Figure 7 · p. 10

5. Related Work

Related work covers agent skills and harness evolution, situating this paper at the intersection of model-capability mismatch and skill revision from failures.

Agent Skills

  • Agent skills package instructions, code, and resources for on-demand use without changing model parameters, and are commonly distilled from trajectories or written by experts.Prior work also studies skill retrieval, compression, and security.Source: §5 · p. 10

Harness Evolution

  • Harness evolution revises an agent’s harness from execution feedback; Meta-Harness uses a stronger external optimizer, while Self-Harness mines a model’s own failure traces.Other skill-level revisions use failure traces, verifier co-evolution, or held-out gating.Source: §5 · p. 11
  • This work differs by studying strong-to-weak skill mismatch and whether weak models can improve their own skills.The related work links these questions to external optimization and recursive harness improvement.Source: §5 · p. 11

6. Conclusion

The conclusion argues that harness benefits vary with executor capability and presents a fixed meta-skill for adapting skills. It recommends capability-tier evaluation and diagnostic guidance, while noting scope limits and possible future directions.

6. Conclusion

  • The same skill can help frontier models yet leave weak models unchanged or worse, even when violated requirements are already stated.The paper distills four failure modes into a fixed protocol for capable rewriters and weak models’ self-evolution.Source: §6 · p. 11
  • Harness self-evolution should be evaluated across capability tiers and give weak rewriters diagnostic guidance, since evidence alone can lead them to drift.The paper reports that the meta-skill lets weak rewriters hold their gains.Source: §6 · p. 11

Limitations

  • The evaluation supplies skills in context, bundles multiple skills, and fixes self-evolution at five rounds without updating model parameters.Retrieval and progressive disclosure are outside the evaluation’s scope.Source: §6 · p. 11

Discussion

  • Future work could track whether revisions resolve motivating failures and remain useful across repeated attempts.The discussion also suggests exploring similar guidance for tool interfaces and agent workflows.Source: §6 · p. 11

AI use statement

The authors report using generative AI tools to assist with coding and manuscript drafting, revision, and polishing.

AI use statement

  • The authors state they reviewed AI-assisted code and text, verified scientific claims and citations, and remain responsible for the research and manuscript.They report collecting no human-subject data.Source: AI use statement · pp. 11–12

A. Benchmarks and Task Suite

The task suite combines three independently built benchmarks covering different skill authors, agent harnesses, and judges, with conclusions reported only when they replicate across all three.

A. Benchmarks and Task Suite

  • The three benchmarks contribute 607 tasks and 103 selected skill-sensitive pairs, and conclusions are reported only when they replicate across all three independently built suites.Table 3 distinguishes reliably runnable task counts from the selected pairs.Source: A. Benchmarks and Task Suite · p. 16, Table 3 · p. 16

SkillsBench

  • SkillsBench runs engineering and scientific tasks in OpenHands and scores outputs with per-task deterministic verifiers that do not read the skill body.The 32 selected pairs span domains including control, planning, bioinformatics, and finance.Source: A. Benchmarks and Task Suite · p. 16

DocOps

  • DocOps covers four document types in Claude Code, with checkers verifying the produced document’s realized state across 32 selected task-skill pairs.The pairs span atomic edits, composite and cross-document operations, workflows, and publication or repair tasks.Source: A. Benchmarks and Task Suite · p. 16

Tessl

  • Tessl pairs realistic developer requests with vendor or open-source maintainer skills, and an LLM judge scores results against task-specific rubrics.Its 39 selected pairs are single-skill tasks, each with exactly one vendor skill.Source: A. Benchmarks and Task Suite · p. 16

B. Rewriting Protocol

Both experiments use a shared skill-rewriting step, while differing in the rewriter, available evidence, and number of rounds; output validity and scoring rules constrain evaluation.

B. Rewriting Protocol

  • The shared Adapt step rewrites a skill using evidence, and the two settings differ in rewriter, evidence, and number of rounds.The rewriter may return a revised skill or NO_CHANGE.Source: B. Rewriting Protocol · p. 17

Inputs to the rewriter

  • The rewriter receives the task, executor interface, and current skill; self-evolution also supplies the prior attempt’s score, trajectory digest, and verifier output.The skill-adaptation setting supplies no execution history or verifier feedback.Source: B. Rewriting Protocol · p. 17

Output and validity

  • Rewriters return complete changed files or NO_CHANGE, while validity checks enforce loadability and screen for hidden-answer leakage.Invalid rewrites cause the unchanged skill to be rerun and remain in the arm’s score.Source: B. Rewriting Protocol · pp. 17–18

Scoring conventions

  • Self-evolution runs five rounds without selecting the best round or rolling back, and reports each round against a same-window rerun of the original skill.Each model-task-condition cell is run once.Source: B. Rewriting Protocol · pp. 17–18

C. The Meta-Skill and the Baseline Prompts

The meta-skill gives a fixed rewriting protocol tailored to weak executors, while the baselines offer general or minimal instructions without its failure-specific guidance.

Meta-skill

  • The meta-skill diagnoses the earliest failure among four classes and prescribes targeted edits, leaving unrelated skill content alone.The diagnosis draws on 602 measured model-task cells.Source: C. The Meta-Skill and the Baseline… · pp. 18–19
  • For tool-execution failures, it calls for distinguishing callable tools from document names, environment preflight, recovery fallbacks, and executable commands.This class includes unavailable-tool calls or errors after which the agent stops.Source: C. The Meta-Skill and the Baseline… · pp. 18–19
  • For context-management failures, it prioritizes the task, allowed changes, and delivery path, and marks the rest of the skill as reference material.Auxiliary-file content needed for a step should be inlined.Source: C. The Meta-Skill and the Baseline… · p. 19
  • For instruction-following failures, it converts each implicated requirement into an action, target, and completion condition, including a negative branch where needed.The guidance notes that adding more text about an already stated requirement does not help.Source: C. The Meta-Skill and the Baseline… · p. 19
  • For verification failures, it adds an ending check that rereads the artifact, recomputes the reported quantity independently, compares a threshold, and loops until satisfied.A written file or error-free run alone is not evidence of correctness.Source: C. The Meta-Skill and the Baseline… · p. 19

Prompt comparisons

  • The meta-skill also limits edits to evidence-implicated content, preserves documented capabilities, forbids invented names or answer leakage, and allows NO_CHANGE.The no-meta prompt supplies role framing and hard rules, while the plain adaptation instruction simply asks for a skill that works for all models.Source: C. The Meta-Skill and the Baseline… · p. 19

D. Full Self-Evolution Results

After five self-evolution rounds, every executor scores higher with the meta-skill than without it, although gains are not monotonic and the arms sometimes cross early.

D. Full Self-Evolution Results

  • At round 5, all seven executors score 0.020–0.051 higher with the meta-skill than without it.Neither arm improves monotonically, and the arms cross on several executors in early rounds.Source: Table 4 · p. 20, D. Full Self-Evolution Results · p. 20, Figure 8 · p. 21
  • Table 4 reports mean scores by executor, round, and benchmark, with Δ5 marking the round-5 gap between the two arms.Figure 8 plots the four executors not shown in Figure 6, with curves showing gains over the original-skill rerun baseline.Source: Table 4 · p. 20, Figure 8 · p. 21

E. Additional Case Studies

Four case studies show how meta-skill-guided rewrites address concrete execution failures, while the one-shot examples illustrate mechanisms rather than establish aggregate results.

E. Additional Case Studies

  • The four cases illustrate how meta-skill guidance changes skills and task rewards, but each is a single rollout rather than proof of the aggregate result.The cases cover two self-evolution tasks and two one-shot rewrites.Source: E. Additional Case Studies · p. 21

Spreadsheet totals (DocOps, self-evolution)

  • The meta-skill-guided spreadsheet skill distinguishes formula cells from headers and adds a source-row cross-check, after repeated wrong-cell failures.Qwen3.5-9B fails in every round without the meta-skill and in rounds 1–3 with it; the adapted skill passes in round 4.Source: E. Additional Case Studies · p. 21, Figure 9 · p. 22

Project setup files (Tessl, self-evolution)

  • For project setup, the meta-skill rewrite adds task scope, delivery path, and install commands, helping Gemma-4-E4B finish both files even after tool errors.It passes every round, while the executor fails in three of five rounds without the meta-skill.Source: E. Additional Case Studies · pp. 21–22, Figure 10 · p. 23

PDDL planning (SkillsBench, one-shot rewriting)

  • For PDDL planning, a gated preflight, named file-writing tool, per-entry completion conditions, and disk check help Gemma-4-E4B recover from an import error.The one-sentence rewrite instead leads to a hanging terminal and no written plan.Source: E. Additional Case Studies · p. 22, Figure 11 · p. 24

Protein expression workbook (SkillsBench, one-shot rewriting)

  • For the protein workbook, a python3 fallback and final data cross-check let Qwen3.6-flash recover from a missing command and catch an offset lookup range.After two fix–recalculate–verify iterations, all six tests pass; the recalculation script alone had passed the wrong lookup with zero errors.Source: E. Additional Case Studies · pp. 22–23, Figure 12 · p. 24

Related papers

Loading 2610.33024v1…