Agent Skill Evolution: How Revisions Affect Coding Agents

• Most Skill revisions change a rule or procedure, and similarly sized commits that revise Skills alter harness configuration more often than other commits.Changes chiefly involve CLAUDE.md, AGENTS.md, and agent and command definitions.Source: §1 · p. 2
• Added rules raise compliance across all 21 models and increase required actions in sandbox agents; Claude Sonnet 4.5 gains +0.19 under both loading modes.Blind judges also find modestly higher final correctness; three open models retain about 38% of their gain with on-demand loading.Source: §1 · p. 2
• Revisions lengthen prompts but shorten answers; they add no detectable episode cost, whereas loading a Skill body adds 37 to 57% more tokens.Source: §1 · p. 2
• The study reconstructs revision histories for 3,159 Skills and measures rule-change effects on 21 models and agent effectiveness and cost in a sandbox.It compares old and new Skills on the same request and model, focusing on mechanically checkable rules naming a command, path, or identifier.Source: §1 · p. 2
• The authors release a dataset of 3,159 Skills with revision histories, 396 revision-derived probes, an 80-task executable benchmark, human judgments, and a replication package.Source: §1 · p. 3

2.1 Studies on Agent Skills

• Studies of Agent Skills compare a Skill with its absence or analyze it at one version, reporting mixed effects on pass rates, behavior, and token overhead.Related work also examines Skill selection, procedural anchoring, actionability, and reuse defects.Source: §2.1 · p. 3

2.2 Studies on Agent Harnesses

• Harness studies examine context files, configuration co-evolution, and cost, while most compare a configuration file with its absence or treat the harness as one unit.The paper instead measures what a Skill revision changes in agent effectiveness and cost.Source: §2.2 · p. 3

2.3 Studies on Agentic Cost

• Agentic-cost studies examine cost-controlled evaluation, cost-of-pass, trajectory costs, and methods that shorten trajectories.Source: §2.3 · p. 3

3 Study Design

• The design rebuilds each sampled SKILL.md history, pairs its first and last versions, and turns added or deleted directives into requests that require those directives.The resulting probes are evaluated with the Skill name and description alone or with the old or new body, across single-answer models and multi-step agents.Source: §3 · p. 3, §3.2 · p. 5

3.1 Dataset Construction

• The dataset follows SKILL.md files through repository histories, retaining differing first/last pairs; the sampled histories span a median of three versions, or two revision commits.Repositories that could not be cloned within the time budget count as unrevised, making the observed revised share a lower bound; a separate high-star set addresses clone-budget limits.Source: §3.1 · p. 4

3.2 Probe Construction

• A directive is converted into a probe by generating a request that needs its rule, checking required or forbidden strings, and comparing none, old, and new Skill conditions.The probe set includes added-, deleted-, and dilution-directive probes, and its measured effects apply only to requests that need the rule.Source: §3.2 · p. 5
• The study checks its string-based grader against blinded annotator judgments, finding higher directive-following gains in those judgments than in the string check.On reweighted main-set answers, annotators’ labels give about +0.52 versus +0.41 for the string check; the authors identify blind judgments of sandbox artifacts as the primary outcome evidence.Source: §3.2 · p. 6

3.3 Model and Agent Settings

• Evaluation spans 16 open-weight models, five closed models, and sandbox agents, including read-only and executing setups with different turn limits.In multi-step tests, agents may decide whether to read the Skill; the executing sandbox holds the repository at the new version’s commit.Source: §3.3 · pp. 6–7

3.4 Evaluation Metrics

• Effectiveness is measured as directive compliance, required agent action, and final artifact correctness, while token counts and latency capture cost.Differences are paired over probes with repository-clustered bootstrap confidence intervals and Holm-adjusted p-values; token costs include revision changes and the cost of carrying a Skill body.Source: §3.4 · p. 7

4 RQ1: How Do Agent Skills Co-Evolve with Their Harness?

• The study classifies added and deleted lines by content type and compares harness-file changes with commits matched by repository and files changed.The line classifier misses most MUST/NEVER and conditional lines, so its rule-change estimate is conservative with respect to those forms.Source: §4 · p. 8

4.1 Most revisions change a rule or procedure

• 55.0% of 2,608 revisions add or delete a MUST/NEVER constraint, conditional rule, or numbered step; 19% change examples or code without changing a rule or procedure.Another 14% change only prose.Source: §4.1 · p. 8
• Only 0.4% of commit subjects name a model version, while 37.6% of revision commits carry an AI-agent or bot signature, close to 40.5% of other commits.The model-motive counts cover documented motives only.Source: §4.1 · pp. 8–9

4.3 Skill revisions touch harness files more often

• 23.0% of Skill-revision commits change harness configuration versus 18.5% of matched commits, a difference of +4.5 points [+1.6, +7.3].The excess is concentrated in CLAUDE.md, AGENTS.md, other agents’ configuration, and Claude Code agent and command definitions; Skill-creation commits show a larger matched difference of +14.8 points [+9.4, +20.4].Source: Figure 3 · p. 9, §4.3 · pp. 9–10

4.2 Few Skills gain executable references

• Executable references rise only modestly: about 0.44 more paths and 0.25 more commands per Skill against 101 more words.Most Skills leave their executable surface unchanged, and the share of paths and commands does not grow.Source: §4.2 · p. 9

5 RQ2: How Does Skill Evolution Affect Task Effectiveness?

• The study focuses on rule changes that add or delete a concrete, mechanically checkable rule naming a command, path, or identifier.Of 2,608 revisions, 1,074 (41%) add at least one directive of this kind.Source: §5 · p. 10
• Effectiveness is measured through single-answer compliance, required agent actions, and final correctness, supplemented by token-only controls, action traces, and blind judgments.The study compares old and new Skill versions on the same request and model.Source: §5 · p. 10
• Effects are smaller in the tested agent settings than in single answers, and on the same 36 tasks final correctness rises less than action rate, without a significant difference.On 154 probes where revisions changed only prose or examples, four models’ compliance change was +0.017 [+0.002, +0.034], within a pre-set equivalence margin of 0.08.Source: §5 · p. 10

5.1.1 Every model follows an added rule more often.

• +0.35 to +0.46 compliance gains occur across the 16 open models on the 147 probes answered by the eligible models.All 16 open-weight models follow the added rule more often; the figure’s panels measure different constructs on different axes.Source: §5.1.1 · p. 10, Figure 4 · p. 11

5.1.2 Stating a rule beats naming its token.

• +0.15 [+0.12, +0.19] compliance is added by stating the rule beyond merely naming its token.Adding only the rule line to the old Skill gives about two thirds of the full effect, while unrelated text of equal length has no detectable effect.Source: §5.1.2 · pp. 10–11, Figure 5 · p. 12

5.1.3 The gain comes from tokens new to the Skill.

• New tokens lift compliance by +0.51 to +0.66 from an old-version baseline of 0.05, while already-present tokens start at 0.61 and gain +0.08 [+0.03, +0.12].For rules not needed by a request, models use the rule’s token +0.02 to +0.06 more often under the new version, versus +0.38 to +0.45 when it is needed.Source: §5.1.2 · p. 11, §5.1.3 · p. 11, Table 3 · p. 12

5.1.4 Deleting a rule lowers compliance with it.

• Removing a rule lowers compliance by 0.27–0.33 on each of eight models, about three quarters of the gain from adding one.Exploratory hand-checked results distinguish deleting a duplicate (−0.12) from deleting a replaced rule (−0.58).Source: §5.1.4 · p. 12

5.1.5 Older and newer model generations gain similarly.

• No difference between older and newer model generations is significant after Holm correction across the tested within-family comparisons.One new-token comparison differs: Qwen 2.5-7B gains less than Qwen 3-8B by −0.083 (Holm-adjusted p=0.018).Source: §5.1.5 · p. 13

5.1.6 Gains shrink when the Skill fills the window.

• The benefit falls from 0.424 at low Skill occupancy to 0.270 when the Skill fills three quarters of the context window.Below that, growth alone barely matters, and growth around an unchanged rule has no detectable effect.Source: §5.1.6 · p. 13

5.1.7 Part of the gain survives in long contexts.

• In contexts of about 15k and 40k tokens, the new version raises compliance by +0.07 to +0.28, compared with +0.37 to +0.42 in short answers on the same probes.About three fifths of the short-answer effect survives at 15k tokens and about two fifths at 40k; when the context already shows a token missing from the old Skill, the revision still adds +0.31 [+0.20, +0.42].Source: §5.1.7 · p. 13

5.2 Multi-Step Setting

• The sandbox removes the Skill file from the checkout in each comparison, so agents see only the Skill version supplied in that setting.The benchmark evaluates action, execution, and final correctness, with final correctness checked on 36 tasks.Source: §5.2 · p. 13, §5.2.2 · p. 13

5.2.1 Read-only agents follow the new rule more often.

• Read-only agents comply more often with the new Skill, gaining +0.18 to +0.33 on three open models across 40 probes.Source: §5.2.1 · p. 13

5.2.2 Sandbox agents take the required action more often.

• +0.23 [+0.15, +0.31] is the average action-rate gain across four sandbox agents on 80 tasks.Devstral gains +0.36 and Sonnet 4.5 gains +0.19; the gain remains +0.22 [+0.14, +0.30] on 62 tasks where all old-Skill references still exist.Source: Table 4 · p. 14, §5.2.2 · p. 14

5.2.3 Human-judged correctness rises for two open models.

• +0.10 [+0.04, +0.17] is the increase in human-judged final correctness pooled over the three judged agents.For the two Qwen models, final correctness rises by +0.13 [+0.05, +0.20]; Sonnet 4.5’s +0.06 [−0.06, +0.18] estimate is inconclusive.Source: Table 5 · p. 14, §5.2.3 · p. 15

5.2.4 On-demand loading keeps about half of the action gain.

• On-demand loading retains 51% of the four agents’ action gain: the average falls from +0.23 to +0.12 [+0.06, +0.18].The three open models keep about 38% of their gain.Source: §5.2.4 · p. 15

5.2.5 Agents that read the rule do not always act on it.

• After the rule is displayed, Sonnet acts on it in 88% [77, 97] of episodes and Qwen in 61% [48, 74].The task-paired difference is +0.24 [+0.12, +0.36]; the paper reports that Qwen’s on-demand drop does not come from lower uptake once the rule is seen.Source: §5.2.5 · p. 15

6.1.1 A revision adds input tokens but shortens the answer.

• 17% more tokens on open models and 24 to 27% more on four Claude models are used per request when input and output are counted.For juxt/allium, adding the allium check rule shortened the answer from 201 to 81 tokens while raising compliance from 0.07 to 0.99.Source: §6.1.1 · p. 15

6.1.2 One rule–model pair in five adds tokens with no detected gain.

• 20% of rule–model pairs add prompt tokens with no detected compliance change, making them overhead candidates rather than proof of no effect.Across 55 rules classified as overhead by at least half the models, 27 were already followed and 28 were ignored whether present or not.Source: §6.1.2 · p. 16

6.2.1 A revision adds no detectable cost per episode.

• −2.3 to +3.0% episode-token changes across four agents are not detectable; every interval spans zero and lies within −14 to +12%.The longer prompt is offset by fewer steps.Source: §6.2.1 · p. 16

6.2.2 Carrying a Skill body costs far more than revising it.

• 37 to 57% more tokens per episode are used by the old Skill body than by no body, averaging 50% across four agents.The body is re-sent each turn; fewer turns do not prevent the Skill-body episodes from using more tokens.Source: §6.2.2 · p. 16

6.3 Every revision type adds input, prose-only the least

• +248 median tokens are added by revisions changing a rule or procedure, compared with +6 for prose-only revisions.For 154 dilution probes, prose- or example-only revisions add +34 input and +2 output tokens per request.Source: §6.3 · p. 17
• 18–19% more input tokens per single answer accompany a revision, while shorter answers reduce the increase in total tokens.The section’s overall account is that the marginal revision cost is mostly input; in agent episodes, fewer steps leave no detectable change.Source: §6.3 · p. 17

7.1 Threats to Validity

• The automated final check inspects artifacts rather than running tests and agrees only moderately with blind annotators (𝜅= 0.50–0.55), so correctness claims rely on human judgement.Compliance checks can credit token repetition, and five samples per condition leave small real changes compatible with equal measured compliance.Source: §7.1 · p. 17
• Probe requests were generated by Qwen 3.5-27B, whose models gained slightly more than other models; a Phi-4-request probe set showed gains for non-Qwen models too.Using all 304 eligible probes changed the main result from 0.405 to 0.401.Source: §7.1 · p. 17
• Maintainer motives may be undocumented, so RQ1 counts cover only stated motives; RQ2 tests whether revisions work across generations, not why they were made.Source: §7.1 · p. 18
• The findings apply to mechanically checkable rules; the literal filter retains 17% of candidate directives and under-represents prohibitions, conditional rules, and permission guardrails.The sandbox uses one published scaffold, and only the first and last 2,000 characters of command output are shown, likely lowering on-demand gains for long Skills.Source: §7.1 · p. 18

7.2 Implications for Skill Maintainers

• 25% of sampled later in-scope commits needed the added rule; under the stated scenario, this implies about 0.10 more single-answer compliance and 0.06 more required action.The scenario assumes conditional effects hold for requests needing the rule and are zero otherwise; maintainers are advised to add rules that supply missing information and verify before deleting rules.Source: §7.2 · p. 18

8 Conclusion and Future Work

• Added rules mainly change behavior when they name a command or path absent from the old Skill; gains are similar across generations and smaller with on-demand loading.The conclusion also reports that unrelated text of the same length has no detectable effect and deleting a rule lowers compliance by about three quarters of the gain from adding one.Source: §8 · p. 18
• A revision adds mostly input tokens and no detectable episode cost, whereas loading the Skill body costs far more.Source: §8 · p. 19
• Future work includes graders for non-literal rules, natural task streams, production harnesses and reasoning models, pre-merge probes, and Skills authored after newer models.Source: §8 · p. 19