PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance

Chen Shen, Estevam Hruschka

arXiv:2609.05677v1cs.CLcs.AIcs.HCcs.SE

TL;DR

Research on automated skill curation has largely left human maintenance unmeasured. This study mines commit histories from five public repositories and codes their edits, finding a human-governed, often AI-assisted process and an unreliable rule-likeness measure.

  • Problem

    Work on automated skill curation largely evaluates against automated baselines, leaving the human maintenance process unmeasured.

    Source: §1 · p. 1

  • Method

    The study mines histories from five public skill repositories and codes substantive edits with pre-registered governance, operation, and trigger-evidence codebooks.

    Source: §1 · p. 2

  • Results

    Every substantive edit was authored or merged through a named human account, 62% carried an AI co-author trailer, and the rule-likeness axis failed its reliability gate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that automated curators should be evaluated against human-governed maintenance loops, not only fully automated skill-evolution pipelines.

    Source: §9 · p. 9

    The five repositories are purposively selected, so maintenance-depth quantities are upper-bound-like and governance shares are descriptive and organization-dependent, not population base rates.

    Source: Limitations · p. 13

Abstract

from arXiv · show

Lifelong LLM agents increasingly rely on external skill artifacts as one element for preserving and reusing capabilities over time. These skills (usually portable Markdown files such as SKILL.md) describe when and how to apply a capability and must be corrected, expanded, and consolidated as tools and usage patterns shift over deployment. Recent work seeks to automate skill curation, but it largely evaluates against automated baselines and treats human maintenance as an unmeasured bottleneck. We study that missing process directly. We mine the full commit histories of five public AI-skill repositories, a purposive sample of AI-tooling organizations, covering 873 commits, 143 skill files, and 254 substantive post-creation edits from October 2025 to June 2026. We code each edit with pre-registered governance, operation, and trigger-evidence codebooks. Three findings emerge. First, every substantive edit is authored or merged through a named human account, while 62% carry an AI co-author trailer, with large repository-level variation. Second, these edits are genuine curation: an audited sample shows that most change skill content, and the coded operations are dominated by additions and corrections. Third, a pre-registered rule-likeness axis fails its reliability gate; reliably coding rule-likeness from commit artifacts remains an open measurement problem. We release the corpus, codebooks, mining scripts, and a replay protocol for automated skill curators. For self-evolving agents, public skill maintenance currently looks less like an autonomous pipeline than a human-governed, AI-assisted loop that future curators must measure against and operate within.

1 Introduction

This paper examines who maintains public agent skills and what maintenance entails, addressing a process largely unmeasured in work on automated skill curation. It mines five purposively sampled repositories, while limiting its claims to the histories they expose.

1 Introduction

  • Agent skills are version-controlled Markdown artifacts that maintainers correct, expand, reorganize, or retire as tools and usage patterns shift.Recent automated-curation work edits skills from experience, but human maintenance is mostly unmeasured.Source: §1 · p. 1
  • Prior studies examine a different artifact, cross-sectional marketplaces, or synthetic decision histories, leaving longitudinal public skill-file maintenance unstudied.The closest context-file study lacks a coded operation taxonomy, authorship governance, and failure-trigger evidence.Source: §1 · pp. 1–2
  • The study mines five purposively sampled public repositories, tracking 143 SKILL.md files and coding 254 substantive post-creation edits from 873 commits.Each edit is coded under three pre-registered codebooks.Source: §1 · p. 2
  • Every substantive edit is authored or merged through a named human account, while 62% carry an AI co-author trailer.The repositories vary substantially in trailer-visible co-authorship.Source: §1 · p. 2
  • The study is descriptive: its five repositories are purposively selected AI-tooling organizations, not a representative sample.Claims are limited to what their maintenance histories contain.Source: §1 · p. 2

2 Related Work

Prior work develops autonomous skill libraries and automated curation, but does not measure the human maintenance process. Related artifact-mining studies offer nearby methods, while this study focuses on skill files and edit governance.

2 Related Work

  • Automated skill-library research evolves skills from experience and benchmarks resulting skills, but does not study human maintenance; Ouyang et al. (2026) has no manual-curation arm.Related memory approaches operate on internal stores or runtime artifacts rather than mining longitudinal skill-file histories or writing generalized rules into skill files.Source: §2 · pp. 2–3
  • Studies of prompts, context files, rules, and workflows mine evolving text artifacts with typed change taxonomies, but vary in artifact and edit patterns.Prompt and context/rule-file studies report additive-dominant changes, whereas workflow edits skew modification-heavy.Source: §2 · p. 2
  • Table 1 positions this study among prior work by longitudinal coverage, artifact, and coded human-curation analysis.The study reports that prior work had not mined longitudinal maintenance histories of human-curated Markdown skill files with coded operations and governance.Source: Table 1 · p. 3
  • The paper uses the Co-Authored-By commit trailer as an AI-authorship marker, following repository-mining practice.Larsen & Moghaddam (2026).Source: §2 · p. 3

3 Corpus and Method

The study builds a commit-level corpus from five public skill repositories and applies pre-registered codebooks to substantive file edits. It combines deterministic metadata coding with LLM-coded operation and trigger labels, alongside reliability checks and an audit of edit content.

3 Corpus and Method

  • The corpus covers five purposively selected repositories, 873 commits, 143 SKILL.md files, 631 file-history records, and 254 substantive post-creation edits.A substantive edit changes at least five lines after file creation and is neither bot-authored nor trivial; the study removed only eight bot-authored commits that otherwise met the substantive threshold.Source: §3 · p. 3
  • A 50-edit audit classified 70% as skill-content changes, with the remainder comprising migration, dependency-version churn, or style.The study reports 27, 22, 16, and 15 deeply maintained skills across four sensitivity-analysis cells.Source: §3 · p. 4
  • Three pre-registered codebooks cover governance, eight curation operations, and trigger evidence on a 0–3 ladder.Governance derives from commit metadata; Opus 4.8 coded operation, trigger, and rule-likeness labels, with a separate 50-edit pass to estimate stability.Source: §3 · p. 4
  • Governance labels matched deterministic metadata on 252 of 254 edits, while operation labels received human-recode validation at κ = 0.72 for the two-way collapse.The eight-way operation taxonomy had κ = 0.46 in the blind human recode; the authors treat it as descriptive and note that two LLM passes may share systematic bias.Source: §3 · p. 4

4 What Maintenance Does

Maintenance tilts toward enhancement over correction, though the eight-way operation labels are less reliable than the two-way collapse. The results also show bulk-refactor sensitivity, infrequent pruning, and an unreliable pre-registered rule-likeness measure.

4 What Maintenance Does

  • 60% of edits were enhancement, 38% correction, and 2% other under the two-way operation collapse.Content expansion (72) and factual correction (56) together account for half of the 254 edits; the collapse groups the eight operations for interpretive grounding rather than recoding.Source: §4 · p. 4
  • The two-way collapse reached κ = 0.72 in a blind human recode, while eight-way labels showed only moderate inter-LLM stability at κ = 0.62.Cross-family recodes agreed less; weighting commits equally yields 67% enhancement, 31% corrective, and 2% other over 187 commits, versus 60% enhancement at the edit-row level.Source: §4 · p. 5
  • 75% of factual-correction edits came from multi-file mass-refactor commits, while consolidation/merging and deprecation accounted for only 11 edits.After mass-refactor exclusion, factual correction falls from the second-largest category to fourth; content expansion, fixing-from-failure, and restructuring remain robust.Source: §4 · p. 5
  • The pre-registered rule-likeness axis failed its reliability check at κ = −0.02, so the authors do not report its apparent distribution.Disagreement concentrated at the instance-bound/mixed boundary; the authors describe the failure as an instrument and base-rate issue as well as a measurement obstacle for rule-generalizing curators.Source: §4 · p. 5

5 How Skills Evolve

Skill maintenance was frequent but usually produced stable-to-additive size trajectories, while edits concentrated on instructions and embedded code.

5 How Skills Evolve

  • 32 of 120 skills grew by more than 10% in resident token length, 7 shrank by more than 10%, and 81 remained stable.Successive commits touched a skill every 5 days at the median; the 5–8 month observation windows do not support a strong growth-or-decay narrative.Source: §5 · p. 5
  • Net resident-token trajectories here are not directly comparable to prior studies’ per-commit word add/delete counts.The measures capture different quantities.Source: §5 · p. 5
  • 85% of substantive edits touched instructions and 56% touched embedded code, compared with 38% for the router and fewer edits to other components.Frontmatter, examples, and references were touched by 15%, 11%, and 4% of edits, respectively.Source: §5 · p. 6

6 Who Maintains, and Why

Substantive skill edits are released through human accounts, although trailer-visible AI co-authorship varies by repository. The study detects no governance-based division of labor by edit size or skill component.

6 Who Maintains, and Why

  • 62% of 254 substantive edits carried an AI co-author trailer, but every edit was authored or merged through a named human account.The remaining 38% were AI-trailer-absent; the claim concerns commit attribution, not verified authorship.Source: §6 · p. 6, Table 3 · p. 7
  • No governance split survived the pre-registered controls for skill component, and edit size and scope also showed no detectable separation.The component null indicates no detectable division of labor, not strict uniformity, given low power and multiple testing.Source: §6 · p. 6
  • Trailer-visible governance varied sharply by organization, from nearly all-human maintenance to organizations co-authoring a large majority of edits.The pooled 62% summarizes heterogeneous repository regimes, not a population base rate.Source: §6 · p. 6, Figure 2 · p. 7
  • Automated curators operating on these public skill artifacts would need to fit the observed human authorship-or-merge process.Every substantive edit across the five repositories was human-authored or human-merged.Source: §6 · p. 6
  • Commit metadata does not measure true AI use or establish verified human review; a human-merged edit denotes a pull-request web-merge, not confirmed approval.Trailer-visible co-authorship may undercount undisclosed use or overcount auto-injected trailers.Source: §6 · p. 6
  • 24% of edits carried concrete failure evidence in the stricter pass, but a cross-family recode assigned 63%, making trigger prevalence instrument-dependent.The authors treat trigger counts as descriptive rather than a stable quantity.Source: §6 · p. 7

7 Does Maintenance Help? A Bounded Transfer-Task Test

Under the preregistered transfer-task harness, maintained skills showed no detectable average quality improvement over their earliest versions, though individual skills moved in both directions. The result is bounded to this harness and does not establish that maintenance is inert.

7 Does Maintenance Help? A Bounded Transfer-Task Test

  • The replication tested 13 public skills with at least six substantive edits across four repositories on 143 transfer tasks, using full skill packages, multi-turn interaction, and blind judges from a different model family.The preregistered skill-level inference had power of 0.77 to 0.90 for the pilot-sized effect; judge agreement was moderate and below the preregistered gate.Source: §7 · pp. 7–8
  • The skill-level mean v new − v old difference was −0.09 on the 1–5 scale (95% CI [−0.28, +0.10]), with no significant test and 5 of 13 skills favoring v new.Figure 3 shows individual skills moving both ways but no average effect; the authors interpret this as no evidence of a pilot-sized benefit, not evidence that maintenance is inert.Source: Figure 3 · p. 8, §7 · p. 8
  • Illustrative skill outcomes diverged: brainstorming’s added process structure reduced judged coverage from 2.06 to 1.33, while pr-writer’s output constraints raised non-redundancy from 3.45 to 4.58.Both skills roughly doubled to tripled in length, so size alone did not distinguish their outcomes; the 13-skill aggregate split was five positive and eight negative.Source: §7 · p. 8
  • The null is specific to the constructed transfer-task harness: native repository workflows or external skill benchmarks could produce different results.Unlike in-the-wild evaluations that vary deployment or library, this study held the harness fixed and varied the maintained version.Source: §7 · pp. 8–9

8 Discussion: Toward Rule-Generalizing Curation

The observed maintenance regime motivates curator designs that retain human control, account for accretive edits, and treat rule generalization as a measurement challenge. The proposed design implications are hypotheses grounded in these observations, not validated systems.

8 Discussion: Toward Rule-Generalizing Curation

  • Maintenance is a human–AI loop that grows and corrects skills more often than it prunes them, while rule-likeness remains difficult to operationalize reliably from commit artifacts.Two independent LLM coding passes did not reach acceptable agreement on whether edits encode generalizable rules.Source: §8 · p. 9
  • The proposed curator would record traceable edit rationales and consolidate them into maintainer-approved generalized guidance rather than rewriting skill text from individual instances.The authors describe this as a design hypothesis; designing such a curator remains future work.Source: §8 · p. 9
  • The design implications include an explicit pruning budget, human-owned release or merge control, operation-first triage, and repository-specific governance models.Consolidation and deprecation account for 4.3% of edits (11/254); operation type was the highest-agreement coded axis, whereas governance did not predict edit size or scope.Source: §8 · p. 9
  • Failure provenance is not yet stable enough to serve as a curator’s primary supervision target, and rule-likeness failed its reliability gate.Concrete-failure-evidence share was 24% under the primary pass and 63% under a cross-family recode; the trigger ladder was only borderline-reliable within one family.Source: §8 · p. 9
  • The released corpus supports replay evaluation: a curator receives a prior skill version and context, proposes the next edit, and is scored against the human edit.Building a baseline remains future work.Source: §8 · p. 9

9 Conclusion

The paper finds that public skill maintenance is human-governed and often visibly AI-assisted, and releases resources for studying the process. It concludes that future curators should be evaluated against human-governed maintenance loops, not only fully automated pipelines.

9 Conclusion

  • Every substantive edit was authored or merged through a named human account, while commit trailers marked 62% as AI co-authored.The paper releases a commit-level corpus, three codebooks, and reproducible scripts, and reports a preregistered rule-likeness reliability failure as an open measurement problem.Source: §9 · p. 9

Limitations

The study’s scope and measurements constrain how its maintenance findings should be interpreted. Its sample, observable public histories, coding reliability, and short observation window limit generalization and measurement claims.

Limitations

  • The five repositories were purposively selected from organizations with sustained maintenance, so maintenance-depth quantities are upper-bound-like and governance shares are organization-dependent rather than population base rates.The observations should not be read as evidence of lower engineering quality, greater AI dependence, or weaker maintenance practices.Source: Limitations · p. 13
  • Requiring observable public histories creates a transparency-selection effect: richer histories make corrections, AI-attribution signals, and maintenance activity more measurable than less-open counterparts.Source: Limitations · p. 13
  • Observed trajectories are right-censored over 5–8 months.Source: Limitations · p. 13
  • The corrective/enhancement collapse was human-validated on 50 edits by a single coder at κ = 0.72, while the human-checked 8-way leaves had only moderate agreement (κ = 0.46).Source: Limitations · p. 13
  • The trigger axis had no human check, and a cross-family recode reproduced neither its labels nor distribution (κ = 0.18; 63% vs. 24% L2+L3), making trigger counts primary-pass-specific.Source: Limitations · p. 13
  • Co-Authored-By trailer prevalence can under- or over-count true AI involvement; trigger evidence is bounded by commit records, and “human-merged” denotes web-merge flow, not verified review approval.Source: Limitations · p. 13

Ethics and reproducibility

The release uses only public repository metadata and coding labels, and reports analyses at the organization level. It provides records, codebooks, scripts, and materials for reproducing the analyses.

Ethics and reproducibility

  • The study derives governance signals from public committer, author, and AI-tool identities, without adding inferred attributes or de-anonymizing beyond recorded trailer and merge-committer fields.The released records join public identity fields to curation labels; no repository file content is redistributed.Source: Ethics and reproducibility · p. 13
  • Results are reported only as organization-level aggregates, although aggregation is effectively individual-level for a single-maintainer repository.The authors note that public availability does not itself constitute consent to aggregation.Source: Ethics and reproducibility · p. 13
  • The release includes labeled commit records, codebook datasheets, mining and analysis scripts, reproduction materials, and pointers to five repositories pinned at fixed commits.Corpus-level commit counts can be reproduced by recloning the pinned repositories; labeled-edit analyses recompute offline from released records and annotation passes.Source: Ethics and reproducibility · pp. 13–14

A Codebooks and Reliability

This section details the study’s coding rules, reliability checks, sensitivity analyses, transfer-task studies, and replay protocol. The checks distinguish robust findings from coding or study-design boundaries.

A Codebooks and Reliability

  • The operation codebook separates edits by dominant purpose, while three-way rule-likeness coding had κ = −0.02; operation-label agreement varied across model families, and inter-pass estimates are imprecise.A binary rule-likeness recode reached κ = 0.43, but the authors still do not report a rule-likeness distribution; κ estimates are based on a 50-edit overlap without confidence intervals.Source: A Codebooks and Reliability · pp. 14–15, Table 4 · p. 14, Table 5 · p. 14

B Component Decomposition and Governance

  • No component’s association with governance survives all three controls; apparent frontmatter enrichment disappears after excluding mass refactors and after repository stratification.The frontmatter result falls from 90% AI-trailered in the raw comparison to p = 1.0 after mass-refactor exclusion and p = 0.22 under repository stratification.Source: B Component Decomposition and Governance · pp. 15–16

C Sensitivity Analyses

  • After excluding 89 multi-file mass-refactor edits, content expansion (59), fix-from-failure (41), and restructuring (31) remain prominent.The audit found 35 of 50 sampled edits were curation-confirmed; among those, 77% were enhancement and 23% corrective. Across 120 skills, 81 were stable within ±10% token length, while cadence ranged from a 1.5-day median in obra to 14.5 days in trailofbits.Source: C Sensitivity Analyses · p. 16, Figure 4 · p. 16

D Does Maintenance Help? The Transfer-Task Study

  • The powered transfer study used 13 skills and 143 frozen tasks, with old and new skill versions compared under an equalized multi-turn protocol; no rubric dimension survives Holm correction.The supplied study limitations note that tasks are constructed transfer scenarios, and the no-skill baseline is more verbose than skilled conditions; the old-versus-new comparison is length-balanced.Source: D Does Maintenance Help? The… · pp. 16–17, Table 6 · p. 17

D.1 Pilot study

  • The pilot was underpowered and its two tests diverged: the sign test favored maintained versions at 15 wins to 6 with 9 ties (p = 0.08), while rubric scores gained +0.27 points (p = 0.02).Under the corrected powered design, the pilot’s positive trend does not reappear.Source: §D.1 · pp. 17–18

E A Replay Protocol for Skill Curation

  • The replay protocol predicts the next human substantive edit from a prior skill version and historical context, scoring operation, component, and patch or summary similarity.It supports chronological-per-skill or repository-held-out splits, but historical context makes the task hindsight-based: high match does not establish prospective discovery or deployable autonomous curation.Source: E A Replay Protocol for Skill Curation · p. 18
Loading 2609.05677v1…