PaperlayerView exact revision on arXiv

Source-linked AI summary

Repo2Skill-Evo: Repository Skills Go Stale in Silence

Chenyuan Duan, Ge Shi, Zineng Mao, Ge Zhang, Hao Liang, Yinzhu Piao, Yuchen Wu, Zhixin Yao, Kaiyu Huang, Wenhao Huang, Linzhuang Sun, Shen Yan, Wentao Zhang

arXiv:2608.21964v1cs.AIcs.SE

TL;DR

Whether repository-specific agent skills can stay reliable across releases remains unclear, as outdated guidance can remain retrievable without an explicit signal. Repo2Skill-Evo tests patch-guided skill maintenance across release transitions, finding that six frontier agents achieve only 29.9%–69.7% avg@3 macro F1.

  • Problem

    Whether repository-specific agent skills remain reliable across releases is unclear, even though version-specific guidance can remain retrievable after becoming outdated.

    Source: §1 · pp. 1–2

  • Method

    Repo2Skill-Evo frames release transitions as patch-grounded maintenance: agents update obsolete V1 skill content while preserving guidance that remains valid.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    29.9%–69.7% avg@3 macro F1: six frontier agents achieved limited maintenance performance across 57 repositories and 105 selected transitions.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest treating repository skills as versioned knowledge assets; oracle skill-file localization alone leaves substantial residual errors.

    Source: §6 · p. 12

    Because the benchmark is a curated staleness-positive challenge set, results characterize difficulty on releases known to affect skills, not staleness prevalence across releases.

    Source: §5 · p. 12

Abstract

from arXiv · show

Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.

1 Introduction

Repository skills encode version-specific procedural knowledge, but can remain loadable and retrievable after a release even when their guidance no longer matches the repository. Repo2Skill-Evo studies maintaining these assets and finds that agents’ release-to-release updates remain unreliable.

1 Introduction

  • Repository-level agents need procedural knowledge about APIs, scripts, module configuration, and release conventions, which is costly and unreliable to extract from scratch for each task.Agent skills externalize reusable knowledge into on-demand units, and prior work reports downstream performance improvements [16] [41].Source: §1 · p. 1
  • Silent staleness occurs when release-specific skill guidance remains retrievable after repository evolution, but no explicit signal marks that it is outdated.The study treats repository-grounded skills as versioned knowledge assets and focuses on removing or revising stale content while preserving guidance that remains valid.Source: §1 · pp. 2–3
  • Across ten randomly sampled repositories, skills yield the largest absolute gains on tasks with the lowest baseline utility.This sample suggests maintenance may matter most where agents have limited repository knowledge without external support.Source: §1 · p. 2
  • 29.9%–69.7% avg@3 macro F1 is achieved by six frontier agents across 57 repositories and 105 selected release transitions, each containing patch-verified stale V1 skill content.The median stale content per transition is 92 lines. Incomplete coverage of affected skill files leaves stale content untouched, while broader edits are associated with higher recall but lower patch-grounded precision; oracle file localization improves performance on the 20 hardest transitions but leaves substantial errors.Source: §1 · p. 3
  • Repo2Skill-Evo introduces a patch-grounded benchmark with fixed V1 skill sets and patch-verified obsolete-line annotations across 57 repositories and 105 official release transitions.Its maintenance task gives an agent the V1 skills and official V1-to-V2 patch, and evaluates removal or revision of obsolete content while penalizing edits to other V1 skill lines.Source: §1 · pp. 2–3

2 Related Work

Prior work studies repository-level agents, software evolution, and skill construction and use, but leaves open whether repository skills stay valid across releases. Repo2Skill-Evo addresses this gap by studying patch-conditioned maintenance of external, repository-grounded skills.

2.1 Repository-Level Agents and Software Evolution

  • Repository-agent and software-evolution work studies repository repair, navigation, cross-file context, and code or client migrations, whereas Repo2Skill-Evo maintains agent-facing repository-use knowledge.SWE-bench evaluates issue repair, SWE-agent studies agent–computer interfaces [35], and RepoBench emphasizes cross-file context [18]; software-evolution work includes migrations and release-level upgrades [2] [13] [14].Source: §2.1 · p. 3

2.2 Agent Skill Construction and Utility

  • Skill research studies construction, retrieval, and utility, but most evaluations use fixed snapshots and do not show repository skills remain valid across releases.Skills can be generated from source corpora [41] or improved through execution and verifier feedback [21] [37], with heterogeneous utility results [16] [41]. SKILL.md and PyTorch skills illustrate practical repository-local artifacts.Source: §2.2 · p. 3

2.3 Agent Skill Lifecycle and Maintenance

  • Unlike prior skill-lifecycle work on execution-triggered repair, contracts, libraries, and iterative revision, Repo2Skill-Evo studies patch-conditioned release maintenance while preserving unaffected guidance.Skills can drift like code comments, and silent staleness may leave no execution-failure signal for runtime repair [23]. Related systems address contract violations [5], skill-library debt and retrieval drift [26] [38], or iterative revision [17]; unlike skill internalization [22], Repo2Skill-Evo keeps the skill external and release-tied.Source: §2.3 · pp. 3–4

3 Repo2Skill-Evo: Maintaining Repository Skills across Releases

Repo2Skill-Evo studies whether agents can maintain repository-specific skills across official releases: it fixes a traceable V1 skill set, then asks agents to update obsolete guidance while retaining valid content.

3.1 Repo2Skill: Distilling a Traceable Skill Set

  • Repo2Skill creates a fixed, source-traceable V1 skill set by distilling repository procedures into packages with provenance metadata.Candidates are checked for source-path and symbol resolvability, then an expert selects, revises, and verifies one; the grounding gate does not establish semantic correctness or procedural coverage.Source: §3.1 · p. 5

3.2 Maintenance Task

  • Each maintenance instance gives an agent the fixed V1 skill set, official V1-to-V2 patch, and tool interface, and requires an updated skill set.Successful maintenance entails localizing patch-affected guidance, removing or revising obsolete knowledge, and preserving content that remains valid.Source: §3.2 · p. 6

3.3 Agent Scaffold

  • The scaffold compares agents in a shared, minimal single-agent environment with six generic tools, a common prompt and inputs, and a fixed turn budget.Runs that exhaust the budget still have their partial skill sets evaluated.Source: §3.3 · p. 6

3.4 Evaluation Metric

  • The headline metric scores removal or revision of patch-verified obsolete V1 lines while penalizing changes to lines outside the gold obsolete set.The gold set is manually verified against direct patch evidence; precision, recall, and F1 balance stale-content coverage against edit restraint, while a complementary NL Judge evaluates replacement content and final-state quality.Source: §3.4 · pp. 6–7
  • The gold obsolete set contains original V1 skill lines that the official release patch directly shows are no longer valid under V2.The set covers patch-exposed changes across prose, examples, tables, and scripts, and is constructed through automated candidate discovery followed by line-level manual verification.Source: §3.4 · p. 6
  • The complementary NL Judge scores V2 API correctness, file consistency, retained information, prompt adherence, and whether prose describes the post-release state.Each dimension receives 0–2 points; the five scores sum to 0–10, and structurally invalid skill sets receive zero.Source: §3.4 · p. 7

4 Experiments

The experiments show that fixed repository skills provide useful, compact knowledge, but agents do not reliably keep them current across releases. The study also links maintenance failures to incomplete localization and poorly targeted or unfinished edits.

4.1 Setup

  • The corpus contains 105 selected release transitions from 57 repositories and 12,217 patch-verified stale V1 skill lines.It spans agent and RAG frameworks, training and inference infrastructure, data and classic ML, compiler/kernel/systems software, and general-purpose libraries and applications; observed difficulty is based on mean avg@3 F1 across six models.Source: §4.1 · p. 7, Figure 3 · p. 8

4.2 Motivating Study: Fixed Skills Provide Meaningful Repository Utility

  • Across ten repositories, skill-only raises mean utility from 5.88 to 8.68, a 2.80-point gain, nearly matching source-only at 8.64.The gain is 3.98 points for low-baseline repositories (4.48 to 8.46) versus 1.61 points for high-baseline repositories, in this sample.Source: §4.2 · p. 8
  • Skill-only achieves mean utility of 8.68 using 51,821 mean tokens and 3.9 iterations, versus source-only's 8.64 using 272,620 tokens and 10.8 iterations.With source already available, adding skills raises mean utility from 8.64 to 9.01 while modestly reducing mean token usage.Source: §4.2 · pp. 8–9

4.3 Agents Do Not Reliably Maintain Repository Skills across Releases

  • Claude-opus-4.6 leads at 69.7% avg@3 macro F1, while GPT-5.4 has the highest avg@3 macro recall at 74.6% but only 58.8% avg@3 macro F1.GLM-5.1 reaches 64.3% avg@3 macro F1; Kimi-K2.5, Doubao-Seed2-pro, and MiniMax-M2.5 score 47.0%, 39.8%, and 29.9%. Best@3 raises scores, but even the strongest model reaches only 76.1%; the complementary NL Judge also ranks Claude-opus-4.6 first, GPT-5.4 second, and GLM-5.1 third. Across transitions, 85 of 105 fall below the Easy threshold defined by mean avg@3 F1 across the six models.Source: §4.3 · pp. 9–10, Table 2 · p. 9
  • GPT-5.4's 74.6% avg@3 macro recall comes with 55.7% macro precision, while MiniMax-M2.5's 74.9% precision accompanies removal of only 22.6% of verified stale content.The profiles show opposing errors: broader edits can raise recall while lowering precision, whereas high precision can coexist with substantial under-editing.Source: §4.3 · p. 9

4.4 Failure Diagnosis: Coverage and Editing Bottlenecks

  • Affected-file coverage correlates with F1 (r = 0.650) and recall (r = 0.701); full-coverage runs average 67.3% F1 versus 41.0% for partial-coverage runs.The associations persist after removing additive model and transition effects, linking incomplete localization with missed stale content.Source: §4.4 · p. 10
  • On the hardest-20 subset, oracle file paths raise avg@3 macro F1 from 32.8% to 45.0%, a 12.2-point mean transition-level gain, but the best model reaches only 62.3%.Recall improves for every model, yet file localization alone is insufficient because agents must still identify affected claims and select appropriate edits.Source: §4.4 · pp. 10–11, Table 5 · p. 11
  • GPT-5.4 pairs high affected-file coverage with a 2.84 removal load and the highest off-target editing rate, while GLM-5.1's high coverage coexists with 41.3% budget exhaustion.Kimi-K2.5, Doubao-Seed2-pro, and MiniMax-M2.5 have lower coverage and removal loads; Kimi-K2.5 and MiniMax-M2.5 also often exhaust the turn budget. Claude-opus-4.6 has a more balanced diagnostic profile, illustrating bottlenecks in both localization and sufficiently complete, targeted editing.Source: Table 4 · p. 10, §4.4 · pp. 11–12

5 Limitations

The evaluation characterizes maintenance difficulty on releases known to affect skills, not staleness prevalence or downstream task utility.

5 Limitations

  • The curated staleness-positive set limits conclusions to known skill-affecting releases, and patch-grounded fidelity does not establish downstream utility or staleness prevalence.The NL Judge offers a complementary final-state assessment, but maintained skill sets are not rerun on downstream repository tasks.Source: §5 · p. 12

6 Conclusion · Appendix

Repository-specific skills can silently become misleading as repositories evolve while remaining loadable and retrievable. Across the evaluated release transitions, even the strongest agent falls short of reliable maintenance, supporting the treatment of skills as versioned knowledge assets that require upkeep.

6 Conclusion

  • The paper calls silent staleness the failure mode in which repository changes make previously correct guidance misleading while it remains loadable and retrievable.Source: §6 · p. 12
  • 69.7% avg@3 macro F1 is the strongest result among six evaluated agents across 57 repositories and 105 selected release transitions.Repo2Skill-Evo evaluates skill updates against patch-grounded targets built from fixed V1 skill sets and official release patches.Source: §6 · p. 12
  • Repo2Skill-Evo operationalizes release-driven skill maintenance by pairing fixed V1 skill sets with official release patches and evaluating updates against patch-grounded targets.Source: §6 · p. 12
  • Trace and output analyses identify incomplete localization of affected skill files and imperfect edit selection as complementary maintenance bottlenecks.Source: §6 · p. 12
  • Oracle skill-file localization improves performance but leaves substantial residual errors, showing that localization alone is insufficient.Source: §6 · p. 12
  • The findings suggest repository skills should carry the release they were distilled from and be maintained as the repository evolves.Source: §6 · p. 12

A Skill-Gain Utility Study

The study tests the utility of fixed V1 skills under controlled source-access conditions, groups repositories descriptively by baseline utility, and reports execution costs.

Experimental protocol

  • GPT-5.4 evaluates ten sampled repositories in a 2 × 2 ablation over access to the fixed V1 skill set and repository checkout.The four conditions are baseline (neither source nor skill), skill-only, source-only, and source+skill; they share the task, rubric, scaffold, and budget. Skill ∆ compares skill-only with baseline, source ∆ compares source-only with baseline, and skill ∆src compares source+skill with source-only.Source: A Skill-Gain Utility Study · p. 17

Baseline-utility grouping

  • Repositories are split at the median of baseline utility as a coarse proxy for repository-specific prior knowledge, but the grouping is interpreted descriptively.Baseline utility may also reflect task difficulty, and lower-baseline tasks have more headroom on the bounded 0–10 scale; Table 6 gives the per-repository scores used for the grouping.Source: A Skill-Gain Utility Study · p. 17, Table 6 · p. 17

Execution cost

  • Adding skills to source access yields a +0.37-point utility gain while reducing mean tokens from 272,620 to 237,020 and mean iterations from 10.8 to 9.3.Each condition aggregates 30 runs: three runs on each of ten repositories.Source: A Skill-Gain Utility Study · p. 17, Table 7 · p. 18

B Maintenance Dataset Construction … C Evaluation Protocol and Diagnostics

Repo2Skill-Evo builds a traceable, staleness-positive corpus from official release transitions and fixed V1 skill sets, with obsolete content grounded in the corresponding V1-to-V2 patch. The construction keeps patch evidence and gold labels out of V1 skill creation and maintenance-agent inputs.

B Maintenance Dataset Construction

  • The corpus combines official release transitions, materialized version and commit data, expert-verified V1 skills, and patch-grounded obsolete-line sets.Transitions with no patch-supported obsolete lines are excluded, leaving a staleness-positive corpus.Source: B Maintenance Dataset Construction · p. 18
  • Gold annotations are stored separately for evaluation, and maintenance agents receive neither the obsolete set nor its annotation-side representation.This keeps the gold labels from being available as maintenance-task inputs.Source: B Maintenance Dataset Construction · p. 18

B.1 Transition Selection and Construction

  • Transitions favor actively maintained projects and releases affecting interfaces, defaults, layouts, commands, configurations, or documented behavior.Each instance records version tags, commit hashes, and the public source diff for traceability; selected transitions include releases after some models’ knowledge cutoffs, without assuming repository absence from pretraining.Source: §B.1 · p. 18

B.2 Fixed V1 Skill Set

  • V1 skill sets are generated and revised using only pre-release repository evidence, with expert review covering the complete package and its factual and structural grounding.The release patch and gold obsolete labels remain outside V1 skill construction.Source: §B.2 · p. 18, B Maintenance Dataset Construction · p. 18

B.3 Gold Obsolete Set Construction

  • Automatic matching of skill references to changed patch lines assists candidate discovery, but experts inspect each transition and manually verify every gold removal target against direct patch evidence.Review removes candidates still valid under V2 and adds stale semantic claims or lines missed by the automated pass.Source: §B.3 · p. 19
  • Patch-supported changes can make dependent instructions, examples, tables, and scripts obsolete alongside the line naming the changed repository element.For Flask 2.2.5-to-2.3.0, removal of environment-mode settings makes related reference rows, prose, and script lines obsolete.Source: §B.3 · p. 19

C.1 Behavioral Diagnostics

The diagnostics measure whether agents attend to gold-affected skill files and relate that coverage to maintenance outcomes. Coverage is positively associated with F1 and recall, while removal-load strata distinguish insufficient from broad editing.

C.1 Behavioral Diagnostics

  • Affected-file coverage is the fraction of gold-affected skill files that agents directly read, explicitly target in traces, or modify in maintained outputs.Paths appearing only in tool-returned search results do not count as localized; coverage uses traces and maintained outputs, while other diff-based diagnostics use the SV1 →SV2 diff.Source: §C.1 · p. 19
  • Budget-exhausted runs still acting on the workspace at turn 50 contribute partial outputs, while a standalone finish on turn 50 counts as completed.Source: §C.1 · p. 19
  • Coverage remains positively associated with F1 (r = 0.477) and recall (r = 0.455) after averaging runs by model–transition pair and residualizing model and transition effects.The associations are attenuated relative to pooled run-level correlations, indicating that the relationship holds within models as well as across them.Source: §C.1 · pp. 19–20
  • Runs with removal load in [0.75, 1.25) average 77.3% F1 versus 25.0% below 0.5; runs at or above 1.25 achieve 80.8% recall but 44.7% precision.These patterns distinguish insufficient editing from broad editing that improves stale-content coverage at the expense of edit selectivity.Source: §C.1 · p. 20

C.2 Oracle Skill-File Localization Ablation · C.3 Maintenance and Judge Instructions · D Additional Results and Corpus Statistics

The appendix details a hardest-20 oracle localization ablation and specifies the maintenance and judging instructions used to evaluate skill updates.

C.2 Oracle Skill-File Localization Ablation

  • The oracle ablation selects 20 eligible transitions with at least 20 gold obsolete lines, ranked from lowest mean baseline avg@3 F1, with at most one transition per repository.This yields the hardest-20 subset without allowing multiple releases from one repository to dominate it.Source: §C.2 · p. 20
  • Across the six models, 360 oracle runs preserve the baseline scaffold, tools, 50-turn budget, patch, and V1 skills; only the localization hint changes.Agents must still inspect the identified files, trace release changes, and determine required updates; each model runs three times per transition.Source: §C.2 · p. 20
  • Oracle runs are compared with baseline runs on the same 20 transitions using the identical patch-grounded removal metric.Per-model confidence intervals bootstrap the 20 transition-level ∆F1 values; the overall interval bootstraps transition means after averaging ∆F1 across models within each transition.Source: §C.2 · p. 20
  • Baseline and oracle precision aggregate over transitions where precision is defined in each condition, while ∆P averages paired oracle-minus-baseline transition differences.Consequently, ∆P need not equal the difference between the displayed aggregate precision values.Source: §C.2 · p. 20

C.3 Maintenance and Judge Instructions

  • Baseline maintenance agents receive only the fixed V1 skill set and official release patch; gold-affected paths are supplied only in the oracle localization ablation..Source: §C.3 · p. 20
  • Maintenance instructions require minimal patch-grounded updates to affected prose, scripts, and references while preserving unaffected package structure unless changes are required.Agents describe the post-release state directly rather than narrating migration, and work within a fixed 50-turn budget whose total and remaining turns are reported throughout.Source: §C.3 · p. 20
  • The NL Judge assesses outputs using the official patch, fixed V1 skills, produced V2 skills, and limited run metadata, independently of G∆ and patch-grounded removal scores.The full rubric and instructions are included in the released materials.Source: §C.3 · pp. 20–21

D.1 Full NL Judge Results

The section reports full NL Judge results across five dimensions, with form-gated outputs counted as zero and scores averaged over 315 runs per maintenance model.

D.1 Full NL Judge Results

  • Table 10 reports five 0–2 dimensions—API correctness, Files consistency, Info retention, Prompt adherence, and NL current-state prose correctness—averaged over 315 runs per model; form-gated outputs score zero.Sum is the 0–10 total across the dimensions, and Invalid counts form-gated runs.Source: Table 10 · p. 21, §D.1 · p. 21

D.2 Statistical Robustness of the Main Results · D.3 Obsolete-Set Size Distribution · D.4 Corpus Composition and Difficulty

Repository-clustered comparisons preserve separations among the top models, while alternative aggregation and run-to-run variability qualify the headline results. Most selected transitions contain substantial obsolete skill content, and observed difficulty and outcomes vary across the corpus.

D.2 Statistical Robustness of the Main Results

  • 6.1 for Claude-opus-4.6 and 14.7 for MiniMax-M2.5 bracketed average run-to-run F1 standard deviation, which was generally larger for lower-scoring models.The other values were 9.0 for GLM-5.1, 11.6 for GPT-5.4, 12.0 for Kimi-K2.5, and 11.9 for Doubao-Seed2-pro.Source: §D.2 · p. 21
  • 5.4-point Claude-opus-4.6-over-GLM-5.1, 5.4-point GLM-5.1-over-GPT-5.4, and 11.9-point GPT-5.4-over-Kimi-K2.5 differences all had 95% confidence intervals excluding zero.The respective intervals were [+2.8, +8.0], [+2.0, +8.7], and [+8.1, +15.8].Source: §D.2 · p. 21, Table 11 · p. 22
  • +0.9 to +3.4 points separated repository-macro scores from transition-macro scores, while model ordering was preserved.Repository-macro averaging first averages transition-level avg@3 F1 within each repository, then across the 57 repositories.Source: §D.2 · p. 21, Table 11 · p. 22

D.3 Obsolete-Set Size Distribution

  • 12,217 obsolete V1 skill lines spanned the 105 transitions, ranging from 5 to 375 per transition, with a mean of 116.4 and quartiles of 46, 92, and 156.Only 7 transitions had fewer than 20 obsolete lines, while 48 had at least 100.Source: Table 12 · p. 22, §D.3 · p. 22

D.4 Corpus Composition and Difficulty

  • 25 transitions were Hard, 60 Medium, and 20 Easy under avg@3 thresholds of <0.40, 0.40–<0.65, and ≥0.65.Mean tractability was 29.9% for Hard, 53.0% for Medium, and 74.5% for Easy; strata were descriptive, and neither patch size nor obsolete-set size defined them.Source: §D.4 · p. 22
  • 57 repositories were assigned to five domains, with each transition inheriting its repository’s domain.The groups were agent and RAG frameworks (19 repositories, 33 transitions), training and inference infrastructure (15, 32), data and classic ML (9, 17), compiler, kernel, and systems (9, 16), and general-purpose libraries and applications (5, 7).Source: §D.4 · p. 23
  • 53.8% was the agent-and-RAG domain mean F1, while infrastructure had 46.4% and was concentrated in Hard transitions.Other domain means were 57.8% for data and classic ML, 51.6% for compiler, kernel, and systems, and 49.5% for general-purpose libraries and applications; infrastructure contributed 9 of 25 Hard transitions but 2 of 20 Easy transitions. Domain assignments were descriptive and did not affect G∆, maintenance runs, or headline metrics.Source: §D.4 · p. 23

Related papers

Loading 2608.21964v1…