Source-linked AI summary
Repo2Skill-Evo: Repository Skills Go Stale in Silence
Chenyuan Duan, Ge Shi, Zineng Mao, Ge Zhang, Hao Liang, Yinzhu Piao, Yuchen Wu, Zhixin Yao, Kaiyu Huang, Wenhao Huang, Linzhuang Sun, Shen Yan, Wentao Zhang
TL;DR
Repository skills encode useful but version-specific procedural knowledge that may silently become obsolete after releases. Repo2Skill-Evo tests whether agents can update fixed V1 skills using official V1-to-V2 patches while preserving valid guidance, and finds unreliable maintenance across 57 repositories and 105 transitions. Six frontier agents achieve only 29.9%–69.7% avg@3 macro F1, with failures concentrated in affected-file localization and edit selection.
Problem
It is unclear whether the performance benefits of repository skills remain durable when releases invalidate their version-specific guidance without an explicit signal.
Method
Repo2Skill-Evo pairs fixed V1 skill sets with official V1-to-V2 release patches and evaluates agents on patch-grounded removal or revision of obsolete content while preserving valid guidance.
Results
29.9%–69.7% avg@3 macro F1 was achieved by six frontier agents across 57 repositories and 105 selected release transitions, with localization and edit-selection bottlenecks.
Takeaways & Limitations
Repository skills should be treated as versioned knowledge assets whose maintenance remains unreliable even for frontier agents.
Takeaways & Limitations
The evaluation uses a curated staleness-positive challenge set and measures patch-grounded maintenance fidelity without rerunning maintained skills on downstream repository tasks.
Abstract
from arXiv · showhide
Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.
1 Introduction
Repository skills preserve version-specific procedural knowledge, but repository releases can silently invalidate that guidance. Repo2Skill-Evo frames keeping skills current as a patch-grounded maintenance task and finds that agents struggle with both locating affected files and selecting precise edits.
- Motivation: Repository-specific procedural knowledge includes APIs, scripts, module configuration, and conventions expected by the current release.Extracting this knowledge from scratch is costly because it requires cross-file retrieval and structural understanding.
- Motivation: Skills externalize reusable procedural knowledge into retrievable units that can improve downstream agent performance across tasks and releases.Their value comes from compact, on-demand access to repository guidance.
- Problem: After a release, version-specific skill guidance may remain loadable and retrievable while no longer matching the repository, a failure mode called silent staleness.The mismatch has no explicit signal, so outdated guidance may continue to be retrieved and followed.
- Approach: Repo2Skill-Evo gives a maintenance agent a fixed V1 skill set and official V1-to-V2 patch, requiring obsolete content removal or revision while preserving valid guidance.The maintained object is agent-facing guidance derived from the repository rather than repository code itself.
- Results: 29.9%–69.7% avg@3 macro F1 was achieved by six frontier agents across 57 repositories and 105 selected release transitions.Every selected transition contained patch-verified stale content, with a median of 92 stale lines.
- Results: Incomplete coverage leaves stale content untouched, while broader edits are associated with higher recall but lower patch-grounded precision.Oracle localization improves performance on the 20 hardest transitions, but residual errors show that localization alone is insufficient.
2 Related Work
Prior work studies repository-level agents, software evolution, and skill construction or use, but Repo2Skill-Evo focuses on maintaining agent-facing repository knowledge across releases. It isolates release-conditioned skill maintenance rather than source-code migration or fixed-snapshot skill evaluation.
- 2.1 Repository-Level Agents and Software Evolution: Repository-level agent benchmarks evaluate codebase understanding, issue resolution, localization, repair, and repository navigation through tools and interfaces.Examples include SWE-bench, SWE-agent, RepoBench, and RepoMirage.
- 2.1 Repository-Level Agents and Software Evolution: Software-evolution research covers library migration, API refactoring, migration detection, release-note tasks, continuous-integration loops, and chained release upgrades.These works primarily center on source code or downstream client usage.
- 2.2 Agent Skills: Skill research studies lifecycle stages including representation, acquisition, retrieval, evolution, generation, execution feedback, and verifier feedback.Utility results are heterogeneous: curated skills can help, whereas self-generated or mismatched skills do not reliably help.
- 2.2 Agent Skills: Most skill work evaluates construction or use within a fixed environment snapshot, leaving post-release validity insufficiently established.Practical repository-local skill formats and shipped skills demonstrate use but not continued validity after repository advancement.
- 2.3 Agent Skill Lifecycle and Maintenance: Natural-language artifacts can drift out of sync with code, while repository skills carry direct operational consequences because autonomous agents can retrieve and execute them.Runtime feedback can support repair after observed execution failure, but silent staleness may provide no error log.
- 2.3 Agent Skill Lifecycle and Maintenance: Repo2Skill-Evo isolates release-conditioned maintenance by mapping an official patch onto a fixed repository-grounded skill set while preserving unaffected guidance.Unlike skill internalization, the maintained object remains an external artifact whose validity is tied to a software release.
3 Repo2Skill-Evo: Maintaining Repository Skills across Releases
Repo2Skill-Evo freezes a traceable V1 skill set, then treats each release transition as targeted maintenance: locate affected guidance, remove obsolete content, and preserve valid knowledge. Its evaluation combines patch-grounded removal scoring with a complementary semantic judge.
- 3.1 Repo2Skill: Distilling a Traceable Skill Set: Repo2Skill distills repository-specific procedures into grounded skill packages, recording source paths and symbols for later tracing.The resulting skill set is frozen before maintenance evaluation begins.
- 3.1 Repo2Skill: Distilling a Traceable Skill Set: The distillation pipeline analyzes repository symbols, builds read-maps and evidence collections, drafts skills, validates references, and finishes after validation succeeds.Validation checks source paths and symbols against the V1 repository and rejects candidates exceeding an unverifiable-reference threshold.
- 3.2 Maintenance Task: Each maintenance instance supplies a fixed V1 skill set, the official V1-to-V2 patch, and a tool interface, producing an updated V2 skill set.The task evaluates the set-level transformation from SV1 to SV2.
- 3.2 Maintenance Task: Successful maintenance requires patch-to-skill localization, obsolete-knowledge removal, and information retention.Agents must update stale references, signatures, examples, tables, and prose without discarding guidance that remains valid.
- 3.3 Maintenance Scaffold: The benchmark uses a minimal single-agent environment in which every agent receives the same V1 skills, release patch, SOP prompt, tools, and turn budget.The scaffold provides generic shell, discovery, inspection, editing, and termination tools without task-specific localization or repair.
- 3.4 Evaluation Metric: The headline metric rewards removal or revision of patch-verified obsolete V1 lines while penalizing edits outside the patch-supported obsolete set.New V2 additions are evaluated separately by a complementary NL Judge because valid replacements may have multiple realizations.
4 Experiments
The experiments test repository-skill utility and release maintenance across a corpus of 105 transitions from 57 repositories. Skills provide compact, task-relevant knowledge, but agents maintain them unreliably because localization and edit selection remain difficult.
- Setup: 105 release transitions across 57 repositories are evaluated, spanning agent and RAG frameworks, infrastructure, machine-learning libraries, systems software, and general-purpose libraries.Observed difficulty is classified from the mean avg@3 F1 across six maintenance agents.
- Motivating Study: Skill-only raises mean utility from 5.88 to 8.68 across ten repositories, nearly matching source-only at 8.64.The fixed skills provide task-relevant repository knowledge that models do not consistently recover without external context.
- Motivating Study: Low-baseline repositories gain 3.98 utility points from skill-only, compared with 1.61 points for high-baseline repositories.The grouping is descriptive and reflects larger absolute gains where baseline performance is lower.
- Motivating Study: Skill-only achieves mean utility of 8.68 versus 8.64 for source-only while using 51,821 versus 272,620 mean tokens and 3.9 versus 10.8 iterations.When source is already available, adding skills raises mean utility from 8.64 to 9.01 while modestly reducing mean token usage.
- Maintenance Results: 29.9%–69.7% avg@3 macro F1 is achieved by the six agents, with Claude-opus-4.6 highest at 69.7% and four models below 60%.GPT-5.4 has the highest avg@3 macro recall at 74.6% but macro precision of 55.7%, yielding 58.8% avg@3 macro F1.
- Maintenance Results: Broader edits are associated with higher recall but lower patch-grounded precision, while high precision can coexist with substantial under-editing.GPT-5.4 follows the high-recall, low-precision pattern; MiniMax-M2.5 reaches 74.9% precision but removes or revises only 22.6% of verified stale content.
- Maintenance Results: Best@3 raises macro F1 by 6.4 points for Claude-opus-4.6 and 9.6–16.1 points for the remaining models, yet the strongest model reaches only 76.1%.The best-of-three measure reflects attainable repeated-attempt performance rather than single-attempt reliability.
5 Limitations
The evaluation uses a curated staleness-positive challenge set and measures patch-grounded maintenance fidelity rather than downstream utility. Maintained skills are not rerun on future repository tasks.
- Scope: The dataset is a curated staleness-positive challenge set, so results characterize maintenance difficulty on releases known to affect skills rather than staleness prevalence across releases.The prevalence of staleness across releases is treated as a separate question.
- Scope: The evaluation measures patch-grounded maintenance fidelity, and maintained skill sets are not rerun on downstream repository tasks.The NL Judge assesses final-state quality, but future-work utility remains unevaluated.
6 Conclusion
Repository skills are versioned knowledge assets whose validity does not persist automatically as repositories evolve. Repo2Skill-Evo finds that agents do not reliably maintain them, with failures stemming from both incomplete localization and imperfect edit selection.
- Repository-grounded skills can become misleading after repository evolution while remaining loadable and retrievable; this release-driven failure mode is called silent staleness.
- Incomplete localization leaves stale content untouched, while broader edits are associated with higher recall but lower patch-grounded precision.Oracle skill-file localization improves performance but leaves substantial residual errors.
- Repository skills should be treated as versioned knowledge assets that carry their release provenance and require maintenance as repositories evolve.
7 Contributions
The listed paper authors are Hao Liang, Yinzhu Piao, Yuchen Wu, Zhixin Yao, Kaiyu Huang, Wenhao Huang, and Linzhuang Sun.
- Hao Liang and Yinzhu Piao are listed among the paper’s authors.
- Yuchen Wu and Zhixin Yao are listed among the paper’s authors.
- Kaiyu Huang, Wenhao Huang, and Linzhuang Sun are listed among the paper’s authors.
A Skill-Gain Utility Study
The utility study compares agent performance with and without repository source and skills under a controlled four-condition protocol. Adding skills to source access improves utility while slightly reducing execution cost.
- Experimental protocol: The study evaluates ten repositories using a 2 × 2 ablation over access to fixed V1 skills and the V1 repository checkout.Each repository is represented once, with GPT-5.4 performing three runs in each condition.
- Experimental protocol: The four conditions are baseline, skill-only, source-only, and source+skill, differing only in source and skill access.
- Experimental protocol: The contrasts measure skill utility without source access, source utility without skills, and the incremental utility of skills when source is available.
- Baseline-utility grouping: Repositories are split at the median into five low-baseline and five high-baseline groups, using baseline utility as a coarse proxy for repository-specific prior knowledge.The authors note that baseline utility may also reflect task difficulty.
- Results and execution cost: +0.37-point utility gain for source+skill over source-only accompanies a reduction in mean tokens from 272,620 to 237,020 and mean iterations from 10.8 to 9.3.These values aggregate the four access conditions across ten repositories and three runs per repository and condition.
B Maintenance Dataset Construction
Repo2Skill-Evo constructs release-transition instances with fixed V1 skills and patch-grounded obsolete-content annotations. The dataset emphasizes traceability, isolation from evaluation labels, and direct evidence of skill invalidation.
- Dataset construction: The dataset selects official release transitions with publicly available source diffs and records repository versions, commit hashes, and release patches.
- Dataset construction: Candidate V1 skill sets are generated from pre-release repositories, then expert-selected, revised, and verified for maintenance evaluation.
- Gold obsolete annotations: The gold obsolete set G∆ contains V1 skill lines directly shown invalid under V2 by the release patch, and transitions with G∆= ∅ are excluded.
- Gold obsolete annotations: Gold annotations remain separate from maintenance instances, so agents receive neither G∆ nor its annotation-side representation.
- Scope and validity: Selection favors actively maintained projects and release diffs affecting interfaces, defaults, file layouts, commands, configurations, or documented repository behavior.
- Scope and validity: The corpus spans releases that postdate some models’ knowledge cutoffs, while performance depends on aligning fixed V1 skills with a specific V1 →V2 patch.
- Candidate discovery and verification: Automated candidate discovery matches skill-referenced symbols, paths, commands, and configuration keys against changed patch lines before manual inspection of each transition.
- Obsolescence categories: Obsolete content includes dependent instructions, examples, table entries, and script statements when the patch-supported change invalidates them.
C.1 Behavioral Diagnostics
Behavioral diagnostics connect maintenance quality to file coverage and editing load. The oracle ablation isolates localization by adding only an affected-file hint while preserving the original maintenance setup.
- Coverage diagnostics: Affected-file coverage counts gold-affected V1 skill files that the agent directly reads, explicitly targets, or modifies.Paths appearing only in tool-returned search results do not satisfy localization.
- Coverage diagnostics: Coverage remains positively associated with F1 (r = 0.477) and recall (r = 0.455) after residualizing model and transition fixed effects.The robustness analysis averages runs within model–transition pairs before computing correlations.
- Removal-load diagnostics: 77.3% F1 occurs at removal load [0.75, 1.25), versus 25.0% below 0.5.At removal load at or above 1.25, recall reaches 80.8% while precision is 44.7%, distinguishing insufficient from broad editing.
- Oracle localization ablation: The oracle ablation selects 20 hardest transitions with at least 20 gold obsolete lines, allowing at most one transition per repository.Selection is based on ascending mean baseline avg@3 F1 across the six maintenance models.
- Oracle localization ablation: 360 oracle runs retain the original scaffold, tools, budget, patch, and V1 skills, making the oracle localization hint the only intervention.Agents must still inspect identified files, trace release changes, and determine the updates.
D.1 Full NL Judge Results
The NL Judge reports five dimensions of maintenance quality across all evaluated runs, with form-gated outputs receiving zero scores.
- Scoring protocol: 315 runs per maintenance model contribute to the averaged dimension-level NL Judge results.Form-gated outputs contribute zero to every dimension and to the Sum score.
- Scoring dimensions: The NL Judge scores API correctness, Files consistency, Info retention, Prompt adherence, and natural-language current-state prose correctness.Each dimension ranges from 0 to 2, and their sum forms a 0–10 total.
D.2 Statistical Robustness of the Main Results
Robustness analyses show that the headline model ordering persists under repository-level aggregation and clustered bootstrap comparisons, while run-to-run variability is larger for lower-scoring models. The corpus also contains substantial obsolete skill content across diverse repository domains and difficulty levels.
- Aggregation robustness: Repository-macro scores preserve the model ordering and differ from transition-level scores by only +0.9 to +3.4 points.Repository-macro averaging first aggregates transition scores within each repository, then averages across 57 repositories.
- Clustered comparisons: All three paired repository-clustered bootstrap intervals exclude zero: the top-four adjacent gaps are 5.4, 5.4, and 11.9 points.The reported comparisons are Claude-opus-4.6 over GLM-5.1, GLM-5.1 over GPT-5.4, and GPT-5.4 over Kimi-K2.5.
- Run-to-run variability: Average three-run F1 standard deviations range from 6.1 for Claude-opus-4.6 to 14.7 for MiniMax-M2.5.Run-to-run variability is generally larger for lower-scoring models.
- Corpus scope: The 105 transitions contain 12,217 patch-verified obsolete V1 skill lines, ranging from 5 to 375 per transition.The mean is 116.4, with quartiles of 46, 92, and 156; 48 transitions contain at least 100 obsolete lines.
- Difficulty stratification: Observed difficulty classifications comprise 25 Hard, 60 Medium, and 20 Easy transitions, based on mean six-model avg@3 F1.The thresholds are Hard <0.40, Medium [0.40, 0.65), and Easy ≥0.65, and are descriptive only.
- Domain composition: Mean F1 varies by domain from 46.4% in training and inference infrastructure to 57.8% in data and classic ML.Training and inference infrastructure contributes 9 of 25 Hard transitions but only 2 of 20 Easy transitions.