Source-linked AI summary
Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries
Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He
TL;DR
Self-evolving skill libraries can suffer silent library drift when skills accumulate without lifecycle management, degrading retrieval and stagnating performance. The paper diagnoses this failure with reproducible ablations and trace-level evidence, then shows that governed skill management delivers a rolling gain of +0.328.
Problem
Library drift is a silent failure mode in which unbounded skill accumulation degrades retrieval precision, injects stale or harmful guidance, and causes performance to stagnate or decline.
Method
The paper combines minimal ablations, append-only per-skill evidence logging, and the Ratchet loop with outcome-driven retirement, a bounded active cap, and a meta-skill authoring prior.
Results
+0.328 rolling gain is achieved by the governed system, while premature retirement produces active harm and the no-injection condition establishes a +0.002 floor.
Takeaways & Limitations
Library drift should be treated as a first-class failure category, with trace-level diagnostics used to detect it before end-task metrics decline and guide safe intervention.
Takeaways & Limitations
Evaluation is limited to MBPP+ hard-100 with Claude Opus 4.7, and broader-domain and cross-model generalization remain unestablished.
Abstract
from arXiv · showhide
Self-evolving skill libraries face a silent failure mode we term \emph{library drift}: unbounded skill accumulation without outcome-driven lifecycle management causes retrieval degradation, false-positive injections, and performance stagnation. Recent evaluation confirms the symptom (LLM-authored skills deliver +0.0pp gain while human-curated ones deliver +16.2pp (SkillsBench)), yet the underlying mechanism has not been isolated. We provide (1) a reproducible trigger: ablations that isolate drift: one disables skill injection (flat floor, +0.002), one imposes premature retirement (active harm, $-$0.019); (2) trace-level diagnostics: an append-only evidence log with per-skill contribution scores, attribution verdicts, and router engagement metrics that make the failure visible before it reaches end-task scores; and (3) a verified fix: a minimal governance recipe (outcome-driven retirement + bounded active-cap + meta-skill authoring prior) that lifts held-out pass@1 from a 0.258 baseline to a late-window mean of 0.584 (rolling gain $+$0.328) on MBPP+ hard-100 over 100 rounds. Eight ablations decompose which governance mechanisms are load-bearing and which are subsumed, providing a concrete playbook for diagnosing library drift in any self-evolving agent.
1. Introduction
Self-evolving skill libraries promise reusable procedural knowledge but can drift when accumulation lacks lifecycle management. This paper defines reproducible triggers, trace-level diagnostics, and a governance recipe for detecting and fixing that failure.
- Motivation: Self-evolving libraries accumulate reusable procedural knowledge without weight updates, but lifecycle management remains largely neglected.SkillsBench reports +0.0pp for LLM-authored skills versus +16.2pp for human-curated skills.
- Motivation: Library drift degrades retrieval precision, injects stale or harmful guidance, and can reduce performance below the no-skill baseline.The failure may remain silent in end-task metrics, especially in multi-step agents where misleading injections compound downstream decisions.
- Contributions: +0.002 and −0.019 bracket drift through no-skill and prematurely retired ablations, respectively.These provide minimal reproductions of a flat floor and active harm.
- Contributions: An append-only evidence log combines per-skill contribution scores, attribution verdicts, and router engagement metrics to expose drift before aggregate pass@1.The diagnostics operate at per-skill granularity rather than only at the final-task level.
- Contributions: +0.328 held-out pass@1 gain comes from Ratchet’s outcome-driven retirement, bounded active-cap, and meta-skill authoring prior.Eight ablations identify which governance mechanisms are load-bearing or safely removable.
- System overview: The Ratchet loop links Router, Solver, Grader, and Capsule inference with Skill Bank, Meta-Skill, Evidence Log, Critic, and Curator components.The Curator retires under-performers and enforces the bounded cap.
2. Background and Related Work
Prior systems build persistent skills, insights, manuals, and meta-skills, but generally lack measured per-task retirement. The paper distinguishes library drift from other agent failures and extends attribution-based diagnostics as an early warning.
- Self-evolving skill libraries: Voyager pioneered executable skill libraries, while later systems extract insights, compile manuals, or add meta-skills without outcome-driven retirement.AutoSkill adds versioning but does not retire skills using measured per-task contribution.
- Agent failure modes: Library drift is distinct from tool-use, planning, and self-correction failures because it arises from persistent cross-task memory degrading future performance.Its degradation substrate is accumulated skill artifacts rather than updated neural weights.
- Diagnostic signals: Per-skill attribution verdicts extend LLM-as-judge evaluation into HELPED, HURT, and NEUTRAL signals for early drift detection.At least three failures sharing a canonical pattern are required before a skill is created.
3. Library Drift: Definition and Trigger
Library drift occurs when accumulated artifacts reduce expected performance below the no-skill baseline through systemic retrieval and injection failures. Two ablations reproduce its boundary conditions: no injection produces a flat floor, while harsh retirement causes active harm.
- Definition: Library drift is defined by E[pass@1 | S_t] < E[p_0] for some t > 0.The failure can arise even when individual skills appear reasonable because retrieval dilution and stale injection operate systemically.
- Mechanism: Unvalidated accumulation dilutes retrieval, surface-similarity routing crowds out useful skills, and stale injections silently mislead the solver.These are the three compounding stages of drift.
- Sub-modes: Stagnation, bloat, and erosion describe drift under absent routing, unbounded growth, and over-aggressive governance, respectively.Erosion leaves only two active skills in A4.
- Reproducible triggers: +0.002±0.005 gain in A1 shows that creating skills without injecting them produces the no-skill floor.The Router is forced to NONE while synthesis, criticism, and curation continue.
- Reproducible triggers: −0.019±0.010 in A4 falls below the no-skill baseline after reducing Nmin from 100 to 20 and tightening τ to 0.0.The bank collapses to two active skills, and the effect is consistent across all three seeds.
- Interpretation: A1 shows injection is necessary for skills to matter, whereas A4 shows that insufficient-evidence lifecycle management can be worse than no skills.The triggers bracket the failure mode from both sides.
4. Trace-Level Diagnostics
Final pass@1 and solve rate cannot disentangle task difficulty, model stochasticity, and library quality. The paper therefore uses contribution scores, attribution verdicts, and router engagement as actionable early-warning signals.
- Motivation: End-task metrics are insufficient because they conflate task difficulty, model stochasticity, and library quality.Three complementary trace-level signals decompose the failure into actionable components.
- Per-skill contribution score: Mean per-skill contribution ˆc declines when the bank accumulates skills that do not help.It rises in the Default as harmful skills are retired and oscillates in A4 after premature removal of useful skills.
- Attribution verdicts: Critic verdicts label each failure capsule HELPED, HURT, NEUTRAL, or INAPPLICABLE and support both synthesis and early-warning diagnosis.Clusters of at least three failures sharing a canonical pattern trigger new skills.
- Attribution verdicts: A rising HURT proportion signals drift before aggregate pass@1 declines substantially.In A4, the bank reaches two active skills by round 30 and router engagement drops to 18.9%.
- Router engagement: Healthy conditions maintain 70–80% router engagement, while drifting A4 falls to 19% and retrieval-only A2 reaches 98% with lower quality.The LLM gate’s ability to decline injection is itself a drift-prevention mechanism.
5. Verified Fix: The Ratchet Recipe
Ratchet prevents library drift through outcome-driven retirement, a bounded active-cap, and a meta-skill authoring prior. Together, these mechanisms constrain accumulation, retrieval degradation, and harmful skill creation.
- 5. Verified Fix: The Ratchet Recipe: Ratchet combines outcome-driven retirement, a bounded active-cap, and a meta-skill authoring prior to prevent library drift.The recipe is implemented in a single-agent loop that writes, retrieves, curates, and retires natural-language skills.
- 5.1. Outcome-driven retirement: Retirement removes a skill after sufficient evidence when its empirical contribution is at or below the negative threshold.The default uses Nmin = 100 trials and τ = 0.10, balancing protection against stochastic noise with removal of harmful skills.
- 5.2. Bounded active-cap: A hard active-cap of 50 evicts the lowest-contribution skill when synthesis would exceed capacity, limiting retrieval degradation from unbounded growth.Combined with retirement, the cap supports a non-divergence bound relative to the no-skill floor.
- 5.2. Bounded active-cap: The non-divergence guarantee bounds expected pass@1 degradation below the no-skill floor by τ + ϵ + Cδ under the stated cap and retirement conditions.Here ϵ is the Hoeffding estimation tolerance and δ is the per-skill failure probability.
- 5.3. Meta-skill authoring prior: The meta-skill enforces stylistic consistency, reducing harmful or redundant skills before downstream retirement must handle them.Removing it costs 43% of the Default’s gain, while explicit deduplication is slightly unnecessary at this scale.
- 5.4. Deduplication: Explicit canonicalisation and cover-guard are subsumed by the meta-skill, with both ablations slightly exceeding the Default.This result is reported as a design insight for systems with LLM-authored artifacts.
6. Experiments
Experiments on MBPP+ hard-100 evaluate Ratchet over 100 rounds using controlled ablations. Full governance improves held-out pass@1, while the ablations expose routing, retirement, authoring, cap, refresh, and cost trade-offs.
- 6.1. Protocol: The protocol samples 100 difficult MBPP+ tasks, splits them into 60 train and 40 eval tasks, and averages three seeds over 100 rounds.The solver is a single direct LLM call without execution feedback, self-refinement, or tool use.
- 6.2. Main result: 0.584 late-window mean held-out pass@1 rises from a 0.258 baseline under the Default, with a 0.658 peak.The Default is the full-governance condition.
- 6.3. Decomposing the fix: +0.002 gain under A1 establishes the no-skill floor, while A2’s retrieval-only routing reaches +0.077 and confirms value beyond similarity retrieval.A1 always routes NONE; A2 bypasses the LLM gate and injects the top-ranked retrieval hit.
- 6.3. Decomposing the fix: +0.187 gain under A3 is 57% of the Default gain, making the meta-skill the most valuable tested component.A3 removes the authoring prior from the Synthesizer prompt.
- 6.3. Decomposing the fix: −0.019 gain under A4 falls below the no-skill floor when the evidence floor is reduced to 20 trials and τ is tightened to 0.0.The resulting Hoeffding deviation is ϵ ≈0.44, and the bank collapses to 2 active skills.
- 6.3. Decomposing the fix: A5 and A6 exceed the Default with +0.374 and +0.363 gains, indicating explicit deduplication is unnecessary at this scale.Both results are within ±2σ at n=3, and the meta-skill’s guidance may make explicit filters discard useful skills.
- 6.3. Decomposing the fix: A7 produces comparable +0.317 mean performance but higher variance, while A8 reaches +0.372 and a 0.725 peak at 55% more wall time.The cap primarily controls variance at this scale, whereas more frequent refresh adds compute without meaningfully improving the learning curve.
- 6.4. Cost: The Default uses ∼14.5k LLM calls and 6.5 h per 100 rounds, compared with 10k calls and 2.3 h for A1.The solver accounts for 63–99% of total calls across conditions.
7. Discussion
The paper generalizes library drift beyond skill libraries and shows that trace-level diagnostics make failures visible early enough for safe intervention. Results also reveal that governance settings are scale-dependent and can trade mean performance, variance, and compute cost.
- Library drift applies to rule-based systems, workflow memories, and episodic stores that persist LLM-authored artifacts without retirement.
- Trace-level diagnostics detect drift before end-task metrics decline and support counterfactual intervention through contribution scores, attribution verdicts, and router engagement.
- A5/A6 slightly exceed the Default, showing that the meta-skill subsumes explicit deduplication at this scale.
- A7 achieves comparable mean gain but higher variance, while A8 matches relaxed-dedup gains at 55% more wall time.A7 doubles the cap; A8 requires 10.1 h versus 6.5 h.
- Governance knobs are scale-dependent: deduplication may become load-bearing on larger suites, while relaxing the cap exposes synthesis-order variance.
- The evaluation is limited to MBPP+ hard-100, one model, and empirically chosen diagnostic thresholds; broader validation remains future work.
8. Conclusion
The paper identifies library drift as a silent systemic failure caused by unmanaged accumulation and presents governance, diagnostics, and ablations as a transferable response.
- +0.328 is the gain attributed to adding retirement, a bounded cap, and an authoring prior to the frozen LLM.
- A1 and A4 bracket the failure with a no-injection floor and governance-induced harm, while diagnostics detect drift before end-task metrics decline.
- Eight ablations identify retirement and the meta-skill as load-bearing, while explicit deduplication can be safely removed in this setting.
- The playbook transfers to systems that persist LLM-authored artifacts across episodes.
A. System Architecture
Ratchet uses a five-phase round loop that evaluates, trains, attributes failures, synthesizes skills, and curates the active library. A router retrieves candidate skills and gates whether to inject one or NONE.
- Ratchet cycles through Eval, Train, Critic, Synthesizer, and Curator phases on every round.The Critic assigns per-failure verdicts, the Synthesizer clusters recurring patterns, and the Curator retires under-performers while enforcing the cap.
- The Router selects one skill or NONE using two-stage tf-idf and embedding retrieval followed by an LLM gate.
- The Default configuration is documented as the system’s reference hyperparameter setting.
D. Non-Divergence Proof Sketch
The non-divergence argument combines a capped active set with retirement based on sufficient evidence and low contribution. Its bound is loose empirically but excludes unbounded degradation under the stated assumptions.
- The proof assumes an active-set cap and retirement when a skill has at least Nmin trials and contribution estimate at most −τ.
- With probability ≥1 −Cδ, every surviving skill has contribution at least −τ −ϵ, yielding an expected pass floor of E[p0] −(τ + ϵ) −Cδ.
- With τ = 0.10, Nmin = 100, C = 50, and δ = 10^-3, the bound gives ϵ ≈0.20 and floor = E[p0] −0.35.
- The bound is loose at this scale: the system gains +0.328 rather than losing 0.35, while unbounded degradation is ruled out.