PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Libraries

Linfeng Fan, Yuan Tian, Ziwei Li, Zhiwu Lu

arXiv:2605.10990v1cs.SEcs.AI

TL;DR

Long-lived skills can become stale as environments change, while existing monitors track changed values rather than their operational role. SKILLGUARD extracts and checks role-bearing contracts, reducing false alarms while supporting drift discovery and localized repair.

  • Problem

    As reusable skill libraries grow, external services, packages, APIs, and configurations evolve, while existing monitors track changed values rather than their role within a skill.

    Source: §1 · p. 1

  • Method

    SKILLGUARD extracts executable environment contracts from skill documents, validates role-bearing assumptions against known or live conditions, and uses violations to localize repair.

    Source: §1 · p. 2

  • Results

    40% false positives for contract-free CI probes contrast with zero false alarms over 599 controls; strongest-backbone known-drift verification reaches 100% precision and 76% recall, and live discovery reaches 86% conservative precision.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Violated contracts provide localized repair evidence, improving one-round success from 10% without localization to 78%.

    Source: §1 · p. 2

    The contract view is strongest for localizable assumptions and weaker when repair changes interaction mechanisms; in three authentication cases, repair succeeds in 0%.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

LLM agents increasingly rely on reusable skill libraries, but these skills silently decay as the external services, packages, APIs, and configurations they reference evolve. Existing monitors detect such changes at the wrong granularity: they observe values, not the role those values play in a skill. A version string in a comment is noise; the same string in a pinned dependency is an operational obligation. We formulate skill drift as contract violation and introduce \sgname{}, which extracts executable environment contracts from skill documents and validates only those role-bearing assumptions against known or live conditions. This distinction turns noisy monitoring into a precision-first maintenance signal. Contract-free CI probes produce 40\% false positives, while \sgname{} raises zero false alarms over 599 no-drift and hard-negative cases (Wilson 95\% CI $[0,0.6]\%$). In known-drift verification, \sgname{} achieves 100\% precision and 76\% recall with the strongest backbone; in a pre-registered study over 49 real skills, it discovers live drift with 86\% conservative precision. Violated contracts also make repair actionable, improving one-round success from 10\% without localization to 78\%. We release \dbname{}, an 880-pair benchmark for skill degradation.

1 Introduction

Long-lived skills can become stale as environments change, but existing monitors track changed values rather than their operational role. SKILLGUARD extracts and checks role-bearing contracts, reducing false alarms while supporting drift discovery and localized repair.

1 Introduction

  • As agents shift from one-off tasks to reusable skill libraries, evolving APIs, packages, URLs, configurations, and authentication can make previously correct skills fail.Existing monitors observe environmental values, not whether they play an operational role in a particular skill.Source: §1 · p. 1
  • The paper frames skill drift as contract violation: only role-bearing assumptions that must remain valid for a skill to execute as intended are monitoring targets.Source: §1 · pp. 1–2
  • SKILLGUARD extracts and validates environment contracts against known or live conditions, and DRIFTBENCH provides an 880-pair benchmark for degradation.Violated contracts also localize repair; one-round success improves from 10% without localization to 78%.Source: §1 · p. 2
  • 40% false positives for contract-free CI probes contrast with zero false alarms over 599 no-drift and hard-negative cases.In known-drift verification, the strongest backbone achieves 100% precision and 76% recall; live drift discovery reaches 86% conservative precision.Source: §1 · p. 2, Figure 1 · p. 2
  • In known-drift verification, the strongest backbone achieves 100% precision and 76% recall, while a pre-registered study over 49 real skills finds 86% conservative precision for live drift discovery.Source: §1 · p. 2

2 Related Work

Prior work studies skill acquisition, security auditing, reactive repair, contract mining, structured drift monitoring, and agent benchmarks. SKILLGUARD focuses on post-deployment maintenance of skills whose environment-facing assumptions have become stale.

2 Related Work

  • Skill-library research studies acquisition, usefulness, and governance, while security auditing examines whether skills are safe to load; SKILLGUARD instead studies post-deployment staleness.Examples include Voyager [27], SkillFlow [26], SkillsBench [16], systematization [13], and security auditing [19].Source: §2 · pp. 2–3
  • Reactive repair methods such as automated program repair [4] [5] [24], Self-refine [20], and Reflexion [25] wait for execution failures, whereas SKILLGUARD detects contract violations proactively.The repair result is evidence for localization, not dominance over stronger multi-round repair methods.Source: §2 · p. 3
  • Unlike contract mining from programs, traces, or documentation [3] [6] [8] [9] [15], SKILLGUARD must distinguish operational dependencies from incidental mentions in mixed-format skills.It validates role-bearing obligations such as pinned dependencies, endpoints, schema fields, authentication mechanisms, and configuration values.Source: §2 · p. 3
  • Structured dependency, API, and configuration monitors often assume manifests, telemetry, or executable environments, while skill assumptions may be embedded in prose and snippets.SKILLGUARD validates role-bearing contracts rather than every observed value.Source: §2 · p. 3
  • Agent benchmarks evaluate task-solving, software engineering, or skill quality, whereas DRIFTBENCH tests degradation of previously valid skills and both drift recall and false-alarm resistance.It pairs skills with generated drifts, LLM-free real-world drifts, and hard negatives.Source: §2 · p. 3

3 DRIFTBENCH: A Benchmark for Skill Degradation

DRIFTBENCH is designed to test both missed operational drift and false alarms on changes that preserve a skill’s contract. It combines controlled and real-world drift pairs with identity and hard-negative controls, then audits a sample by hand.

3 DRIFTBENCH: A Benchmark for Skill Degradation

  • DRIFTBENCH combines controlled drifts, LLM-free real-world drifts, identity pairs, and hard negatives to evaluate detection and false-alarm resistance.Hard negatives change text or values without violating operational contracts.Source: §3 · p. 3, Table 1 · p. 4
  • The benchmark defines eight drift types, from URL changes and version bumps to API migrations, schema changes, authentication changes, and dependency updates.The types include both direct surface changes and less syntactically obvious changes.Source: §3 · pp. 3–4
  • The controlled split contains 174 drift pairs derived from 49 skills across seven programming domains, with 99.6% adjudicated validity.Qwen3.6-Plus proposed 422 candidate pairs; DeepSeek-R1, registry-backed evidence, and author review supported validation.Source: §3 · p. 4
  • The LLM-free real-world split contains 107 naturally occurring drifts collected from live registries and changelogs, including PyPI, npm, GitHub Actions, and Docker Hub.These examples provide an external-validity split for observed software-ecosystem changes.Source: §3 · p. 4
  • The 599 negative controls comprise 49 identity pairs, 300 formatting hard negatives, and 250 semantic hard negatives that preserve operational contracts.Semantic negatives alter actual values, testing whether methods monitor obligations rather than raw values.Source: §3 · p. 4
  • A stratified 20% human audit covered 176 of 880 pairs; 14 required corrections, but no audited pair changed its drift/no-drift label.Corrections concerned value normalization, evidence spans, hard-negative wording, or source metadata.Source: §3 · p. 4

4 SKILLGUARD: Contract-Based Monitoring

SKILLGUARD treats skill drift as a contract violation: it filters environmental mentions for operational obligations before validation, then uses failed contracts to localize repair.

4 SKILLGUARD: Contract-Based Monitoring

  • SKILLGUARD validates only role-bearing operational assumptions, suppressing incidental mentions before probing known or live conditions.The system represents contracts with their type, role, value, and evidence span, and reports violations only when drift matches an OPERATIONAL contract.Source: §4 · pp. 4–5

4.1 Extracting Role-Bearing Contracts

  • Candidate extraction finds explicit environmental mentions, while a semantic pass labels obligations and captures implicit assumptions such as response formats, tool availability, and schema fields.The deterministic pass covers 15 pattern families; expanding regex families from 7 to 15 raises regex-only recall on real-world drifts from 45% to 78%, preserving 0% FPR.Source: §4.1 · p. 5

4.2 Validating Contracts

  • After role filtering, a type-gated matcher compares operational contracts with known drift events; live validation uses public evidence when available but may miss cases without reliable validators.Live sources include package registries, URL responses, and ecosystem conventions; the mode is precision-oriented rather than exhaustive.Source: §4.2 · pp. 5–6

4.3 Localizing and Verifying Repair

  • Failed contracts localize repair inputs, and the literal verifier checks that required new values appear while corresponding old values disappear.The verifier is not proof of semantic correctness and can reject valid repairs that do not use the exact literal value.Source: §4.3 · p. 6

4.4 Complexity and Scope

  • SKILLGUARD targets environment-contract violations within extraction coverage and available validators, rather than fully verifying skill correctness.Known-drift validation costs O(|C||D|) lightweight string/type comparisons, and repair runs only for reported violations.Source: §4.4 · p. 6

5 Experiments

The experiments test whether contract-based monitoring suppresses false alarms while retaining drift detection, transfers to live discovery, and localizes repair. Results show precise but incomplete detection and higher one-round repair success with failed-contract localization.

5 Experiments

  • The evaluation asks whether role-bearing contracts suppress false alarms without collapsing recall, transfer to live conditions without drift specifications, and provide useful repair evidence.It uses five backbone LLMs and compares detection and repair against multiple baselines, reporting precision, recall, FPR, and strict repair success.Source: §5 · pp. 6–7

5.1 Q1. Do contracts change the monitoring error profile?

  • Across five backbones, SKILLGUARD reaches 100% precision; its strongest backbone has 76% recall, with zero false alarms across 599 controls, versus 40% FPR for contract-free probes.Figure 3 also compares alternatives: Grep/diff reaches 30% recall, Dependabot-style scanning 11% recall and 10% FPR, and NL2Contract-style extraction 45% recall and 0% FPR.Source: Figure 3 · p. 7, §5.1 · p. 7, Table 2 · p. 7

5.2 Q2. Does contract monitoring transfer to live discovery?

  • In a pre-registered scan of 49 real skills, SKILLGUARD flags 14, achieving 86% conservative precision, 55% recall, and 7% FPR.The primary result retains two apparent false positives, later adjudicated as genuine drift; the authors characterize live discovery as high-precision surfacing, not exhaustive execution-based checking.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Q3. Do failed contracts make repair actionable?

  • Failed-contract localization raises one-round repair success from 10% without localization to 78%, but does not significantly outperform stronger repair baselines in this sample.Plain drift text reaches 60% and three-round Self-refine reaches 80%; the authors frame the result as actionable localization rather than repair-method dominance.Source: Figure 4 · p. 8, §5.3 · p. 8

6 Analysis

The analysis attributes precision to filtering for operational obligations, while recall depends on extracting those obligations and having validators that can check them. It also identifies measurement and mechanism boundaries for repair and detection.

6.1 What explains the precision–recall profile?

  • Precision mainly follows the role gate, while recall depends on extracting relevant obligations and whether validators can check them; the role-label audit achieves 89.5% precision, 83.4% recall, and 86.3% F1.On semantic hard negatives, role precision is 100.0% and role recall is 72.7%.Source: §6.1 · pp. 8–9
  • Expanding the deterministic extractor from 7 to 15 pattern families raises regex-only recall from 13% to 33% on the controlled split and 45% to 78% on real-world drift, with FPR remaining 0%.The added patterns cover forms such as Docker images, GitHub Actions versions, environment variables, cloud regions, CLI flags, and configuration filenames.Source: §6.1 · p. 9

6.2 What does repair evidence actually show?

  • Failed contracts identify the invalidated operational assumption, its location, and the triggering evidence, turning detection into a constrained edit request rather than an open-ended rewrite.Repair success rises from 10% without localization to 78% with failed contracts in one round, but differences from plain drift text and three-round Self-refine are not statistically significant.Source: §6.2 · p. 9

6.3 Where does the current contract view break?

  • The strict verifier checks that required new values appear and corresponding stale values disappear, but it is not a semantic proof; it accepts 165/174 cases versus 168/174 for a type-aware verifier.The roughly 2% conservative rejection gap appears mostly in dependency updates where a valid edit need not preserve the exact literal form.Source: §6.3 · p. 9
  • SKILLGUARD is strongest on localizable assumptions; authentication drift is hardest in this slice, with 67% detection and 0% repair across three examples.The authors identify schema-aware validators, execution-backed checks, or richer repair plans as complements when interaction mechanisms must change.Source: §6.3 · p. 9

7 Conclusion

SKILLGUARD frames skill degradation as invalidated operational assumptions, detects and localizes those violations, and improves one-round repair from 10% without localization to 78%. Its scope remains limited to extracted contracts and available validators, not exhaustive verification.

7 Conclusion

  • 40% false positives from contract-free CI probes contrast with zero false alarms across 599 negative cases; live discovery reaches 86% conservative precision, and one-round repair rises from 10% without localization to 78%.The paper frames these results as making skill degradation measurable while leaving exhaustive semantic verification to complementary execution-based methods.Source: §7 · pp. 9–10

A Appendix Overview

  • The appendix organizes supporting evidence on contract representation, limitations, algorithmic ordering, benchmark construction, detection claims, live discovery, repair, and reproducibility.Source: A Appendix Overview · p. 12

B Notation and Scope

  • SKILLGUARD monitors role-bearing assumptions as contracts while suppressing incidental mentions such as comments, tutorial citations, badges, and examples.Source: Table 3 · p. 12, B Notation and Scope · p. 12
  • SKILLGUARD detects environment-dependent assumption violations, but does not provide complete semantic verification, exhaustive live drift discovery, or proof of correctness across executions.Source: B Notation and Scope · p. 12

C Limitations

SKILLGUARD’s guarantees depend on extracted contracts and available validators, and conservative surface matching trades some coverage for precision. DRIFTBENCH combines controlled and naturally occurring drift cases with adjudication.

C Limitations

  • SKILLGUARD can miss implicit, behavioral, or execution-only assumptions, and its live validators do not replace execution-based canaries when exhaustive coverage is required.The role-label distinction is reliable but imperfect; repair localization also does not solve mechanism-level rewrites or prove semantic correctness.Source: C Limitations · pp. 12–13

D Algorithmic Details

  • Role filtering precedes validation: SKILLGUARD checks operational mentions against known drift or live evidence and returns violations with optional localized repair.Source: D Algorithmic Details · p. 13

E Matcher Normalization and Backoff Rules

  • Conservative surface matching favors precision over recovering schema renames, authentication-flow migrations, or API behavior changes, which may need schema-aware or execution-backed extensions.Source: E Matcher Normalization and Backoff… · p. 13

F DRIFTBENCH Construction and Validation

  • DRIFTBENCH includes 174 controlled and 107 LLM-free real-world drift pairs; registry-backed adjudication gives the controlled split 99.6% validity.The benchmark also combines controlled drifts, naturally occurring drifts, identity pairs, and hard negatives.Source: F DRIFTBENCH Construction and Validation · p. 14, Table 5 · p. 14

G Human Audit of DRIFTBENCH

A stratified audit of DRIFTBENCH checks benchmark labels and pair instantiations, while a separate role-label audit examines the OPERATIONAL/INCIDENTAL decision and its coverage limits.

G Human Audit of DRIFTBENCH

  • The human audit sampled 176 of 880 pairs across drift and negative-control splits; corrections did not change any audited pair’s drift/no-drift label.The audit targets label errors, old/new value instantiation, and hard negatives that might violate an operational contract.Source: G Human Audit of DRIFTBENCH · pp. 14–15, Table 6 · p. 15

H Standalone Role-Label Audit

  • Two annotators labeled 200 environmental mentions as OPERATIONAL or INCIDENTAL, with disagreements adjudicated before comparison to SKILLGUARD’s role predictions.Source: H Standalone Role-Label Audit · p. 15
  • The role audit supports precision-first filtering, while lower recall on semantic hard negatives indicates missed operational assumptions and limits claims of exhaustive detection.Source: H Standalone Role-Label Audit · pp. 15–16

I Additional Detection Results

Across evaluated models, precision is stable while recall varies with extraction and model behavior. Negative controls test harmless changes, and expanded mention coverage improves recall while preserving zero-FPR behavior.

I Additional Detection Results

  • Precision remains stable across evaluated backbones, while recall varies with extraction and model behavior.The paper characterizes this as a precision–recall profile rather than a ranking of backbone capability.Source: I Additional Detection Results · p. 17, Table 9 · p. 17

J Negative Controls and Statistical Bounds

  • Formatting hard negatives alter document presentation, while semantic hard negatives change values without breaking operational contracts, testing whether detectors overreact to non-violations.The combined 599 negative controls support the false-alarm claim.Source: J Negative Controls and Statistical… · p. 17, Table 10 · p. 17

K Detection Baselines and Extraction Sensitivity

L Open-World Discovery Details

The live-monitoring simulation detects drift in most drifted skills with no false positives among fresh skills, while repair succeeds for only a subset of detected cases. The reported open-world discovery values remain conservative because they precede adjudication of two apparent false positives.

L Open-World Discovery Details

  • The open-world discovery table reports conservative pre-adjudication values, although both apparent false positives were later judged genuine drift.The table reports discovery at the skill level.Source: Table 13 · p. 18, L Open-World Discovery Details · p. 18
  • Fourteen skills were flagged, spanning CI, build optimization, packaging, evaluation, and other skill areas.The flagged set includes analyze-ci, bazel-build-optimization, changelog-automation, dbt-transformation-patterns, distributed-tracing, gitops-workflow, llm-evaluation, nx-workspace-patterns, python-packaging, rag-implementation, security-review, similarity-search-patterns, spark-optimization, and vector-index-tuning.Source: L Open-World Discovery Details · p. 18
  • The two apparent false positives were later judged genuine drift: a dbt documentation redirect returned 301, and a broken MLflow URL returned 404.The main paper retains pre-adjudication precision to avoid overstating live-monitoring performance.Source: L Open-World Discovery Details · p. 18
  • 8 of 11 drifted skills were detected, with 0 false positives among 38 fresh skills; repair succeeded on 3 detected skills.This yielded 38% conditional repair and 27% end-to-end success.Source: L Open-World Discovery Details · p. 18

M Repair Ablations, Cost, and Verifier Behavior

Repair evidence supports localization as an improvement over vague instructions, but not a significant advantage over other repair approaches in the current sample. The section also describes failure modes, implementation and reproducibility details, safeguards, and artifact licensing.

M Repair Ablations, Cost, and Verifier Behavior

N Per-Type Failure Taxonomy

  • Authentication drift may require replacing an interaction mechanism; schema changes in prose or examples and coupled fields also challenge extraction and repair localization.Per-type results have small counts and are diagnostic rather than definitive, pointing to needs for richer validators and repair plans.Source: N Per-Type Failure Taxonomy · p. 20, Table 17 · p. 20

O Implementation and Reproducibility Details

  • Extraction combines regex detection of environmental mentions with semantic role classification into contract records, then merges and deduplicates the results.Known-drift checks use substring, token-overlap, and type-compatible matching; live checks use public evidence, while contracts lacking reliable validators may be missed.Source: O Implementation and Reproducibility… · p. 20
  • The reproducibility package includes benchmark pairs, contract records, results, prompts, and scripts, but model-dependent reproduction requires access to hosted model endpoints.The protocol reports confidence intervals and explicitly retains nonsignificant repair comparisons rather than treating them as positive findings.Source: O Implementation and Reproducibility… · pp. 20–21

P Broader Impacts and Safeguards

  • The authors frame SKILLGUARD as a maintenance monitor, exclude secrets and private user data from the benchmark, and recommend reviewing repairs before production use.They identify authentication, schema, and behaviorally coupled changes as cases meriting particular review.Source: P Broader Impacts and Safeguards · p. 21

Q Assets and Licensing

  • Benchmark data and documentation use CC BY 4.0 and evaluation code uses MIT, while third-party assets remain under their original licenses or provider terms.Hosted model weights and provider artifacts are not redistributed.Source: Q Assets and Licensing · p. 21, Table 18 · p. 22

Related papers

Loading 2605.10990v1…