When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills

• The study analyzes 1.53 million English GitSkills skills, labeling 2,097 units and training a detector to classify 1.37 million candidate units.Source: §1 · p. 2
• About one in seventeen English skills contains a loosening directive, and 70.1% of these directives apply unconditionally.Source: §1 · p. 2
• At equal amounts of oversight language, skills that run shell code, install packages, use sudo, or pipe downloads into a shell have 1.3–1.8 times the odds of loosening, without more tightening.Source: §1 · p. 2
• 2.4% of families carrying loosening account for half of copies of loosening-carrying skills, and adopters rarely change oversight text.Source: §1 · p. 2
• Existing scanners catch at most 46% of loosening directives at precision below 0.4, versus 61% at precision 0.71 for the same LLM prompted with the codebook.Source: §1 · p. 2
• Prior work focused on maliciousness, vulnerabilities, or instruction topics, while this study treats oversight-loosening directives as distinct from malicious content.Source: §1 · pp. 1–2

2.1 Agent Skills

• Skills contain a SKILL.md file with Markdown instructions, and agents load the body when a skill is invoked; about half of GitSkills’ 3.8 million files are exact copies.Source: §2.1 · p. 3

2.2 Oversight Mechanisms in Agents

• Oversight checkpoints include approval prompts, permission modes [17], allow- and denylists, flags, hooks, and tests; skills can remove instruction-based checks or guide agents around harness safeguards.Source: §2.2 · p. 3

2.3 Analysis of Agent Skills and Agent Instruction Files

• Skills research asks whether skills are malicious or vulnerable [18] [20], while instruction-file studies classify topics rather than whether instructions loosen or tighten oversight [5].Source: §2.3 · pp. 3–4

3 Study Design

• Human labels provide unit-level prevalence estimates and detector ground truth, while detector labels support skill-level estimates and the risk-context and spread analyses.The detector also serves as a baseline for the scanner evaluation.Source: §3 · p. 4

3.1 From GitSkills to Analysis Population

• The analysis population comprises 1,527,281 distinct English skills from GitSkills, representing 3,301,422 exact-copy occurrences across public repositories.The selection excludes files that fail the skill-format, date, symlink, or English-language criteria.Source: §3.1 · p. 4

3.2 From Skills to Candidate Units

• Skills are split into sentences, bullets, table cells, and code lines, and a 36-pattern lexicon retrieves candidate units for classification rather than deciding their labels.Each unit is read with its section heading and neighboring units as context.Source: §3.2 · pp. 4–5

3.3 Developing the Codebook

• The codebook classifies units as directives, mentions, or neither, then labels directives by polarity and mechanism and loosening directives by scope.Decision rules resolve recurring ambiguous cases identified during coding.Source: §3.3 · p. 6

3.4 Labelled Unit Sets

• The study labels 2,097 distinct units across prevalence, detector-training, and census sets; the gold set evaluates the detector and scanners.On the gold set, three-way label agreement reaches Cohen’s κ=0.82, with polarity 0.92, mechanism 0.84, and scope 0.85.Source: §3.4 · pp. 6–7

3.5 Detector

• The detector averages RoBERTa and Jev probabilities, uses Jev for mechanism labels, and reaches loosening F1 0.65 on the 400 gold units.Its precision is 0.70 and recall 0.61; it does not separate mentions from non-directives or predict scope.Source: §3.5 · pp. 7–8

4 Results

• Permission flags (33.9%), confirmation (27.4%), and persistence (16.0%) account for over three quarters of loosening directives, and 70.1% apply unconditionally.Among the most-copied units, transparency is the most common mechanism (22.6%, against 4.5% in the prevalence sample).Source: §4 · pp. 8–9
• 5.8% of English skills contain a loosening directive, while 13.3% of distinct candidate units loosen oversight and 36.4% tighten it.The prevalence sample identifies about 49,000 distinct loosening directives; excluding security-tool repositories and test fixtures changes the loosening share only marginally, from 13.2% to 13.3%.Source: §4 · pp. 8–9

4.2 RQ2:What kinds of skills are more likely to contain loosening directives?

• At equal amounts of oversight language, shell code, package installs, and sudo are associated with 1.7–1.8 times the odds of loosening, without more tightening.The association holds across the reported refits; skills with package installs, sudo, or curl | sh have lower odds of tightening, while git writes are associated with more tightening.Source: §4.2 · pp. 10–11
• Popularity is not associated with loosening, while skills copied into multiple repositories are more likely to loosen and less likely to tighten.12,484 skills (0.82%) combine loosening with destructive commands, curl | sh, or sudo; 6,918 (0.45%) remain when counting only high-confidence predicted loosening.Source: §4.2 · p. 11

4.3 RQ3:How do loosening directives propagate as skills are copied and edited?

• Copying does not amplify loosening, and 93.4% of variants leave their ancestor’s oversight directives untouched.When variants do change oversight language, their share of loosening among added units is 20.3%, versus 20.2% among removed units (+0.1 points [−5.3, +4.7]).Source: §4.3 · pp. 12–13
• 2.4% of loosening-carrying skill families account for half of copies of loosening-carrying skills, and 1.1% of distinct loosening units account for half of loosening occurrences.The 100 families with the most loosening hold 23% of loosening-carrying skill occurrences but 1.4% of all skill occurrences; Fig. 3 plots family and wording concentration.Source: §4.3 · pp. 12–13, Figure 3 · p. 12

4.4 RQ4:How well do existing skill scanners detect loosening directives?

• With LLM analysis, the two scanners catch 15–46% of loosening directives at precision below 0.4, while a codebook-prompted LLM reaches F1 0.65.Between 23% and 29% of scanner flags fall on tightening directives; the codebook-prompted model is statistically indistinguishable from the study’s detector.Source: §4.4 · p. 14

5 Discussion and Implications

• Loosening directives commonly make workflows more convenient, but shared files can carry them to agents and contexts their authors did not anticipate.Examples include passing --yes to package managers or letting test loops run uninterrupted; the authors report no sign of intent to harm.Source: §5 · p. 14
• A flagged loosening directive should prompt review rather than be treated as a verdict.A skill is a convenience mechanism, not necessarily an attack; the concern is that a shared directive may travel beyond its intended context.Source: §5 · p. 14
• Over two thirds of loosening directives are unscoped, setting one autonomy level across every context where a skill runs.Authors can preserve convenience with a scope clause such as “pass --yes only in CI”; vendors can warn about unscoped loosening at install time, especially in shell-running skills.Source: §5 · p. 14
• 960 of 86,048 distinct loosening units (1.1%) account for half of occurrences, while copying does not amplify loosening and 92–97% of variants leave oversight text untouched.The authors therefore favor upstream fixes: reviewing 960 units by hand covers half of occurrences, and editing the 100 most affected families could reach almost a quarter of loosening-carrying skills agents load.Source: §5 · p. 14
• Existing scanners frame skills as a malware problem, while loosening directives contain no malware, motivating an oversight-aware check.The proposed extension uses polarity-aware rules or an oversight question for an existing model, evaluated on labelled data.Source: §5 · p. 15
• Prior studies categorize instruction topics or detect malicious and vulnerable skills, but do not ask how instructions shift control between agents and humans.This work examines instructions that add or remove oversight.Source: §5 · p. 15
• The study measures what skills say, not whether agents follow the directives; testing agent behavior and scope-clause effects remains future work.The authors also call for improving lexicon recall, extending analysis to other instruction files, and tracking loosening over time.Source: §5 · p. 15

6 Threats to Validity

• The directive boundary and polarity involve judgment; agreement was between a human and an LLM, not two human coders.The codebook was refined over three versions and disagreements were adjudicated; the LLM made double coding of 1,000 units feasible.Source: §6 · p. 15
• Counting persistence broadly includes instructions to continue past failure or until task completion, rather than only skipping a human check-in.Counting only the latter kind gives loosening prevalence of 12.7% [11.0, 14.4] rather than 13.3%.Source: §6 · p. 15
• The lexicon captures an estimated 32% [14%, 75%] of loosening directives, so prevalence estimates are conditional on candidate units and absolute counts are lower bounds.The estimate is 61% without one unit of sampling weight 28; the detector has loosening F1 of 0.65, and its recall below precision leads to undercounting.Source: §6 · p. 15
• GitSkills covers public GitHub repositories and English skills; LLM-mode scanning was tested on 400 gold units with one model, leaving wide confidence intervals.Skill ancestry in RQ3 is inferred from first-commit dates or copy counts, and the two ancestor rules disagree on small shifts.Source: §6 · p. 16

7 Conclusion

• About one in seventeen English skills—about 88,000—contains a directive that loosens human control, and over two thirds apply unconditionally.Permission flags, skipped confirmation, and instructions not to stop are the most common mechanisms described.Source: §7 · p. 16
• Loosening concentrates in skills that can execute code and spreads through a few widely copied upstream skills rather than adopters weakening their copies.Source: §7 · p. 16
• Existing scanners miss most loosening because they are not asked to look for it, rather than because their models cannot recognize it.The authors say their codebook, labels, and detector can help stakeholders weigh convenience against oversight.Source: §7 · p. 16