PaperlayerView exact revision on arXiv

Source-linked AI summary

When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems

Su Wang, Pin Qian, Yihang Chen, Junxian You, Xiaoyuan Wang, Xiaochong Jiang, Lifei Liu, Haoran Yu, Jingzhou Xu

arXiv:2606.00448v1cs.SEcs.AIcs.CR

TL;DR

Agent skill ecosystems can make individually safe skills jointly unsafe, but evidence about how often this occurs and when models realize it has been limited. SkillReact measures the gap with deterministic composition screening, human calibration, and action-based trials, finding an 18.2% calibrated pair-pattern validity rate and model-dependent realization. The findings support install-time compositional checks and capability isolation alongside per-skill scanning.

  • Problem

    Per-skill scanning misses compositional risks because individually safe skills can jointly satisfy forbidden capability conjunctions in a unified runtime.

  • Method

    SkillReact combines a deterministic static-composition benchmark, a two-rater LLM-assisted human-adjudication pipeline, and an action-based exploitability harness.

  • Results

    18.2% of flagged pair-pattern memberships were valid after human calibration, while realization varied by host model from full chains to refusal.

  • Takeaways & Limitations

    Static composition provides a durable composition-specific signal and motivates install-time compositional checks and capability isolation as complements to per-skill scanning.

  • Takeaways & Limitations

    Static candidate rates depend on the declared taxonomy and regex labels, while action-based realization is model- and runtime-conditional rather than registry-wide exploit prevalence.

Abstract

from arXiv · show

LLM agents increasingly rely on community-contributed skills that expand an agent's operational capability set. We study a core safety problem in agentic AI systems: whether individually safe skills can compose into unsafe installed skill sets. We present SkillReact, a compositional security measurement framework with three components: a deterministic static-composition benchmark, a two-rater LLM-assisted human-adjudication pipeline, and an action-based exploitability harness. On 1,520 ClawHub skills, 651 pass individual inspection and form 211,575 pairs; the benchmark flags 22.25% of these as structural candidates. We treat this raw rate as a recall-oriented scanner ceiling and calibrate it against human judgment: in a pattern-stratified audit, roughly one in five flagged pair-pattern hits survives as a real compositional risk (population-weighted validity 18.2%, our headline result), implying about 14K genuine risk memberships in a single registry that per-skill scanning misses by construction, since every pair is individually safe. An action-based harness then probes when these candidates become model-issued tool calls, and finds realization gated by host-model disposition: on an anchor-conditioned dropper subset, Haiku-4-5 issues the dropper-stage tool call on all 39 direct-prompt trials (36 of them the full download-then-execute chain, 3 download-only), Opus-4-7 stops at the download, and Sonnet-4-6 refuses outright. A control that holds the request fixed and varies only the installed skills finds compliance highest with no skills installed: a composition fixes which capabilities are reachable, while the host model decides whether to use them. Together these motivate install-time compositional checks and capability isolation as complements to per-skill scanning.

1. Introduction

SkillReact reframes security from individual skills to installed skill sets, measuring how often individually safe skills form structural compositional-risk candidates and when models realize them as tool calls.

  • Problem: Per-skill scanning misses violations that emerge when individually safe capabilities unite in one runtime.A file-read skill and a network-only skill can jointly satisfy a forbidden file-read ∧ network-outbound conjunction.
  • Framework: SkillReact combines deterministic static composition, calibrated human adjudication, and action-based exploitability measurement.The framework distinguishes candidate detection, human-calibrated validity, and model-issued tool-call attempts.
  • Realization: Realization depends on host-model disposition: models differed from issuing full chains to refusing, while the composition control found highest compliance with no skills installed.The action-based results are model- and runtime-conditional rather than registry-wide exploit prevalence.
  • Measurement: 18.2% of flagged pair-pattern hits survived human calibration, implying roughly one in five represented real compositional risk.The calibrated rate is the headline measurement, unlike the raw scanner rate.

2. Background and Related Work

Agent skills expand capabilities inside a unified runtime, creating a combinatorial security surface that per-skill defenses do not fully address. SkillReact adopts compositional security concepts while treating host-model behavior as an additional realization variable.

  • Agent Skill Ecosystems: Installed skills operate in a unified execution context, so the agent inherits their combined capabilities.This increases the security surface as the installed set grows.
  • Security Model: An individually safe skill avoids the paper’s 10 forbidden conjunctions but is not necessarily benign under every policy.“Individually flagged” is a scanner outcome, not a malicious-intent judgment.
  • Evidence Levels: The paper separates individually flagged skills, structural pair candidates, and model-issued tool-call attempts.The final level depends on host model, prompt, and runtime context.
  • Related Work: Prior work largely measures individual-skill risk, whereas SkillReact changes the unit of analysis to an installed skill set.Its compositional-security lineage includes capability hypergraphs, non-interference, and least-privilege perspectives.
  • Related Work: Unlike deterministic Android composition analyses, agent-skill realization varies with prompt, anchor identity, and host-model disposition.The paper reports model-dependent full-chain, download-only, and refusal outcomes.

3. The SkillReact Measurement Framework

SkillReact is a three-part measurement framework that links deterministic capability-union screening to human calibration and conditional action-based realization evidence.

  • Framework Overview: SkillReact comprises a deterministic static-composition benchmark, an LLM-assisted human-adjudication pipeline, and an action-based exploitability harness.Together these components produce structural candidates, calibrated validity, and tool-call logs.
  • Static-Composition Benchmark: The static benchmark freezes the corpus, capability schema, regex catalog, and forbidden patterns for reproducible candidate detection.Its raw CCR is reported as a recall-oriented scanner ceiling and calibrated by the adjudication pipeline.
  • Exploitability Harness: The action-based harness records whether a host model invokes tools toward a forbidden goal under a stated runtime policy.It includes Capability-Sketch Trials and Skill-Artifact Trials, with results treated as conditional realization evidence.
  • Human Adjudication: The adjudication pipeline audits a stratified 200 pair-pattern gold set using two LLM raters and a human reviewer.Human-audited per-pattern validity rates provide the calibrated CCR.
  • Outputs: Primary outputs include raw CCR, calibrated pair-pattern validity, evidence traces and diagnostics, and per-pair tool-call logs.These outputs span static screening, calibration, and action-based evaluation.

4. Methodology

The methodology combines a frozen ClawHub snapshot, deterministic capability extraction and pairwise union analysis, a stratified human-audited calibration, and an explicit taxonomy of forbidden compositions.

  • Skill Collection: The study analyzes a frozen snapshot of 1,520 retrieved skills from 5,128 registry entries exposed by the crawler.The authors retain the snapshot as the unit of analysis and treat capability-profile independence of attrition as an assumption.
  • Capability Extraction: The regex pipeline is deterministic and recall-oriented, but its capability labels are scanner outputs rather than ground truth.This caveat is inherited by the headline metric and motivates human calibration.
  • Human Adjudication: A 200-unit, pattern-stratified gold set is independently marked by two LLM raters and human-adjudicated regardless of agreement stratum.LLM agreement is used diagnostically, while human verdicts provide the validity labels.
  • Calibration: 18.2% is the population-weighted validity rate for flagged pair-pattern memberships, estimating roughly 14,356 valid memberships.The 4.05% pair-level estimate requires co-validation, while 6.78% is a projected membership-based cap.
  • Forbidden Patterns: The taxonomy defines 10 capability-conjunction patterns, including data exfiltration, credential theft, lethal trifecta, privilege escalation, and reverse shell.Patterns can overlap because longer conjunctions are subsumed by shorter ones.
  • Pairwise Analysis: A structural candidate is an individually safe pair whose capability union satisfies at least one forbidden pattern.The method evaluates 211,575 pairs from 651 individually safe skills.
  • Scope: CCR is conditional on the 10-pattern taxonomy, regex labels, and unified-runtime threat model, so it is not realized exploit prevalence.The checker is complete only for taxonomy-expressible capability-union violations under correct labels.

5. Results

Across individually safe skills, deterministic composition produces many structural candidates, but calibration shows the raw scanner rate substantially overstates validated risk. Data exfiltration is the dominant pattern, while capability-level diagnostics identify substantial extraction noise and network outbound as the main load-bearing detector.

  • 5.1. Individual Skill Scan: 57.2% of the 1,520 analyzed skills are individually flagged, while 651 (42.8%) pass individual inspection.The Individual Flag Rate is a scanner diagnostic rather than a vulnerability-prevalence claim.
  • 5.2. Compositional Candidate Rate: 22.25% of 211,575 individually-safe pairs satisfy at least one forbidden composition pattern under capability union.The fixed-pool benchmark treats this Compositional Candidate Rate as a recall-oriented scanner ceiling, not an exploitability estimate.
  • 5.3. Pattern Distribution: Data exfiltration appears in 33,708/47,075 (71.6%) of structural candidates, reflecting the common union of file-reading and network-outbound capabilities.Pattern counts overlap, and some categories are fully subsumed by shorter patterns rather than representing independent contributions.
  • 5.5. Capability-Level Diagnostics and Reliability: Macro κ = 0.169 indicates slight agreement between regex labels and a capability-level LLM relabel, so the raw 22.25% inherits label noise.The regex pipeline is recall-oriented and can fire on prose, documentation, snippets, and lexical negators.
  • 5.5. Capability-Level Diagnostics and Reliability: Removing the net_out detector collapses CCR by 92% (∆CCR = −0.206), while file_read accounts for another 44% drop.The capability ablation uses a recomputed pool, so small positive changes for other capabilities are denominator artifacts.
  • 5.6. Calibrated CCR via Stratified Sampling: 18.2% of flagged pair-pattern memberships survive human audit, corresponding to an estimated ≈14,356 valid risk memberships.This population-weighted validity rate is measured at the sampled pair-pattern level; pair-level projections require explicit assumptions.

6. Case Studies

The case studies show how individually safe skills combine into distinct compositional-risk patterns, including data exfiltration, credential theft, and a download–write–execute dropper chain.

  • Case studies cover three operationally distinct compositional-risk patterns: data exfiltration, credential theft, and dropper behavior.The examples illustrate structure rather than prevalence, and popularity metadata does not affect CCR.
  • Data exfiltration: Data exfiltration combines file read with network outbound, creating the dominant FR ∧NO pattern among candidate pairs.Each contributing skill is individually safe because neither alone has the forbidden conjunction.
  • Credential theft: Credential theft combines credential access with network outbound, exposing a credential-egress path despite each skill lacking both capabilities individually.
  • Dropper: The dropper pattern combines network outbound, file write, and process spawn into a download–write–execute chain.The shared anchor steipete-apple-notes supports the 13 dropper pairs probed by the action-based harness.

7. Action-Based Exploitability Harness

The action-based harness tests whether structural candidates become model-issued tool-call attempts under specified prompts, models, and runtimes. Results vary sharply by host-model disposition, while composition controls indicate that installed skills determine reachable capabilities but do not themselves raise compliance.

  • The harness distinguishes structural candidates from actual model-issued tool-call attempts under explicit runtime and prompt conditions.It uses action-based classification, counting a trial only when the host model invokes a tool toward the forbidden goal.
  • Capability-Sketch Trials: Across 81 capability-sketch trials, 17 (21.0%) issued forbidden tool-call attempts, with dropper prompts eliciting the most attempts at 13/27.Prompt-injection conditions yielded 0/27 attempts in the reported batch.
  • Skill-Artifact Trials: On 39 anchor-conditioned direct-prompt dropper trials, Haiku-4-5 issued attempts in 39/39 cases: 36 full chains and 3 download-only.The observed command was the canonical curl →bash chain, with the destination redacted.
  • Composition control: With the request fixed and installed skills varied, compliance was highest with no skills installed and was not elevated for the dropper pair.The control compares no skills, the dropper pair, an unrelated pair, and a single skill.
  • Caveats: The harness is selected, dropper-focused, anchor-conditioned, prompt-specific, model-specific, and runtime-specific, measuring issued intent rather than verified remote execution.The study does not extrapolate these results to registry-wide exploit prevalence.

8. Design Implications

The paper proposes install-time compositional checks that detect newly reachable forbidden patterns and expose pair-level evidence for review. These checks are exact and fast within a fixed taxonomy, but they complement rather than replace per-skill scanning, runtime sandboxing, or model and prompt defenses.

  • Compositional install-time checker: Mode 1 gates installation by testing whether adding a skill makes the installed capability union trigger a previously absent forbidden pattern.Critical patterns block installation, other new violations warn, and no new violations allow installation.
  • Compositional install-time checker: Mode 2 reports each triggering skill pair, forbidden pattern, contributing capabilities, and evidence traces for user or registry adjudication.
  • Scope and limits: The checker is complete only for capability-union violations in the declared 10-pattern taxonomy, conditional on correct regex labels.It is not complete for arbitrary attack compositions, novel patterns, or capability mislabels.
  • Implementation: Both checker modes complete in under 1 ms per install on the 1,520-skill snapshot, supporting synchronous gating and an evidence-panel workflow.Mode 1 has O(|F| · |C|) complexity per install, while Mode 2 has O(k · |F| · |C|).
  • Toward learned compositional monitors: A richer learned monitor could inspect skill prose, evidence traces, or tool-call plans, but would trade determinism for coverage and must remain cheap and local for install-time use.
  • Defense in depth: Capability isolation and runtime capability propagation are proposed as complements to detection, while host-model disposition remains a separate factor gating realization.The proposed layers combine static composition checks, model alignment, and runtime policy.

9. Limitations and Threats to Validity

The study’s limitations concern extraction noise, adjudication bias, runtime and pairwise scope, limited exploitability sampling, and restricted ecosystem coverage. These constraints limit interpretation of the structural and action-based measurements, especially beyond the tested models, settings, and pairwise installations.

  • Measurement validity: Regex extraction is recall-oriented and produces false positives from prose, snippets, and lexical negators, so the raw 22.25% rate is only an upper bound.Calibration with a 200-unit LLM-assisted human-adjudication protocol is used to mitigate this noise.
  • Measurement validity: The calibrated headline rate may reflect single-rater and anchoring bias because one auditor saw the LLM markings and human–human agreement was not measured.The authors identify a blind second audit with reported human κ as needed to bound this bias.
  • Runtime and execution: The unified, non-isolated runtime does not measure defenses such as skill isolation, capability scoping, network allowlists, or per-skill confirmation.Such production controls could substantially reduce realization rates.
  • Scope: The analysis covers pairs only, leaving open additional candidates or redundancies introduced by triples and larger installed sets.Large-k analysis is left for future work.
  • Exploitability generalization: Exploitability results are selected, dropper-focused, prompt- and runtime-specific, and conditioned on one anchor and Haiku-4-5 rather than representative of registry-wide prevalence.The capability-sketch trials cover only 3 of 10 patterns and 3 strategies, while the composition control attributes realization to model disposition rather than the specific pair.
  • Harness validity: The broad April runs lacked sandboxing of user-global Claude configuration, although a hardened clean-isolation rerun addressed this for the three-pair Claude protocol.
  • External validity: The 1,520-skill snapshot covers one ecosystem, and replication on other registries is needed to generalize the findings.Filtering prolific authors changes CCR from 22.25% in the full corpus to 17.95% at the ≥5-skills-per-author threshold.
  • Future validation: The authors propose capability-noise bootstrapping, a per-skill FR∧NO baseline, and a larger multi-anchor harness as extensions before broader claims are made.Until then, SKILLREACT reports structural candidates, capability diagnostics, and selected action-based validation rather than an exploitability prevalence estimate.

10. Ethics and Responsible Disclosure

The paper frames the finding as an ecosystem-level structural pattern rather than a vulnerability in an individual skill. Its evaluation uses controlled, aggregate, and contained procedures intended to motivate defenses without enabling misuse.

  • Scope: The study measures host-model behavior on a public third-party skill snapshot and does not attack any live system or user.
  • Responsible disclosure: The named case-study skills are individually safe under the policy, while risk arises only from co-installation with a partner in a permissive, non-isolated runtime.The paper names only public package identifiers and reports aggregate statistics.
  • Containment: The harness issues real curl →bash chains under --dangerously-skip-permissions, so it requires an external network-egress-restricted sandbox with no long-lived credentials.The in-process egress check is only a best-effort tripwire, and an unregistered destination is not treated as a safety control.
  • Human data and dual use: Adjudication labels skill pairs rather than human subjects and collects no personal data, while the paper redacts the weaponizable destination string.The stated purpose is to motivate compositional defenses rather than enable misuse.

11. Conclusion

SKILLREACT measures compositional risk through calibrated structural candidates and separately examines model-disposition-dependent realization. The conclusion treats static composition as the durable composition-specific signal while distinguishing it from model-issued execution behavior.

  • 18.2% is the human-calibrated pair-pattern validity rate, projecting to 4.05% at the pair level under a co-validation assumption.The raw 22.25% rate is reported only as the scanner’s recall ceiling.
  • Haiku-4-5 issues the full download-then-execute chain, Sonnet-4-6 refuses, and Opus-4-7 issues only the download.A composition control attributes this realization gradient to the explicit request rather than the skill pair.
  • Static composition is the durable composition-specific signal, motivating install-time compositional checks and capability isolation alongside per-skill scanning.

Appendix C. Reproducibility and Adjudication Details

The appendix documents the action-based harness, clean-isolation configuration, adjudication rubric, and deterministic benchmark construction. These details make the reported measurements auditable and reproducible within the stated setup.

  • Harness configuration: The harness invokes Claude with Bash and Read tools, permission bypass, a selected model, and no session persistence.
  • Harness configuration: The clean-isolation profile supplies fresh temporary HOME, Claude configuration, and XDG directories, removes them after each trial, and provides skills through the system prompt.
  • Adjudication: Each adjudication unit records verdict, capability validity, target-pattern validity, plausibility, regex misfire, confidence, evidence, and rationale.The rubric distinguishes VALID, FALSE_POSITIVE, and BENIGN_BY_CONTEXT outcomes.
  • Benchmark reproducibility: The gold set samples 20 units per pattern across 10 patterns using fixed random seed 20260422, producing 200 units spanning 198 unique pairs.Capability detection uses source or shell idioms and natural-language prose pattern classes.
Loading 2606.00448v1…