Scan Before You Install: Detecting and Mitigating Local and Composite Vulnerabilities in Skills

1. Introduction

• Existing scanners largely analyze skills individually, while interactions through shared files, environment state, or conversation context can connect capabilities that appear benign in isolation.The paper identifies a gap in open-source scanners that assess both individual weaknesses and exploitable paths introduced by adding a skill to an installed set.Source: §1 · pp. 2–3
• The six-module scanner combines local analysis of instructions, code, sources, and capabilities with dependency-graph analysis, validation, reporting, and mitigation.Composite Analysis models file handoffs, shared environment state, and named invocations; optional model enrichment proposes additional instruction-mediated dependencies.Source: §1 · pp. 3–4
• Three analysts confirmed 53 of 59 findings (89.8% precision) across 23 repositories; PoC attacks covered 41 vulnerabilities, and 10 medium- and high-severity cases were disclosed.The confirmed weaknesses included injected-content-controlled execution, persistent instructions, approval bypasses, sensitive-directory writes, and data or credential disclosure.Source: §1 · pp. 3–4
• On the 556-case SkillTrustBench leaderboard subset, Local Analysis achieved 95.90% recall and a 4.22% false-positive rate; on 1,000 MalSkillBench skills, it achieved 98.70% precision and 94.69% F1.The paper reports the highest recall and F1 on the SkillTrustBench subset and the highest precision and F1 on the MalSkillBench sample.Source: §1 · p. 4
• Composite Analysis achieved 98.7% precision and 99.6% recall on 1,325 SCR-Bench sets and identified two GitHub cases, including the first reported in-the-wild composite vulnerability.The two cases linked conversation-history retrieval with Slack messaging, and token disclosure with feed monitoring and OAuth diagnostics.Source: §1 · p. 4
• Mitigations blocked attacks in all three trials for 47 of 55 findings while preserving the tested task; six were partially mitigated, and two could not be evaluated experimentally.The evaluation replayed each PoC three times and checked whether the legitimate task still worked.Source: §1 · p. 4

2.1. Anatomy of a Skill

• A skill’s SKILL.md contains YAML front matter and natural-language instructions, and a skill may also include scripts, templates, and reference documents.Agents load skill components in stages, activating instruction bodies as needed and potentially passing data between skills through files, environment variables, or conversation context.Source: §2.1 · p. 6

2.2. Security Terminology

• A vulnerable skill has an unintended security weakness in its instructions or code, whereas a malicious skill deliberately embeds harmful behavior.The distinction turns on author intent: a vulnerable skill’s weakness is not malicious intent.Source: §2.2 · p. 6
• A composite vulnerability arises when interactions among skills create a harmful path that no constituent skill enables on its own.The example links a backup skill’s credential archive to a debugging skill’s upload through a shared cache, with prompt injection inducing the sequence.Source: §2.2 · pp. 6–7, Figure 3 · p. 7

• Earlier studies scanned public skills for risky patterns or examined malicious campaigns, while credential-leakage research found that many cases require joint analysis of instructions and code.These studies documented risks in published skills and how attackers take advantage of them.Source: §3 · p. 7
• Existing individual-skill methods combine static checks, instruction review, and sometimes sandbox execution, but some rely on static triage or predefined instruction patterns.The paper says Local Analysis can run optional instruction review independently of static findings and can assess trust or authorization boundaries contextually.Source: §3 · pp. 7–8
• Prior interaction-focused work includes SCR-Bench for evaluating risks, CompoSkill and ColluSkill for constructing attacks, and several approaches for identifying unsafe skill combinations.SCR-Bench provides sandboxed cases but no detector; the passage describes limits on public releases of CompoSkill and ColluSkill artifacts.Source: §3 · p. 8
• SkillReact and SkillProbe relate skills through capability or risk labels, while the paper’s Composite Analysis derives dependencies and traces shared resources to operations.It reports candidate findings as ordered skill paths with quoted source evidence.Source: §3 · p. 8

4.1. Baseline Assumptions

• The model assumes the agent runtime and model are uncompromised, with configured tool approvals and sandbox restrictions in effect.Skills and adversary actions remain within the operations permitted by the runtime.Source: §4.1 · pp. 8–9
• A skill’s tool declaration is not a separate permission boundary; runtime settings determine which operations can execute.Using a tool outside the declared set signals a declaration mismatch but does not itself initiate an unauthorized operation.Source: §4.1 · p. 9
• Skills may influence one another through shared files, environment state, or agent context, and individual approval does not authorize every data transfer between them.The model does not assume installed skills were jointly reviewed by users.Source: §4.1 · p. 9

4.2. Attacker Model

• The adversary controls content processed during a legitimate task but cannot directly modify installed skills, the runtime, or its configuration.Controlled content may induce skill invocations, influence data passed between skills, or cause text to be stored and later read as instructions.Source: §4.2 · p. 9

5. Method

• The method defines a six-module pipeline in which validated candidates become findings, reporting selects findings involving the input skill, and mitigation proposes substitutions, patches, and configuration changes.Sandbox tests assess whether proposed changes prevent the reported vulnerability while preserving required functionality.Source: §5 · pp. 9–10, Figure 4 · p. 10

5.1. Content Extraction

• Content Extraction represents each skill’s instructions, supporting files, declarations, and source references for both analysis modules.Declarations describe claimed permissions, capabilities, inputs, outputs, and invoked or provided names; an unknown declaration alone does not establish a permission mismatch, while an explicitly empty declaration claims no permissions.Source: §5.1 · p. 10

5.2. Local Analysis

• Local Analysis combines deterministic pattern, value-flow, installation-source, and capability-declaration checks with optional LLM instruction review to identify within-skill vulnerabilities and declaration mismatches.The standalone reviewer treats instructions as untrusted and has no execution, browsing, file, or database tools; accepted proposals require a predefined behavior category, explanation, and exact source evidence.Source: §5.2 · pp. 11–13
• Value-flow analysis propagates source integrity and confidentiality labels through code and checks sensitive sinks, while unresolved assessments and incomplete analyses preserve collected candidates for validation.Unknown calls retain dependence on their inputs; analyses may be marked incomplete when configured limits or review failures prevent completion.Source: §5.2 · pp. 12–14

5.3. Composite Analysis

• The interface layer deterministically models selector-to-impact and assessor-to-deference hand-offs using skill structure, tool signatures, return keys, code, and operative prose.It adds context edges for compatible selector and impact skills, and task-state edges where an assessment covers another skill’s action.Source: §5.3 · pp. 16–17
• Composite Analysis builds a dependency graph from shared resources, invocations, and skill interfaces, then optionally uses LLM-proposed dependencies and capability labels to detect additional cross-skill hand-offs.Model enrichment queries the input skill and selected partner skills, accepts only additions with valid references, predefined types, and justifications, and retains accepted additions even if processing is incomplete.Source: §5.3 · pp. 15–18
• Composite Analysis applies seven graph-based rules to skill dependencies, evaluates node, edge, and bounded-path candidates, and selects representative paths for validation.Path selection prioritizes fewer hops and preserves separate evidence for paths that include the input skill and those that do not.Source: §5.3 · p. 18

5.4. Validation

• Local candidates are accepted only when evidence verifies operative behavior, lower-trust reachability, and consequences beyond user-granted authority; composite candidates must satisfy producer, consumer, linkage, and consequence conditions.The verifier quotes evidence for each condition, and the module checks that evidence against cited files; unsupported acceptance is rejected.Source: §5.4 · pp. 19–20

5.5. Reporting

• Reporting records validated findings with evidence, categories, severity, source locations, and analysis provenance, and outputs structured JSON and SARIF.Reports also distinguish processing states and analysis failures; local findings may include value-flow or generated-program information, while composite findings retain contributing skills, paths, and edge confidence labels.Source: §5.5 · pp. 20–21

5.6. Mitigation

• Mitigation proposes skill substitution, code or instruction patches, and permission or approval changes, then tests whether changes block attacks while preserving the required task.For composite findings, the intervention may target a capability or instruction enabling the attack path, and testing covers that path across the installed skill set.Source: §5.6 · pp. 22–24

6.1. Implementation

• The evaluation used Python 3.13 and language-specific parsers, with Kimi K3 for model-assisted detection and validation except in RQ2 and RQ4, which used DeepSeek-V4-Flash.The evaluation included 1,587 skills from the 75 most-starred GitHub repositories, public single-skill benchmarks, and an adapted SCR-Bench set of 1,325 skill sets: 668 risky and 657 negative.Source: §6.1 · pp. 24–25, §6.2 · p. 25

6.2. Experimental Setup

• The adapted SCR-Bench evaluation labels risky skill compositions using recorded runtime outcomes and compares them with controls, cross-case shuffles, and clean skill pairs.For in-the-wild evaluation, 2,197 collected skills were reduced to 1,587 after excluding translated or per-agent copies; the 52-skill recall audit was drawn from eight repositories already containing confirmed findings.Source: §6.2 · p. 25, §6.3 · pp. 26–27

6.3. RQ1: Detection Accuracy in the Wild

• 53 of 59 reported findings were confirmed (89.8% precision), yielding 41 distinct vulnerabilities across 23 repositories.The 41 vulnerabilities involved code or command injection, persistent instruction injection, approval bypass, data or credential exfiltration, or sensitive-directory writes; none of the affected skills was malicious. Recall was 88.9% on a selected 52-skill reference set, which does not establish full-corpus recall; PoCs were developed for all 41 vulnerabilities.Source: §6.3 · pp. 26–27

6.4. RQ2: Comparison with Existing Scanners

• 96.95% precision, 95.05% recall, 95.99% F1, and 7.06% false-positive rate were achieved on full SkillTrustBench; Local Analysis led recall and F1 on its 556-case subset.On the MalSkillBench subset, it achieved the highest F1 (94.69%) and precision (98.70%), with 91.00% recall and a 1.20% false-positive rate.Source: Figure 5 · p. 28, §6.4 · pp. 28–29

6.5. RQ3: Composite Vulnerabilities

• Two composite vulnerabilities involving four skills in three repositories were identified, neither overlapping with the 59 Local Analysis findings.One paired a history reader with a Slack sender, allowing stored prompt injection to trigger unauthorized messages; the other paired a feed watcher with an OAuth gateway, exposing a bearer token. In PoC tests, injected instructions triggered unauthorized Slack requests in 2/12 GPT-5.2 runs and 7/12 GLM-5 runs; token disclosure occurred in 5/12 and 6/12 runs, respectively.Source: §6.5 · pp. 30–31

6.6. RQ4: Ablation Study

• 54.7% recall was achieved by the interface layer, which detected no SCR-AuthBlur pairs; model enrichment alone reached 0.967 F1, rising to 0.968 with the interface layer.The Composite Analysis ablation compared graph configurations with and without model-proposed edges, the interface layer, and validation over five runs.Source: §6.6 · p. 31, Table 8 · p. 32

6.7. RQ5: Mitigation Recommendations

• The Mitigation module recommended a mitigation for each of the 55 findings and selected a final option for each.For testable mitigations, the study replayed the corresponding PoC three times and checked whether the affected skill or skill set could still complete its documented task with legitimate approvals.Source: §6.7 · pp. 31–32
• Substitutes were selected for 13 of 16 code-based local findings; the other three received executable-code patches, while 37 instruction findings received patches alone or with configuration changes.Of the 37 instruction findings, 17 were assigned instruction patches alone and 20 patches combined with configuration changes; the two composite findings received different mitigations: a substitute with configuration change and an executable-code patch.Source: §6.7 · p. 32
• Tasks completed in the sandbox for 53 of 55 findings; two substitutes required Artifactory installations blocked by network restrictions, preventing execution-based verification.The substitutes’ documented behavior was reviewed against the original task requirements, but task completion could not be verified through execution.Source: §6.7 · p. 33
• Effectiveness required preventing the reported attack in all three repetitions; mere non-reproduction was insufficient, and configuration changes were assessed by blocking or approval behavior.Two findings requiring further files were not evaluated for effectiveness; configuration changes could leave the skill unchanged and the scanner finding reportable.Source: §6.7 · p. 33
• For the history-retrieval and Slack-sending composite, the module evaluated eight candidate senders and excluded openclaw/slack as incompatible with the Codex and Zcode harnesses.It checked each candidate’s sending mechanism, approval gate, environment compatibility, and Local Analysis results.Source: §6.7 · p. 33
• Among seven remaining substitutes, four lacked a required approval step, two relied on prompt-injection-vulnerable confirmation instructions, and slackapi/slack-messaging enabled separate approval for Slack messages.The module recommended replacing the original sender with slackapi/slack-messaging and configuring its send tool to require approval while retaining shell pre-approval.Source: §6.7 · pp. 33–34

7.1. Unintended Vulnerabilities in Skills

• Confirmed vulnerabilities arose in intended workflows where attacker-controlled content could influence agent actions, despite no evidence of malicious intent.The examples include treating issue details as authoritative instructions and inserting an externally influenced path into a shell command.Source: §7.1 · p. 34

7.2. Permissions and autonomy

• Permissions to read history and send Slack messages did not establish authorization to send retrieved information to a recipient specified by injected instructions.The discussion says permissions and autonomy shape the scope of actions an attacker can induce, and calls for runtime restrictions that account for data transfers.Source: §7.2 · p. 35

7.3. Screening through a common skill marketplace

• A common marketplace could provide shared screening information across participating repositories, but screening individual versions would not establish the safety of installed skill combinations.The proposed marketplace could display findings and analysis coverage alongside user ratings; composite analysis and runtime controls could complement that screening.Source: §7.3 · p. 35

7.4. Advantages over existing scanners

• The scanner’s separation of broad candidate generation from evidence-based validation produced the highest recall and F1 among the five scanners.Local checks generate candidates independently, after which Validation assesses source evidence and context; the compared scanners either miss issues or produce unsupported detections.Source: §7.4 · pp. 35–36

8.1. Static analysis

• The scanner does not execute skills, trace value flows in languages beyond Python, JavaScript, and shell, or follow network-fetched or runtime-generated code beyond its normalizations.Composite dependency matching uses literal paths and environment-variable names unless graph enrichment is enabled; installation-time content loaded in later sessions is also untraced.Source: §8.1 · p. 36

8.2. Recall on the in-the-wild corpus

• Because the in-the-wild corpus lacks ground-truth labels, recall was measured only on a reference set of eight repositories and 52 skills.The authors propose a larger, independently audited sample spanning more repositories and vulnerability types.Source: §8.2 · p. 36

8.3. Adversarial effectiveness of the scanner

• The injection checks cover only a limited set of patterns, and neither they nor the LLM instructions establish robustness against evasive or manipulative adversarial inputs.Evaluating robustness and developing stronger mechanisms remain future work.Source: §8.3 · p. 36

8.4. Composite benchmarks

• SCR-Bench is the only public benchmark with labeled multi-skill cases, so composite evaluation rests on that benchmark and the in-the-wild corpus.Future work could release additional labeled multi-skill benchmarks covering more dependencies and attack paths.Source: §8.4 · p. 36

9. Conclusion

• Local Analysis examines instructions, code, installation sources, and capability declarations, while Composite Analysis builds a dependency graph to identify potentially harmful paths across skills.Both analyses submit candidates to Validation, which checks supporting source evidence before reporting findings.Source: §9 · p. 37
• 53 of 59 findings across 1,587 skills were confirmed (89.8% precision), representing 41 distinct vulnerabilities in 23 repositories; Composite Analysis also achieved F1 0.991 on SCR-Bench.Local Analysis achieved 95.90% recall with a 4.22% false-positive rate on the 556-case SkillTrustBench subset and 94.69% F1 on the 1,000-skill MalSkillBench sample.Source: §9 · p. 37
• Mitigations prevented attacks in all three trials for 47 of 55 confirmed findings while preserving tested legitimate functionality; six were partially mitigated and two were unevaluable.The authors recommend complementing static analysis with runtime restrictions on resource access and approval controls over transmitted data and destinations.Source: §9 · p. 37

10. Ethics statement

• Composite vulnerabilities were tested in sandbox environments using byte-identical skill copies, canary credentials, and dummy listeners to observe disclosure of test credentials.Ten PoC attacks for medium- and high-severity local vulnerabilities were disclosed to maintainers; mitigations were tested for attack prevention and preservation of legitimate functionality.Source: §10 · p. 37