PaperlayerView exact revision on arXiv

Source-linked AI summary

From Instructions to Exploits: Applying Program Analysis to Malicious Agent Skills Across Natural Language and Code

Shiwu Zhao, Baizhu Wang, Guoqiang Zheng, Tingyu Gao, Yuheng Xie, Rujie Chen, Xingchen Chen, Xuezhang Liu, Qixu Liu

arXiv:2610.33489v1cs.CR

TL;DR

Agent skills combine natural-language instructions and bundled code, but existing approaches provide limited support for representing instruction operations and tracing dependencies across both domains. SCS compiles instructions into an analyzable representation, connects instruction and code dependence graphs, and propagates security properties for risk analysis. Across five LLMs, it reports the highest average accuracy, recall, and F1-score among evaluated methods, and confirms malicious skills in a large ClawHub audit.

  • Problem

    Existing skill-auditing approaches typically treat natural-language content as semantic evidence rather than compiling it into a constrained representation with explicit operations and dependencies, limiting its integration with code-level analysis.

    Source: §I · p. 2

  • Method

    SCS compiles natural-language instructions into an intermediate representation, connects instruction and code dependence graphs in xPDG, and propagates Trust, Sensitivity, and Capability for deterministic security-rule analysis.

    Source: §I · p. 2

  • Results

    85.6% average accuracy, 95.2% recall, and 86.9% F1-score were highest among evaluated methods across five LLMs; SCS confirmed 627 malicious skills among 66,157 ClawHub skills.

    Source: §I · p. 2, §VIII · p. 14

  • Takeaways & Limitations

    SCS demonstrates a program-analysis approach for reasoning about security-relevant behavior chains spanning heterogeneous skill artifacts.

    Source: §VIII · p. 13

    The limited API set is a security-oriented abstraction, not a complete semantic model of natural language, so some instructions may fall outside its current boundary.

    Source: 2) API Set Construction · p. 5

Abstract

from arXiv · show

LLM-powered agents increasingly rely on third-party skills that combine natural-language instructions with executable code, creating supply-chain risks that may span both artifacts. Existing skill scanners often rely on rule matching or direct LLM judgments and provide limited support for tracing such cross-domain behavior. We present the Supply-Chain Scanner for Agent Skills (SCS), a static analysis framework that unifies instructions and code for security analysis. SCS compiles natural-language instructions into a constrained Python-style intermediate representation using a limited set of security-oriented APIs. It then constructs program dependence graphs for instructions and bundled code and connects them into a cross-domain graph (xPDG). A Trust-Sensitivity-Capability (TSC) model propagates security properties over the graph to support structured, largely deterministic risk detection. On a manually verified benchmark of 117 malicious and 117 benign skills, SCS achieves the highest average accuracy (85.6%), recall (95.2%), and F1-score (86.9%) among evaluated methods across five LLMs, with 80.2% precision and average token consumption of 13.7 million. Its NL-to-IR compiler achieves 92.5%-94.5% step-level correctness on 400 human-annotated instruction steps. Finally, analyzing 66,157 real-world ClawHub skills yields 2,473 candidates, of which 627 are confirmed malicious through auxiliary dynamic validation and manual review.

I. INTRODUCTION

Agent skills combine natural-language instructions and bundled code, so security-relevant behavior can span artifacts rather than appear in either alone. SCS proposes unified analysis of both domains, addressing limits in existing approaches to tracing dependencies and making security decisions.

I. INTRODUCTION

  • Existing approaches typically inspect natural-language content as semantic evidence rather than compiling it into a constrained representation with explicit operations and dependencies, limiting integration with code analysis.The paper identifies implicit instruction behavior, cross-domain chain reconstruction, and dependence on LLM judgments as three core challenges.Source: §I · p. 2
  • SCS compiles natural-language instructions into an intermediate representation, connects instruction and code dependence graphs in xPDG, and propagates Trust, Sensitivity, and Capability for deterministic risk analysis.Its security rules identify injection-style, confidentiality-style, and other security-relevant behavior chains.Source: §I · p. 2
  • 85.6% average accuracy, 95.2% recall, and 86.9% F1-score were the highest among evaluated methods across five LLMs; average token consumption was 13.7M.The NL-to-IR compiler correctly translated over 92% of human-annotated instruction steps, and 627 malicious skills were confirmed from 66,157 ClawHub skills.Source: §I · p. 2

A. Threat Model over Agent Skill Supply Chain

  • The threat model considers a malicious third-party skill published through an open marketplace or repository and later executed by an agent on a user's behalf.The end user is the victim; the agent is semi-trusted and may faithfully execute malicious skill content, while attack surfaces include instructions, scripts, metadata, and auxiliary resources.Source: A. Threat Model over Agent Skill… · pp. 2–3

B. Motivation: Joint Reasoning over Natural-Language Instructions and Code

  • In the pptx_skill example, instructions require a backup script after presentation modification, and the script sends the resulting file to a predefined third-party server.The behavior is a local-file exfiltration scenario when the presentation contains sensitive information; its risk depends on both the workflow and the transmitted content.Source: B. Motivation: Joint Reasoning over… · p. 3
  • Treating every script that sends data externally as malicious could create false positives, while existing tools struggle to reconstruct instruction-to-code dependencies or rely heavily on LLM judgments.The paper argues that accurate risk assessment requires joint analysis of the instruction workflow and code implementation.Source: B. Motivation: Joint Reasoning over… · p. 4

III. XPDG: A UNIFIED PROGRAM DEPENDENCY REPRESENTATION FOR NL INSTRUCTIONS AND CODE

SCS represents natural-language instructions and bundled code in a shared graph so security-relevant dependencies can be traced across both domains. Its constrained IR uses security-oriented primitive APIs to make operations and data dependencies explicit, while remaining a limited abstraction of natural language.

III. XPDG: A UNIFIED PROGRAM DEPENDENCY REPRESENTATION FOR NL INSTRUCTIONS AND CODE

  • SCS connects instruction and code behavior in an xPDG, enabling unified representation of security-relevant dependencies across both domains.The graph supports the TSC-based security analysis described in Section IV.Source: §III · p. 4

1) IR Format:

  • A natural-language instruction step maps to zero or more IR statements, allowing non-operational steps to produce none and operational steps to express multiple primitives.For example, reading a file and uploading it requires separate file-read and network-send statements.Source: 1) IR Format · p. 4
  • Each IR statement invokes one limited-API operation, with arguments supplied as literals or previously defined variables.Return-producing APIs assign a generated variable; APIs without a return value use a standalone invocation.Source: 1) IR Format · p. 4
  • Variable use makes cross-step data dependencies explicit, such as passing file content produced by read_file to http_post.The IR exposes operations, arguments, and dependencies that are implicit in free-form instructions.Source: 1) IR Format · p. 4

2) API Set Construction:

  • The API set abstracts recurring behaviors observed in software supply-chain attacks and agent-skill security studies into a compact set of primitive operations.The cited analyses include file or credential access, command execution, downloads, modifications, and data exfiltration.Source: 2) API Set Construction · p. 4
  • The limited API set covers security-relevant primitives including file and command operations, network communication, environment access, installation, and cross-skill invocation.The set also includes user input and pure data transformation; each API models one primitive operation.Source: 2) API Set Construction · pp. 4–5
  • SCS compiles operational steps into IR statements but excludes non-operational content such as metadata, formatting guidance, and descriptive text.This distinction avoids forcing every natural-language sentence into an API call, reducing hallucinated operations not present in the skill.Source: 2) API Set Construction · p. 5
  • The API set is not a complete model of natural language, so some instructions may fall outside its current abstraction boundary.It is intended to model the main behavior classes relevant to skill supply-chain risks.Source: 2) API Set Construction · p. 5

B. NL-to-IR Compilation

SCS compiles skill instructions into constrained Python-style IR, using segmentation, batching, and validation to manage LLM translation and its errors. A symbol table preserves references across batches, while cleaning and legality checks normalize and verify generated statements before later analysis.

B. NL-to-IR Compilation

  • SCS translates natural-language instructions into constrained Python-style IR, with error-handling and verification mechanisms intended to improve compilation reliability.Compilation errors can affect the Text-PDG and downstream security analysis.Source: B. NL-to-IR Compilation · p. 5

1) Instruction Segmentation and Batch Construction:

  • SCS divides documents at structural markers into ordered instruction steps, then batches consecutive steps within the token budget while preserving order and local relationships.A symbol table carries each generated variable’s meaning and defining step into subsequent batches, helping resolve later references to earlier results.Source: 1) Instruction Segmentation and Batch… · p. 5

2) LLM Compilation:

  • For each batch, the LLM receives the compilation objective, limited API set, required IR format, current symbol table, and instruction steps.Operational steps may produce one or more statements; non-operational steps produce none.Source: 2) LLM Compilation · p. 6

3) IR Cleaning:

  • SCS removes comments, Markdown, explanations, and other non-IR output to produce a canonical representation, without judging the resulting statements’ syntactic or semantic validity.The cleaning stage normalizes format; legality checking follows separately.Source: 3) IR Cleaning · p. 6

4) IR Legality Checking:

  • Legality checking tests syntax, API membership and permitted constructs, argument signatures, and variable definitions; failed statements return to the LLM for revision and rechecking.The syntax parse uses tree-sitter.Source: 4) IR Legality Checking · p. 6

5) Parameter Dependency Reconstruction:

SCS reconstructs dependencies in instruction-derived IR, connects instruction and code analyses in an xPDG, and propagates Trust, Sensitivity, and Capability states to support security-risk detection.

5) Parameter Dependency Reconstruction:

  • SCS repairs lost producer–consumer links by replacing literals with previously defined IR variables only when API signatures and instruction ordering permit the rewrite.This addresses legal IR statements that nonetheless lose a dependency when an earlier result is represented as a literal.Source: 5) Parameter Dependency Reconstruction · pp. 6–7

C. Building xPDG across NL and Code

  • The Text-PDG and supported Python, JavaScript, and Shell Code-PDG are integrated into an xPDG that jointly traces dependencies across instructions and code.The illustrative path reads sensitive data in an instruction and passes it to a script that sends it through a network operation.Source: C. Building xPDG across NL and Code · p. 7

A. TSC Security Property Definition

  • TSC records variable-level Trust and Sensitivity alongside the security-relevant operations that contributed to a value.Trust captures provenance and reliability, Sensitivity captures confidentiality, and Capability tracks operations such as file reads.Source: A. TSC Security Property Definition · p. 7

B. TSC Transfer Rules and State Propagation

  • Each API output state combines an API-specific base state with input-derived states propagated along specified arguments and TSC dimensions.When no transfer rule is defined, input states are joined; multiple contributions merge to the less trustworthy Trust level, higher Sensitivity, and union of Capabilities.Source: B. TSC Transfer Rules and State… · p. 8
  • SCS refines states using statically known literal arguments and iterates downstream propagation until every variable state stops changing.A sensitive file path can raise a returned value’s Sensitivity, while a trusted network destination can lower its returned value’s Trust risk.Source: B. TSC Transfer Rules and State… · pp. 8–9

C. xPDG-Based Security Rule Evaluation

SCS evaluates propagated states and graph structure with deterministic rules for risky state-operation combinations, capability provenance, and agent-control events.

C. xPDG-Based Security Rule Evaluation

  • A sensitive input reaching an outbound payload argument of a network-capable API triggers a potential data-exfiltration risk under the Trust/Sensitivity-to-operation rules.These rules match an input’s Trust or Sensitivity threshold together with the invoked API’s required Capability.Source: C. xPDG-Based Security Rule Evaluation · p. 9
  • Capability-provenance rules flag flows such as an OBFUSCATED value entering an API with EXEC Capability as potential covert execution.The rule checks both the operation provenance carried by the input and the Capability of the current API.Source: C. xPDG-Based Security Rule Evaluation · p. 9
  • An attempt_policy_override event is checked with an evidence-constrained LLM because its occurrence alone does not establish an operational policy violation.The LLM receives the exact instruction, surrounding text, normalized IR statement, and matched rule.Source: C. xPDG-Based Security Rule Evaluation · p. 9

V. EVALUATION

SCS is evaluated against open-source scanners and academic methods on a manually verified, balanced benchmark. Across five LLMs, it leads in average accuracy, recall, and F1-score while using fewer tokens than A.I.G.

1) Datasets:

  • The manually verified benchmark contains 117 malicious and 117 benign skills, for 234 samples total.The malicious samples come from MalSkills and MaliciousAgentSkillsBench; benign samples are retained from those sources and supplemented with highly installed ClawHub skills.Source: 1) Datasets · p. 9

2) Baselines:

  • SCS is compared with three open-source scanners and four academic baselines selected for controlled, reproducible evaluation.Snyk Agent Scan is excluded because its proprietary APIs and internal signature database prevent a fully controlled and reproducible comparison.Source: 2) Baselines · p. 9

B. RQ1: Comparison with SOTA Baselines

  • LLM-using methods are evaluated across five representative models, while SkillFortify and SkillLeakBench are run once because their static analyses are deterministic.The evaluation also records input-plus-output token consumption for each LLM-based method on the full benchmark.Source: B. RQ1: Comparison with SOTA Baselines · p. 10
  • SCS achieves the highest average accuracy (85.6%), recall (95.2%), and F1-score (86.9%) across five LLMs, with 80.2% precision and 13.7M average tokens.A.I.G. has higher average precision (82.0%) but lower recall (84.3%) and F1-score (82.9%), and uses 56.9M average tokens; Table III and Figure 5 summarize detection and token results, respectively.Source: B. RQ1: Comparison with SOTA Baselines · pp. 10–11, Table III · p. 10, Figure 5 · p. 10

C. RQ2: Quality of NL-to-IR Compilation

The evaluation tests NL-to-IR compilation against human-annotated instruction steps, then applies SCS to a ClawHub snapshot. Compilation is highly accurate across five LLMs, and manual review confirms hundreds of malicious skills in the marketplace.

1) Gold Set Construction:

  • The compiler is evaluated on 400 randomly sampled ClawHub skills, with one instruction step selected per skill to avoid over-representing longer instructions.The gold set uses property constraints rather than a single fixed reference IR, accommodating semantically equivalent compilations while checking required calls, grounding, dependencies, and abstention.Source: 1) Gold Set Construction · p. 11

2) Results:

  • Across five LLMs, NL-to-IR achieves 92.5%–94.5% overall step-level accuracy, with every individual property accuracy above 94%.A step counts as correct only when all five compilation properties are satisfied; the results are reported in Table IV.Source: 2) Results · pp. 11–12, Table IV · p. 12

D. RQ3: Real-World Malicious Skill Analysis

  • On a snapshot of 66,157 ClawHub skills, SCS produces 2,473 candidates, of which 627 are confirmed malicious through auxiliary dynamic evidence and manual review.Dynamic validation supplies supporting evidence; final maliciousness decisions are made through manual review.Source: D. RQ3: Real-World Malicious Skill… · p. 12
  • Two publishers account for 88.04% of confirmed cases, and all confirmed malicious skills from those publishers had been removed by the July 15, 2026 follow-up check.Only 23 of the 627 confirmed skills remained identifiable in the later registry records.Source: D. RQ3: Real-World Malicious Skill… · p. 12

A. Coverage of Security-Relevant Behaviors

SCS addresses hybrid skill security by analyzing natural-language instructions and bundled code together, while its conclusions are bounded by API/rule coverage, static observability, and LLM interpretation uncertainty. The paper reports strong benchmark and real-world results, alongside prior work spanning agent-skill attacks, auditing, program analysis, and procedural-language representations.

A. Coverage of Security-Relevant Behaviors

C. LLM-Related Uncertainty

  • NL-to-IR compilation and context-dependent judgments for ambiguous instructions rely on LLMs, which may omit or hallucinate operations or infer incorrect dependencies.Constrained IR, legality checks, and dependency reconstruction reduce but do not eliminate this uncertainty.Source: C. LLM-Related Uncertainty · p. 12

B. Security Studies on Agent Skills

C. Static Analysis and Information-Flow Techniques

VIII. CONCLUSION

  • SCS unifies instruction and code analysis through constrained NL-to-IR compilation, xPDG dependence modeling, and TSC-based security-property propagation.This design supports reasoning over behavior chains spanning heterogeneous skill artifacts.Source: §VIII · p. 13
  • Across five LLMs, SCS achieves the highest average accuracy (85.6%), recall (95.2%), and F1-score (86.9%); its audit of 66,157 ClawHub skills confirms 627 malicious skills.The real-world findings were confirmed through auxiliary dynamic validation and manual auditing.Source: §VIII · pp. 13–14

ETHICS CONSIDERATIONS

  • The study analyzes publicly available skill artifacts and uses isolated environments, synthetic files, and nonfunctional canary credentials for dynamic validation.It reports that potentially malicious findings are manually reviewed before disclosure.Source: ETHICS CONSIDERATIONS · p. 14

APPENDIX

The appendix documents xPDG structure and TSC propagation, the supported API-property setup, AI assistance, and open-science artifacts.

A. Generative AI Usage

  • The authors report using GPT-5.4 for linguistic refinements and Claude Code to assist with parts of the experimental codebase and analysis scripts, with manual review and repeated validation.Validation included code inspection, repeated experiment execution, and consistency checks.Source: A. Generative AI Usage · p. 15

B. Open Science

  • The anonymous repository provides the SCS implementation and evaluation artifacts, including datasets, configurations, benchmark comparisons, and analysis results.The implementation includes NL-to-IR compilation, xPDG construction, TSC propagation, and security-rule evaluation.Source: B. Open Science · p. 15

C. Node and Edge Types in xPDG

  • xPDG represents skills, functions, statements, interfaces, and control predicates, with edges for containment, invocation, data flow, bindings, and control dependence.These connections preserve within-domain structure and link dependencies across instruction and code domains.Source: C. Node and Edge Types in xPDG · p. 15, Table V · p. 16

D. TSC Propagation Algorithm

  • TSC propagation uses a Kildall-style worklist to revisit dependent statements when states change, combining information arriving through multiple predecessors until a fixed point.Termination follows from a finite TSC domain and monotonic transfer operations.Source: D. TSC Propagation Algorithm · p. 16

E. Limited IR API Security Properties

  • Limited IR APIs receive base Trust and Sensitivity properties for return values and an intrinsic Capability for the API invocation; transfer rules and literal conditions can refine them.The full API-property table and supported language-specific function lists are documented in the paper’s artifact.Source: E. Limited IR API Security Properties · pp. 16–17

1) Sample Construction and Dataset Characteristics:

The NL-to-IR evaluation gold set samples one instruction step from each of 400 distinct ClawHub skills and tests whether annotations preserve required operations, literals, dependencies, and abstention decisions.

1) Sample Construction and Dataset Characteristics:

  • The gold set contains 400 instruction steps from 400 randomly selected distinct skills, covering varied document forms, instruction lengths, and nine behavioral categories.Instruction lengths range from 25 to 476 characters, with a median of 78.5 and an average of 97.5 characters.Source: 1) Sample Construction and Dataset… · p. 17

2) Annotation Contract:

  • The annotation contract records required and prohibited API calls, literals to preserve, producer-consumer dependencies, and whether an instruction should abstain from executable IR.The labels constrain semantics while allowing equivalent IR implementations.Source: 2) Annotation Contract · p. 17

3) Annotation Example:

  • For the example instruction to read a field from USER.md and extract its value, the gold annotation requires both operations, preserves the file and field identifiers, and links their dependency.The annotation specifies semantic constraints without prescribing variable names or a particular IR program.Source: 3) Annotation Example · p. 17

4) Annotation Procedure and Criteria:

The evaluation uses independent, context-aware annotation with adjudication, and the real-world study supplements static findings with controlled dynamic validation rather than treating it as part of SCS’s primary detection pipeline.

4) Annotation Procedure and Criteria:

  • Two annotators independently label each instruction using its surrounding context, record only text-supported behavior, and resolve remaining disagreements through a third annotator.They calibrate the guideline first, calculate Cohen’s κ for each property, and record ambiguity rather than forcing speculative labels.Source: 4) Annotation Procedure and Criteria · p. 17

G. Dynamic Validation of Real-World Findings

  • Auxiliary dynamic validation gathers runtime evidence for statically identified candidates; it does not replace static analysis or independently classify all skills.Validation is conducted at the finding level, while the real-world results are reported at the skill level.Source: G. Dynamic Validation of Real-World… · pp. 17–18

1) Validation Workflow:

  • The validation workflow exercises each candidate in a disposable sandbox with synthetic files and nonfunctional canaries, then correlates runtime events with SCS’s source, sink, and evidence path.An instrumented agent loads the skill through its intended activation mechanism; deterministic rules classify each finding as confirmed, refuted, or inconclusive.Source: 1) Validation Workflow · p. 18

2) Static-Guided Skill Execution:

Dynamic validation turns SCS findings into ordinary skill tasks, then checks whether monitored execution supports their predicted security consequences. Its coverage-aware decisions distinguish confirmation, refutation, and inconclusive runs, with runtime evidence linked back to findings for real-world review.

2) Static-Guided Skill Execution:

  • For each candidate finding, validation constructs ordinary user-request tasks from the skill’s declared functionality, relevant instructions, and SCS-reported source-to-sink path.Tasks may use templates or LLM assistance to improve path coverage, but neither the triggering model nor executing agent decides whether a skill is malicious.Source: 2) Static-Guided Skill Execution · pp. 18–19

3) Isolation and Runtime Evidence:

  • Disposable isolated runs use synthetic files and unique canaries, allowing observed sensitive-data use or disclosure to be attributed to a specific skill and execution.In one example, a canary SSH key appearing in an outbound request directly supports a finding that the skill reads and sends a private key.Source: 3) Isolation and Runtime Evidence · p. 19, 2) Static-Guided Skill Execution · p. 19
  • Agent, process, file-access, and network events are associated with the validation task and static finding, so runtime behavior can be compared with SCS’s reported path.The dynamic result is linked to the original finding and used with manual inspection when confirming malicious skills in the real-world study.Source: 3) Isolation and Runtime Evidence · p. 19, 4) Validation Decisions · p. 19

4) Validation Decisions:

  • Runtime validation confirms a finding when execution reaches the suspicious operation and evidence supports its predicted consequence, such as a canary secret in an outbound request.A finding is refuted only when its location is exercised under complete monitoring and the predicted consequence is absent; otherwise, insufficient coverage or evidence is inconclusive.Source: 4) Validation Decisions · p. 19

Related papers

Loading 2610.33489v1…