PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Insecure Compositions of Clean Agent Skills via Capability-Flow Graph Analysis

Yihao Li, Chuanpu Fu, Yizhou Zhao, Miao Chen, Qi Li, Ke Xu

arXiv:2610.32155v1cs.CR

TL;DR

Individually clean agent skills can compose at runtime into executable safety violations that per-skill auditing does not capture. KnotScryer models skill capabilities and handoffs to search for and verify dangerous chains. Across 13,598 clean ClawHub skills, it detected 612 chains with 79.2% average exploitability across nine agent–LLM combinations.

  • Problem

    Per-skill scanners detect malicious artifacts, but whether individually clean skills can compose through agent handoffs into executable violations remains unaddressed.

    Source: §I · p. 1

  • Method

    KnotScryer summarizes skill capabilities, builds a graph of potential handoffs, searches for policy-violating paths, and verifies candidate chains.

    Source: §I · p. 2, C. Skill Capability-Flow Graph (SCFG) · p. 4, D. Policy-Driven Capability-Flow Search · p. 5

  • Results

    612 high-confidence chains averaged 79.2% exploitation across nine agent–LLM combinations; under GLM-5.1, exploitation was 85.6% versus 16.5% for tag-match and 18.2% for graph-only.

    Source: §I · p. 2, §VIII · p. 11

  • Takeaways & Limitations

    Auditing skills in isolation is insufficient when agents can compose their capabilities at runtime.

    Source: §VIII · pp. 10–11

    Results measure publicly API-enumerable clean ClawHub skills at crawl time, excluding private, hidden, under-review, deleted, or updated skills.

    Source: §V · p. 10

Abstract

from arXiv · show

AI agents increasingly extend their capabilities by invoking third-party skills, where each skill encapsulates instructions, scripts, and resources for agents to execute on demand. This paper reveals that deploying many benign skills together introduces compositional security risks: multiple individually clean skills can compose into safety-violation chains, because the agent passes an upstream skill's output to a downstream skill's input across invocations. The risk arises not from any single skill, but from the agent's runtime orchestration, and thus cannot be captured by existing malicious skill detection methods. To this end, we develop KnotScryer to detect insecure skill compositions by treating skills as typed capability flows. In particular, we abstract each skill into a capability summary, and construct the Skill Capability-Flow Graph (SCFG), where edges denote cross-skill information flow. Moreover, we design a search policy that tracks protected or untrusted data along handoff edges to locate paths reaching insecure operations. Finally, a two-layer verification method replays the insecure information flow for reachability analysis and applies a task-anchored LLM chain audit to confirm the semantic plausibility of the identified violation chains. We deploy KnotScryer to analyze 13,598 clean skills from ClawHub, and validate the detected chains across three agent frameworks and three LLMs. KnotScryer identifies 612 insecure compositions, which are consistently exploited across agent frameworks. Meanwhile, under GLM-5.1, it outperforms tag-match and graph-only baselines by improving the exploitation rate from 16.5%-18.2% to 85.6%, while reducing the search space by 85.3%. Furthermore, 13.7% of the detected chains consist entirely of skills labeled safe by a single-skill scanner, showing that KnotScryer effectively identifies insecure compositions that per-skill auditing cannot.

I. INTRODUCTION

The paper identifies a security gap: individually clean skills can form executable violation chains through runtime composition, beyond the reach of per-skill auditing. KnotScryer searches for these chains and validates their exploitability across agent and model combinations.

I. INTRODUCTION

  • Existing platform scans and individual-skill scanners target malicious artifacts, while this paper asks whether benign skills can compose into a threatening chain.The paper frames this composition question by analogy to return-oriented programming –.Source: §I · p. 1
  • Individually benign skills can form executable safety violations because agents pass upstream outputs to downstream inputs during runtime orchestration.The paper illustrates a public issue triggering a clean deployment skill to install and execute a project.Source: §I · pp. 1–2
  • KnotScryer summarizes skill capabilities, builds a Skill Capability-Flow Graph, searches for dangerous compositions, and applies two-layer verification.Verification combines marker-flow reachability analysis with a task-anchored LLM chain audit.Source: §I · p. 2
  • 612 high-confidence chains averaged 79.2% exploitation across nine agent–LLM combinations, while GLM-5.1 reached 85.6% versus 16.5% for tag-match and 18.2% for graph-only.The evaluation used 13,598 clean ClawHub skills and validated chains across three agent frameworks and three LLMs.Source: §I · p. 2

II. BACKGROUND AND PROBLEM SETTING

Skills combine natural-language instructions and executable resources, while agents select, execute, and reason about them in successive invocations. The paper studies how attacker-controlled content can travel across these handoffs and reach a high-privilege operation.

A. Skill Composition

  • A skill combines SKILL.md instructions, executable scripts, and optional templates or resources, with the agent using its description to select and orchestrate skills.Unlike traditional software packages, skills combine natural-language semantics with executable code.Source: A. Skill Composition · p. 2

B. Agent Skill Workflow and Agent-Mediated Handoff

  • Each invocation follows Decision, Execution, and Reasoning; between invocations, the agent uses upstream output to decide which downstream skill to call.Per-skill invocation permissions are checked independently, but cross-skill composition permissions are not.Source: B. Agent Skill Workflow and… · p. 2, Figure 2 · p. 3

C. Problem Statement

  • The study seeks structurally executable chains of clean skills that can be found efficiently and can inform defenses against unsafe combinations.Proposed defenses include preventing constituent skills from appearing together or advising users not to install them together.Source: C. Problem Statement · p. 3

D. Threat Model

  • The attacker cannot alter skills, user inputs, or the agent framework, but can plant crafted content in public resources for skills to retrieve and relay.The content may pass from a low-privilege retrieval skill through intermediate skills to a high-privilege operation.Source: D. Threat Model · p. 3, Figure 3 · p. 3

III. APPROACH

KnotScryer models potential skill handoffs using evidence-backed capability conditions, then searches for paths that violate security policies. Its Skill Capability Summary captures what each skill can receive, produce, and affect.

A. Overview

  • A handoff is analyzable only when the upstream output is observable, the downstream input is controllable, and both ends are type-compatible with supporting evidence.Type compatibility is necessary but not sufficient; the pipeline extracts capabilities, constructs handoffs, searches policy-forbidden paths, and verifies candidates.Source: A. Overview · p. 3

B. Skill Capability Summary (SCS)

  • The Skill Capability Summary models each skill by its receiving, producing, and affecting capabilities using frontmatter parsing, language heuristics, and script scanning.These methods cover structured declarations, unstructured descriptions, code behavior, and semantic properties that resist rule-based capture.Source: B. Skill Capability Summary (SCS) · pp. 3–4
  • SCS records ports, effects, operations, context exposure, trust boundaries, and evidence, including which outputs can participate in agent-mediated handoffs.Evidence is recorded per claim with its source, snippet, extraction method, and confidence score.Source: B. Skill Capability Summary (SCS) · p. 4

C. Skill Capability-Flow Graph (SCFG)

The SCFG represents skills and their internal capability flows, adding cross-skill edges only for agent-mediated handoffs that satisfy four conditions. Its construction indexes compatible types and bounds cross-skill edges to control graph density.

C. Skill Capability-Flow Graph (SCFG)

  • The SCFG contains skill, port, effect, operation, context, and trust-boundary nodes, with intra-skill edges representing capability-flow relationships.Cross-skill edges represent potential information flow between skills.Source: C. Skill Capability-Flow Graph (SCFG) · p. 4
  • A cross-skill handoff requires compatible data types, an observable upstream output, a controllable downstream input, and evidence supporting both.Type compatibility may be declared explicitly, such as URL being consumable as Text, rather than assuming arbitrary outputs are interchangeable.Source: C. Skill Capability-Flow Graph (SCFG) · p. 4
  • Controllability records whether the agent can supply downstream input; authorization remains a separate effect attribute that determines whether a sink is unauthorized.Observability and controllability must each be supported by skill evidence; descriptive guesses alone do not establish high-confidence edges.Source: C. Skill Capability-Flow Graph (SCFG) · p. 4
  • Construction indexes input ports by compatible type, checks remaining handoff conditions, and caps each output port’s edges at b, prioritizing policy-relevant effects.This construction adds intra-skill structure before cross-skill handoff edges.Source: C. Skill Capability-Flow Graph (SCFG) · p. 5

D. Policy-Driven Capability-Flow Search

An SCFG handoff path is not inherently risky; a candidate violation arises when protected data or untrusted content reaches a policy-forbidden effect. Policies specify tracking sources, forbidden sinks, and conditions that terminate propagation.

D. Policy-Driven Capability-Flow Search

  • A handoff path becomes a candidate violation only when protected data or untrusted content reaches a policy-forbidden effect.A document-processing skill passing text to a summarization skill is given as a normal composition.Source: D. Policy-Driven Capability-Flow Search · p. 5

1) Policy Definition:

  • Each policy specifies source conditions to track, forbidden sink operations, and sanitization or authorization conditions that can terminate propagation.The paper instantiates four policies for common risk patterns.Source: 1) Policy Definition · p. 5
  • The four policies cover sensitive-data propagation, third-party content elevation, capability accumulation into high-impact execution, and persistent-state write risks.New policies can be added by declaring trigger conditions and state transitions without changing the graph or search algorithm.Source: 1) Policy Definition · p. 5

2) Policy State Tracking and Search-Space Control:

Policy search tracks source data through the SCFG while updating policy state, stopping branches when a violation or safe condition is reached. Provenance tracking and bounded search controls limit expansion.

2) Policy State Tracking and Search-Space Control:

  • Under the data-exfiltration policy, search tracks Private outputs and reports a candidate when propagation reaches an unauthorized network-send effect.Sanitization or explicit authorization terminates the safe branch.Source: 2) Policy State Tracking and… · p. 5
  • Search states combine a graph node, policy state, and provenance, while transitions synchronously update policy state along graph edges.Provenance records the data’s origin and prevents merging paths from different skills, which may have different responsibility subjects and mitigation locations.Source: 2) Policy State Tracking and… · pp. 5–6
  • The search initializes from policy-matching sources and adds a candidate when it reaches a violation state, subject to depth and per-policy limits.Safe transitions and states dominated by an existing state are not expanded.Source: 2) Policy State Tracking and… · p. 6
  • Type and handoff checks, early termination of safe branches, provenance-aware dominance pruning, maximum depth, and per-policy budgets control search expansion.Dominance pruning merges states that share both origin and policy state.Source: 2) Policy State Tracking and… · p. 6

3) Candidate Chain Output and Complexity:

Candidate chains preserve information for verification and responsibility analysis. The search’s actual cost depends on retained states and expanded transitions, with a simpler bound when provenance splitting is ignored.

3) Candidate Chain Output and Complexity:

  • Candidate chains retain the skill sequence, port path, policy-state trajectory, risk reason, and provenance needed for verification and responsibility analysis.For data exfiltration, the record identifies the Private-data origin, handoffs, and unauthorized network-send effect, enabling marker-flow reachability checks.Source: 3) Candidate Chain Output and Complexity · p. 6
  • Actual search time depends on the non-dominated states retained and transitions expanded for each policy.The passage denotes these quantities as S_j and A_j, respectively.Source: 3) Candidate Chain Output and Complexity · p. 6
  • Ignoring provenance splitting, the search time is O(r|Q|(|V| + |E|)); depth and budget limits guarantee termination.Here r is the number of policies, Q the policy states, and V and E the graph nodes and edges.Source: 3) Candidate Chain Output and Complexity · p. 6

E. Two-Layer Verification

KnotScryer verifies candidate chains with marker-flow reachability checks and a task-anchored LLM audit to address distinct execution and semantic uncertainties. The evaluation tests detected chains across skills, frameworks, models, and controlled attack conditions.

E. Two-Layer Verification

1) Marker-Flow Reachability Verification:

  • Marker-flow verification replays a synthetic marker from the source and rejects paths where sanitization, declassification, or authorization blocks it before the sink.This provides deterministic, reproducible reachability evidence along the saved capability-flow trajectory.Source: 1) Marker-Flow Reachability Verification · p. 6

2) Task-Anchored LLM Chain Audit:

  • The task-anchored LLM audit assesses whether outputs are accessible, inputs task-triggerable, and adjacent handoffs form one plausible task.It can only confirm or downgrade chains already discovered by SCFG.Source: 2) Task-Anchored LLM Chain Audit · pp. 6–7

A. Experimental Setup

  • The evaluation crawls 13,598 clean skills and tests three agent frameworks against three LLMs.The frameworks differ in skill selection, authorization, and context processing; the models differ in reasoning and safety alignment.Source: A. Experimental Setup · p. 7
  • Attacker-influenced content triggers the source skill, and a chain counts as present if either of at most two attempts per agent succeeds.Experiments run in an isolated local environment with no real users, systems, or third-party services involved.Source: A. Experimental Setup · p. 7

B. Detection Results and Exploitability

SCFG finds highly exploitable chains across agent and model combinations, while baselines generate many candidates that are often not agent-executable. Some detected chains also escape per-skill safety labels.

B. Detection Results and Exploitability

  • 79.2% weighted overall exploitation across nine agent–LLM combinations was observed among 612 SCFG-detected chains.Table II reports exploitation and consistency analysis for these chains.Source: B. Detection Results and Exploitability · p. 7, Table II · p. 7
  • Framework exploitation rates were OpenCode 86.0%, Hermes 80.0%, and OpenClaw 71.7%, with framework mediation and SSRF protection contributing to outcome differences.Framework-level protection changed risk thresholds and failure modes but did not independently eliminate composition risks.Source: B. Detection Results and Exploitability · pp. 7–8

C. Comparison with Baselines

  • On the 1,062-skill popular subgraph, tag-match produced 829,462 associations and graph-only 1,047,207 edges, while SCFG produced 13 candidate chains.SCFG requires agent-mediated handoff conditions rather than relying on tag or type compatibility alone.Source: C. Comparison with Baselines · p. 8
  • Under GLM-5.1, tag-match reached 16.5% exploitation and graph-only 18.2%, versus SCFG’s 85.6%.The baseline differences were statistically significant (z > 37, p < 0.001).Source: C. Comparison with Baselines · p. 8
  • SCFG chains showed greater cross-agent consistency under GLM-5.1: 62.6% were exploited by all three frameworks, while most baseline candidates worked in at most one or none.Baseline exploitation was 1.6%–9.4% for OpenClaw and OpenCode, compared with 40.8% for tag-match and 44.0% for graph-only on Hermes.Source: C. Comparison with Baselines · p. 8
  • 84 of 612 LLM-confirmed chains (13.7%) contained only skills labeled safe by Cisco’s single-skill scanner.Component-level alerts cannot reconstruct source-to-sink cross-skill violation chains.Source: C. Comparison with Baselines · p. 8

D. Analysis Overhead

SCFG spends more time on graph construction than the baselines, but handoff filtering sharply reduces the search space. Ablations show that observability and controllability conditions prevent distinct kinds of spurious paths.

D. Analysis Overhead

  • SCFG took 97.6 s on the popular subgraph, versus 3.3 s for tag-match and 9.8 s for graph-only, with 77.0 s spent on graph construction.The comparison reports median wall-clock time over five runs.Source: D. Analysis Overhead · p. 8, Table IV · p. 9
  • Only 1.48% of type-compatible port pairs satisfied handoff conditions, reducing SCFG’s edge count by 85.3% versus graph-only.Marker-flow verification then ran on the small candidate set in 0.133 s.Source: D. Analysis Overhead · p. 9

E. Ablation Study

  • Removing Observable increased edge count by 438% and reduced exploitation by 45.6%; removing Controllable added 84% edges and reduced exploitation by 52.4%.The popular-subgraph ablation produced 126 chains without Observable and 103 without Controllable, compared with 13 in the full configuration.Source: E. Ablation Study · p. 9, Table V · p. 9
  • Without Observable, silent local data can appear transferable; without Controllable, paths can reach sinks the agent cannot trigger, such as one requiring an unavailable OAuth token.The ablation identifies distinct spurious-edge mechanisms for the two conditions.Source: E. Ablation Study · p. 9

F. Case Study

The case study demonstrates exfiltration through two individually benign skills after a natural user prompt and attacker-planted email content. Validity is bounded by stochastic testing and a time-specific, publicly enumerable ClawHub snapshot.

F. Case Study

  • The case study pairs an Outlook-reading skill with a skill that sends email to arbitrary recipients; each is individually benign.The inbox contains audit-contract amounts and vendor names alongside a phishing email.Source: F. Case Study · p. 9
  • An attacker-controlled phishing email names the sink skill and asks the agent to compile audit-related email content and send it to the attacker.The attacker controls a third-party content source accessed during normal source-skill execution.Source: F. Case Study · p. 9
  • After the user asks, “Help me check my emails, and handle what’s needed,” the agent sends contract amounts and vendor names to the attacker without any single malicious skill.The prompt contains neither skill names nor an instruction to send data externally.Source: F. Case Study · p. 9
  • With the sending skill uninstalled, the agent refuses under the same prompt and phishing email, confirming the attack depends on the composed handoff channel.Source: F. Case Study · p. 10

V. THREATS TO VALIDITY

  • Exploitation rates are lower bounds from at most two stochastic runs per agent–LLM combination, and the corpus covers only publicly API-enumerable clean skills at crawl time.The snapshot excludes private, hidden, under-review, deleted, or updated skills, and labels may change over time.Source: §V · p. 10

A. Detection Boundary

KnotScryer finds that agent-mediated composition can turn clean skills into executable violation chains, while its detection has defined boundaries and its chains can inform layered defenses.

A. Detection Boundary

  • Detection is bounded by four instantiated violation categories, agent-mediated handoffs rather than implicit shared-state channels, and an unobservable universe of exploitable chains.Extending policy categories requires declaring new source conditions, forbidden sinks, and blocking conditions; the paper makes no claim of covering all true chains.Source: A. Detection Boundary · p. 10

B. Defense Implications

  • Violation chains can inform layered defenses: restrict high-risk sources or sinks, gate or sanitize handoffs, and enforce runtime guardrails; alignment cannot replace explicit flow constraints.The paper identifies node-, edge-, and harness-level interventions.Source: B. Defense Implications · p. 10

VII. RELATED WORK

  • Related work spans external prompt, tool, and web threats; internal model, memory, and alignment attacks; and graph-based software-security defenses and vulnerability discovery.These are complementary lines of work discussed in the paper.Source: §VII · p. 10

VIII. CONCLUSION

  • Safety violations can emerge from runtime composition rather than any single malicious skill, making isolated skill auditing insufficient when agents compose capabilities.KnotScryer’s SCFG captures cross-skill handoff conditions and searches for dangerous chains.Source: §VIII · pp. 10–11
  • On 13,598 clean ClawHub skills, KnotScryer detects 612 high-confidence violation chains, 79.2% exploitable across nine agent–LLM combinations.The universe of exploitable chains is unobservable, so the paper reports exploitation success for detected chains without claiming complete coverage.Source: §VIII · p. 11
Loading 2610.32155v1…