Source-linked AI summary
SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills
Yinghan Hou, Zongyou Yang
TL;DR
Agent skills combine prose and executable code under inherited privileges, creating malicious behavior that single-modality analyzers can miss. SkillSieve addresses this with layered static, semantic, and jury-based triage, achieving strong benchmark performance at low cost and extending to Feishu/Lark packages. Its main boundaries are runtime-invisible payloads, same-author evaluation bias, and nondeterministic LLM outputs.
Problem
Agent skills combine natural-language instructions with executable code and inherited agent privileges, while attacks can span prose and files beyond regex or code-only analysis.
Method
SkillSieve combines recall-oriented static triage, four parallel LLM security tasks, and an independent three-model jury with debate for disagreement.
Results
F1 = 0.929 (precision 0.912, recall 0.945) on a 390-skill benchmark, while scanning 49,592 skills costs an estimated $0.006 per skill.
Takeaways & Limitations
The architecture supports both offline batch scanning and interactive enterprise vetting, including a deployed Feishu chat bot and contextual analysis of Feishu/Lark packages.
Takeaways & Limitations
Runtime monitoring, dynamic analysis, binaries, and payloads visible only after execution are out of scope, while same-author-heavy evaluation makes fast-path savings a distribution-specific ceiling.
Abstract
from arXiv · showhide
Agent skills combine natural-language instructions with executable code while inheriting an agent's filesystem, credential, and network access. Attacks can span prose and files, whereas regex and code-only analyzers cover only one modality. SkillSieve applies three progressively deeper layers: recall-oriented regex, AST, and metadata triage; four parallel LLM security sub-tasks; and an independent three-model jury with debate on disagreement. We evaluate 49,592 real ClawHub skills, a 390-skill labeled benchmark, and 100 adversarial samples across five evasion techniques on a 440 USD ARM board. The full pipeline achieves F1 = 0.929 (precision 0.912, recall 0.945) at an average cost of $0.006 per skill. An optional XGBoost fast path reduces Layer-2/3 calls by 32% with a 1.7-point F1 decrease while preserving recall. On 52 Feishu/Lark packages, Layer 2 reclassifies 13 of 14 Layer-1 flags as safe after contextual analysis; we also deploy the system as a Feishu chat bot. Code, labels, and aggregate results are open-sourced.
1 Introduction
SkillSieve addresses malicious agent skills that combine natural-language instructions with executable code and inherited privileges. It uses progressively deeper analysis, semantic decomposition, and independent jury review to balance coverage, cost, and uncertainty.
- Agent skills combine prose and optional scripts while inheriting credential, filesystem, and network access, enabling attacks that cross modalities.
- Static scanners miss instructions and cross-file intent, while a single LLM lacks an independent uncertainty signal.
- A three-layer static–semantic–jury pipeline allocates analysis by risk, filtering obvious cases cheaply and escalating high-risk cases.
- Structured Semantic Decomposition splits suspicious skills into four parallel, independently inspectable security tasks.
- A three-model jury uses debate and conservative human-review routing for unresolved high-risk cases.
- The evaluation covers 49,592 archived skills, a 390-skill reviewed benchmark, five adversarial families, 52 Feishu/Lark packages, and a deployed Feishu bot.
2 Background and Related Work
Agent skill ecosystems combine metadata, natural-language instructions, and optional scripts under agent privileges, creating risks that span code and prose. Prior work documents related scanners, analyzers, LLM systems, and broader skill-attack taxonomies.
- ClawHub contains 49,592 skills across roughly 16,800 publisher namespaces, combining instructions with optional code executed under agent privileges.
- Reported audits identify prompt injection, typosquatting, cross-file credential exfiltration, and infostealers, with vulnerability rates varying across corpora.
- Related systems combine static patterns, metadata, dependency, typosquatting, formal code analysis, LLM or multi-agent analysis, and optional neural classification.
3 Threat Model
The threat model considers adversaries publishing skills to steal credentials, exfiltrate data, execute remote code, or socially engineer agents. The defender seeks high recall with manageable false positives and API cost, while runtime-only behavior is out of scope.
- Attackers may steal credentials, exfiltrate data, execute remote code, or socially engineer agents through published skills.
- Evasion includes encoding, cross-file splitting, conditional or delayed triggers, and homoglyph substitution.
- The defender inspects text files and calls remote LLMs before installation but never executes the package, prioritizing recall, false positives, and API cost.
- Runtime monitoring, dynamic analysis, binaries, and payloads visible only after execution are outside the text-only detection scope.
4 The SkillSieve Framework
SkillSieve progressively deepens analysis from recall-oriented static triage to parallel semantic assessment and multi-model voting with debate. Its design filters most packages cheaply while preserving suspicious cases for contextual review.
- Overview: Layer 1 combines four static modules with a recall-tuned heuristic and filters about 86% of packages at zero API cost.
- Layer 1: Static modules inspect rules, parsed code, metadata, and operational features, while parse failures continue to semantic analysis rather than being discarded.
- Fast path: The optional XGBoost fast path misses malicious skills at every tested threshold and is therefore retained only as an optional branch.
- Layers 2–3: Layer 2 computes a weighted risk score from four parallel sub-tasks, while Layer 3 independently votes and debates when jurors disagree.
- Layer 3: The jury returns a unanimous verdict when possible, otherwise revotes after debate and routes remaining contested cases to human review.
- Layer 1: The heuristic maps static signals to r ∈[0, 1], sending skills below τ to Safe; τ = 0.3 was selected on a recall-1.0 plateau.
4.3 Layer 2: Structured Semantic Decomposition
Layer 2 decomposes semantic security judgment across four parallel sub-tasks covering instructions, permissions, and cross-file consistency. Their scores are aggregated into a weighted risk score that determines escalation to Layer 3.
- Task Decomposition: Four sub-tasks separately analyze intent alignment, covert behavior, permission justification, and cross-file consistency.The decomposition addresses both natural-language instructions and executable code or capabilities.
- Task Decomposition: Cross-file consistency checks whether scripts and instructed commands match SKILL.md, catching split logic and script-less external payloads.
- Parallel Execution: All four tasks run concurrently, making total latency the maximum single-task latency, typically 2–5 seconds.
- Risk Aggregation: Each sub-task returns a risk score s_i ∈[0, 1], which contributes to the Layer 2 risk score.
- Risk Aggregation: Layer 2 computes R2 as a weighted sum of the four sub-task scores and sends skills with R2 ≥0.4 to Layer 3.The weights are (0.35, 0.25, 0.25, 0.15) for Tasks A–D, respectively.
4.4 Layer 3: Multi-LLM Jury Protocol
Layer 3 uses three independent model jurors to produce auditable verdicts and introduces structured debate when they disagree. Its reporting preserves an evidence chain and recommended action for malicious findings.
- Independent Voting: GLM-5.1, Qwen3-235B, and DeepSeek-V3.1 independently inspect each skill and prior-layer evidence before voting.
- Structured Debate: Unanimous first-round verdicts are final, while disagreement triggers a second round in which jurors review one another’s evidence and revote.
- Decision Rule: A unanimous or 2-of-3 MALICIOUS vote is accepted, whereas a 2-of-3 SAFE split is held as malicious and sent to human review.The protocol reflects the higher cost of false negatives.
- Explainable Reports: Malicious verdicts include the attack type, a three-layer evidence chain, and a recommended action.
5 Dataset Construction
The evaluation dataset combines a large ClawHub archive, documented malicious samples, campaign samples, and a repeatedly human-reviewed labeled set. It also includes adversarial skills spanning five evasion techniques.
- Data Sources: The dataset draws from the ClawHub archive, Snyk ToxicSkills, ClawHavoc samples, and a human-reviewed set.
- Data Sources: 49,592 ClawHub skill packages across approximately 16,800 publisher namespaces form the full archive source.
- Human Review: The human-reviewed benchmark contains 390 skills: 55 malicious and 335 benign, with labels reviewed across multiple rounds and a final source-level audit.The audit corrected one high-risk but non-malicious developer-tool skill from malicious to benign.
- Annotation Schema: Each skill receives a binary benign/malicious label, multi-label attack types from seven categories, and a stealth rating from 1 to 5.
- Dataset Assumption: Publisher namespaces are counted by the archive’s first path component, representing repository-level identities rather than verified individual people.One person or organization may control multiple namespaces.
- Adversarial Evaluation: The adversarial set covers encoding obfuscation, cross-file logic splitting, conditional triggers, homoglyph substitution, and time-delayed payloads.Each sample combines one evasion technique with a credential-theft payload in a benign skill template; 100 samples provide 20 per technique.
6 Evaluation
Evaluation shows that SkillSieve combines layered semantic and static analysis to preserve recall while improving precision, intercepting adversarial variants, and reducing remote analysis cost on large skill corpora.
- Main Results: Precision rises from 0.342 to 0.663 after semantic analysis without reducing recall, while the jury removes 23 of 28 remaining false positives.Four contested cases are routed to human review.
- Adversarial Robustness: 100 generated adversarial variants are detected across five bypass techniques, including encoding, cross-file splitting, conditional triggers, homoglyphs, and time delays.Each variant combines an evasion technique with a credential-theft payload.
- Efficiency: $0.006 is the estimated average cost per skill, using c1 ≈$0, p2 = 0.14, and c2 + p3c3 ≈$0.04.The cost model assigns per-skill costs to the layers and conditions later-layer costs on escalation probabilities.
- Efficiency: 49,592 skills are processed in 31 minutes on a $440 ARM board, with 6,871 sent to LLM analysis, a 7.2× reduction.Layer 1 runs on-device and filters approximately 86% of packages before remote analysis.
- Efficiency: 32% fewer downstream calls are achieved by the fast path at τhigh = 0.98, while F1 decreases from 0.929 to 0.912 and recall remains unchanged.The full pipeline remains the headline result.
- Limitations: New-author streams require recalibration because the 32% fast-path saving is a same-distribution ceiling and GroupKFold F1 is 0.671.The reported fast-path saving may be lower outside the benchmark’s author distribution.
7 Cross-Ecosystem Generalization
SkillSieve generalizes to Feishu/Lark packages through lightweight format and pattern adapters, while contextual Layer 2 analysis resolves most Layer 1 false positives and supports interactive deployment.
- Adaptation: The Feishu adapter maps SKILL.md packages, OpenClaw plugins, and standalone bot repositories into SkillSieve’s expected input format.It reuses the existing parser for SKILL.md skills and collects platform-specific source files for scanning in other formats.
- Adaptation: Layer 1 was extended with JavaScript/TypeScript patterns for credential reads, outbound requests, shell execution, Feishu endpoints, and hardcoded webhooks.These rules address platform-specific code patterns not covered by the original Python and shell-oriented rules.
- Evaluation: 52 Feishu/Lark packages were discovered across official, community, and independent repositories for cross-ecosystem evaluation.The corpus included 20 official CLI skills, 9 official plugin skills, 9 community plugin skills, 8 alternative CLI skills, and packages from six standalone repositories.
- Layer 1 results: Layer 1 flagged 14 of 52 packages and classified 38 as safe, averaging 30.0 ms per package.The flags included a genuine plaintext credential-storage concern in feishu-bridge.
- Layer 2 results: 13 of 14 Layer 1 flags were reclassified as safe by Layer 2 after contextual analysis, leaving only feishu-bridge flagged.Examples included timestamps, long API URLs, and anti-injection documentation that matched generic detection rules but were benign in context.
- Deployment: SkillSieve was deployed as a Feishu chat bot that scans GitHub URLs or pasted content and returns live, color-coded evidence cards.The deployment uses a WebSocket long connection and requires no public IP.
- Takeaway: The cross-ecosystem architecture preserves the core pipeline while requiring format mappers and platform-specific pattern rules for new enterprise platforms.The paper identifies DingTalk, WeChat Work, QQ Bot, and Baidu Comate as potential targets for the same adaptation strategy.
8 Discussion
The discussion identifies static-analysis blind spots, nondeterministic LLM behavior, and domain-driven false positives as important boundaries, while proposing runtime monitoring and broader ecosystem data as next steps.
- Limitations: Static analysis cannot see payloads fetched at runtime, and time-delayed attacks remain difficult because malicious logic appears inert during scanning.The scope is therefore bounded by file-visible, pre-execution evidence.
- Limitations: LLM outputs remain nondeterministic even at temperature 0, so the archived API run is a fixed-run snapshot rather than a bit-for-bit reproducibility guarantee.This uncertainty applies to Layers 2 and 3.
- Limitations: Domain-inherent patterns caused roughly a third of Layer 2 false positives in DeFi or wallet skills involving private-key handling and exchange-API calls.The paper suggests domain-aware prompting and max-based aggregation as possible remedies.
- Future work: Runtime behavioral monitoring is proposed to catch payloads missed by static analysis, while a small fine-tuned open model could remove Layer 2’s API dependency.The paper also identifies cross-ecosystem data and JavaScript/TypeScript-specific rules as priorities.
9 Conclusion
SkillSieve combines progressively deeper static, semantic, and jury analysis to detect malicious agent skills at low cost. It achieves strong benchmark performance, intercepts tested evasions, generalizes to Feishu/Lark, and supports real-time enterprise vetting.
- Conclusion: F1 = 0.929, with precision 0.912 and recall 0.945, on a 390-skill benchmark drawn from 49,592 real ClawHub skills.Layer 1 filters 86% of input at zero cost; Layer 2 raises precision from 0.342 to 0.663 while retaining recall 1.000, and the jury raises precision to 0.912.
- Conclusion: 49,592 skills were scanned on a $440 ARM board in 31 minutes at an estimated corpus-average API cost of $0.006 per skill.Only 13.86% of skills were escalated to remote LLM analysis.
- Conclusion: All five tested adversarial bypass techniques were intercepted, including conditional triggers, homoglyph typosquatting, and time-delayed payloads.The result comes from the adversarial evaluation described for the full pipeline.
- Conclusion: On 52 Feishu/Lark packages, Layer 2 reclassified 13 of Layer 1’s 14 flags as safe, and the system was deployed as a production Feishu chat bot.The architecture is presented as supporting both offline batch scanning and interactive enterprise use.