PaperlayerView exact revision on arXiv

Source-linked AI summary

"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild

Yi Liu, Zhihao Chen, Yanjun Zhang, Gelei Deng, Yuekang Li, Jianting Ning, Leo Yu Zhang

arXiv:2602.06547v4cs.CRcs.AIcs.CLcs.ET

TL;DR

Agent skills give LLM-based coding agents reusable instructions and helper code with full user privileges, but the ecosystem lacked reliable labeled evidence of malicious behavior. The paper analyzes 98,380 skills using static triage and behavioral verification, identifies 157 confirmed malicious skills with 632 vulnerabilities, and characterizes attack coordination, concealment, and ecosystem impact. Its dataset and pipeline provide public ground truth for evaluating detection systems, within a scope limited to two public registries and excluded skill categories.

  • Problem

    The ecosystem lacked ground-truth data to distinguish malicious intent and executed abuse from suspicious patterns or developer mistakes in agent skills.

  • Method

    The study combines static candidate identification, sandboxed behavioral verification, manual vulnerability labeling, and statistical analysis of skills collected from two community registries.

  • Results

    The analysis confirms 157 malicious skills containing 632 vulnerabilities across 13 attack techniques, identifies two negatively correlated attack archetypes, and finds that evasion increases with sophistication.

  • Takeaways & Limitations

    The publicly released labeled dataset and detection pipeline provide ground truth for evaluating future detection systems and support actionable threat intelligence through confirmed cases.

  • Takeaways & Limitations

    The dataset covers two public registries as of January 2026 and excludes private or unindexed repositories, direct-sharing or enterprise deployments, MCP servers, and official curated collections.

Abstract

from arXiv · show

LLM-based coding agents increasingly rely on third-party extensions called skills, which bundle natural language instructions and helper scripts that execute with full user privileges. Community registries have emerged to distribute these skills, but the security implications remain unstudied due to the absence of labeled threat data. This paper presents a systematic security analysis of 98,380 skills collected from two major registries. Through a combination of static pattern matching and dynamic behavioral verification, we identify 157 skills exhibiting confirmed malicious behavior, encompassing 632 distinct vulnerabilities across 13 attack techniques. Our analysis reveals that these threats are deliberate rather than accidental: each malicious skill contains an average of 4.03 vulnerabilities spanning multiple attack phases. We identify two dominant attack strategies with statistically significant negative correlation -- credential theft via remote code execution, and agent manipulation through adversarial instructions embedded in documentation. Over half of all confirmed cases originate from a single threat actor employing templated brand impersonation at scale. We further observe that attack sophistication correlates with concealment investment, with advanced skills universally employing undocumented capabilities while also exploiting platform-native trust mechanisms. Following responsible disclosure, registry maintainers removed all 157 (100%) of the reported skills. Our dataset and detection pipeline are publicly available to facilitate future research on securing LLM agent ecosystems.

1 Introduction

Agent skills extend LLM agents through reusable instructions and code, but their broad privileges and weak scrutiny create serious security risks. This study builds labeled ground truth and measures the ecosystem’s threat landscape, attack strategies, evasion mechanisms, and disclosure impact.

  • Motivation: 98,380 skills were indexed across community registries within three months of launch, reflecting rapid ecosystem expansion.Skills package reusable workflows, domain expertise, and tool integrations for LLM agents.
  • Security Risk: Full local user privileges and minimal confirmation allow malicious skills to execute arbitrary code or manipulate agent behavior.A documented real-world example embeds a reverse shell that connects to an attacker-controlled server.
  • Findings: 632 vulnerabilities span 13 attack techniques and 6 kill chain phases, while malicious skills average 4.03 vulnerabilities across a median of 3 phases.The authors interpret this multi-phase layering as consistent with deliberate rather than accidental inclusion.
  • Findings: Two negatively correlated attack archetypes emerge: Data Thieves use supply-chain exfiltration, while Agent Hijackers subvert instructions.A single industrialized actor accounts for 54.1% of malicious skills through templated brand impersonation.
  • Findings: Evasion scales with sophistication: shadow features rise from 0% at the basic level to 100% at the advanced level, alongside attacks on platform-native trust mechanisms.Responsible disclosure led to removal of all 157 reported skills, or 100%.
  • Study Contributions: The study constructs a labeled dataset of 157 behaviorally confirmed malicious skills and 632 vulnerabilities, with a detection pipeline reproducing the 98,380→4,287→157 funnel.Behavioral verification achieves 99.6% precision versus ≤1.1% for the strongest static-only baseline.

2 Background and Related Work

Agent skills combine persistent natural-language instructions with optional local helper code, creating both code-level and instruction-level attack surfaces. Prior static analyses identify potential risk but cannot reliably confirm malicious intent or execution, motivating behavioral verification and labeled ground truth.

  • Agent Skill Mechanisms: Agent skills are file-based packages that couple persistent SKILL.md instructions with optional Python, Shell, or JavaScript helper code.Unlike MCP, they execute locally with user privileges and are supported by several coding-agent frameworks.
  • Attack Surface: Skill ecosystems differ from code-only extension systems because pre-granted privileges eliminate the exploitation stage and natural-language files can directly embed malicious behavior.These properties expose both code-level and instruction-level attack vectors.
  • Prior Work: 26.1% of skills were flagged as potentially risky by one static approach, while Cisco’s scanner marked 10.7% as CRITICAL.Neither effort verifies whether suspicious patterns reflect intentional abuse or benign functionality.
  • Research Gap: Behavioral verification and a labeled dataset address the gap between potential risk indicators and confirmed malicious behavior.The paper’s contribution is to distinguish confirmed malice from developer mistakes and unexecuted suspicious paths.

3 Methodology

The methodology constructs a behaviorally grounded dataset from two community registries, filters skills through static and LLM-based analysis, and verifies malicious behavior dynamically. It then labels vulnerabilities and analyzes attack chains, associations, and sophistication while acknowledging the dataset’s coverage boundaries.

  • Definitions and threat model: The study defines maliciousness by intentional abusive behavior, including credential harvesting, unauthorized transmission, remote code execution, privilege escalation, and instruction manipulation.
  • Definitions and threat model: The threat model treats skill publishers as adversaries pursuing data theft, agent hijacking, or persistence through code-level and instruction-level attacks.
  • Data collection: 98,380 skills were collected from skills.rest and skillsmp.com, while excluding private, unindexed, directly shared, enterprise, MCP, and platform-curated skills.
  • Candidate identification: The pipeline combines static scanning of SKILL.md and bundled scripts with LLM-based analysis of instruction-level attack vectors.
  • Analysis and validation: The analysis maps behaviors to six attack phases and tests pattern associations using Fisher’s exact tests, distribution comparisons, correlations, Bonferroni correction, and Louvain clustering.
  • Behavioral verification: 4,287 candidates matched suspicious patterns, and behavioral verification confirmed 157 malicious skills, representing 3.7% of candidates and 0.16% of the full corpus.

4 Measurement Study

The measurement study characterizes confirmed malicious skills by attack scope, technique and phase coverage, documentation-based attack surfaces, and attacker sophistication. It then examines co-occurring patterns to distinguish coordinated strategies within the 157-skill dataset.

  • Study Design: 98,380 skills were processed, with analyses centered on 157 confirmed malicious skills and 632 labeled vulnerabilities.The study organizes measurement around threat-landscape characterization, coordination, and concealment questions.
  • Attack Scope and Severity: 4.03 vulnerabilities per malicious skill and 71.8% rated CRITICAL or HIGH indicate broad and severe attacks.The distribution peaks at four vulnerabilities, while 80.3% of skills contain at least three, suggesting deliberate technique layering.
  • Attack Techniques: Remote Script Execution, Behavior Manipulation, and Credential Harvesting are the three most prevalent patterns, accounting for 25.2%, 18.8%, and 17.7% of vulnerabilities, respectively.Instruction-level attacks collectively account for 33.1% of vulnerabilities, while execution-oriented patterns are usually critical and supporting techniques are less severe.
  • Natural Language Attack Surface: 84.2% of vulnerabilities are embedded in SKILL.md documentation, compared with 8.5% in executable code and 7.3% in configuration or other sources.The authors identify coercive language, false urgency, hidden directives, and secrecy instructions as documentation-based attack content requiring NLP-based detection.
  • Kill Chain Phase Coverage: Execution, Impact, and Credential Access dominate kill-chain coverage at 75.2%, 69.4%, and 68.2%, while Reconnaissance and Defense Evasion occur in 7.6% and 15.9%.The study attributes the lower need for reconnaissance and evasion to skills executing with pre-granted user privileges.
  • Sophistication Stratification: 84.1% of malicious skills are Intermediate or Advanced, and vulnerability density, kill-chain coverage, and E2→E1 prevalence increase 2.6×, 4.1×, and from 3.8% to 90.0% across sophistication levels.The ecosystem is middle-heavy, with 77.7% of malicious skills at Intermediate level and 6.4% at Advanced level.

A Negative Association Reveals Two Archetypes

The analysis separates malicious skills into two negatively correlated attack archetypes: Data Thieves prioritize supply-chain exfiltration, whereas Agent Hijackers target instruction-level control. Evasion increasingly relies on undocumented behavior and platform-native trust mechanisms rather than code obfuscation alone.

  • Attack Archetypes: OR=0.11 (p < 0.001), φ = −0.41, indicates that remote script execution and instruction override rarely co-occur.They co-occur in only 8 skills (5.1%), far below chance expectation.
  • Attack Archetypes: Data Thieves form the dominant archetype, using hardcoded endpoints, remote execution, and credential harvesting for supply-chain exfiltration.They include 110 skills (70.1%) with SC2 without P1 and are enriched for E2, E1, and SC3.
  • Attack Archetypes: Agent Hijackers use instruction overrides, coercive language, secrecy, and silent operation to subvert agent decision-making and suppress user notification.Examples include hidden BCC behavior and explicit prohibitions against consulting the user.
  • Attack Archetypes: 54.1% of malicious skills came from one industrialized actor using templated brand impersonation, with the E2+SC2 fingerprint yielding OR=556.1 (p < 0.001).The fingerprint appeared in 83 of 85 smp_170 skills (97.6%) versus 5 of 72 non-smp_170 skills (6.9%).
  • Evasion and Concealment: 115 of 157 malicious skills (73.2%) contained shadow features, including undocumented endpoints, conditional triggers, obfuscated code, or invisible-markup instructions.Shadow features were defined as capabilities not inferable from public documentation and independently verified by two researchers.
  • Evasion and Concealment: Shadow features increased from 0% at Level 1 to 86.1% at Level 2 and 100% at Level 3, while obfuscation rose from 0% to 7.4% to 60.0%.Every Level 3 skill used at least one evasion technique, whereas no Level 1 skill did.
  • Evasion and Concealment: Documentation-behavior comparison is a higher-yield detection heuristic than code-pattern matching because most concealment operates outside the code layer.Only 15 skills (9.5%) employed code obfuscation, and traditional obfuscation detectors would miss the primary concealment mechanism.
  • Platform-Native Attacks: Platform-native attacks target model substitution, hook manipulation, and other infrastructure trust mechanisms, forming an emerging threat class specific to agent ecosystems.One model-substitution skill redirected Claude API calls through an attacker-controlled proxy, while hook-based skills monitored and exfiltrated tool activity.

5 Discussion

The ecosystem has split into two attack philosophies: scalable, fingerprintable credential theft and instruction-level manipulation that traditional tooling does not address. Defensive priorities therefore include natural-language analysis, platform-native hardening, and prevention rather than only post-hoc removal.

  • Two Archetypes, Two Threat Models: 54.1% of malicious skills came from one actor using 85 brand-impersonating templates with 100% consistency.The E2+SC2 fingerprint identifies this class with OR=556 and 97.6% sensitivity.
  • Two Archetypes, Two Threat Models: Agent Hijackers manipulate instruction-following agents through safety overrides, notification suppression, and permission-seeking suppression.The attack surface exists because an AI agent, unlike a human reader, follows these documentation-level directives.
  • Defense Implications: 84.2% of vulnerabilities were concentrated in SKILL.md, making instructional-text analysis essential alongside code-level scanning.Relevant signals include coercive language, secrecy directives, and autonomy overrides; 73.2% of malicious skills contain shadow features.
  • Defense Implications: Three platform-native attacks abused permission bypasses, shipped MCP configuration, and hooks that intercept tool operations.These mechanisms require permission scoping, credential rotation, hook sandboxing, and semantic monitoring before instructions reach the LLM.
  • Remediation: 157/157 reported skills were removed, but a three-month undetected window shows reactive removal does not prevent user exposure.The removal rate validates maintainer responsiveness rather than prevention.
  • Limitations: The January 2026 snapshot, 60-second analysis window, possible environment-gated behavior, taxonomy limits, and manual judgment constrain interpretation.A 10% sample of unconfirmed candidates contained 93.2% dormant triggers, suggesting the confirmed set is a lower bound; single-actor concentration also affects aggregate statistics and public-registry results may not generalize to private deployments.

6 Conclusion

The paper presents the first large-scale measurement of malicious agent skills, combining a labeled dataset with systematic analysis of attack techniques, archetypes, evasion, and platform-native abuse. It finds a rapidly maturing, concentrated ecosystem and releases the dataset and pipeline for future research.

  • Conclusion: 98,380 skills yielded 157 behaviorally confirmed malicious skills containing 632 vulnerabilities.The dataset is presented as the first large-scale measurement of this ecosystem.
  • Conclusion: Malicious skills averaged 4.03 techniques across a median of 3 kill chain phases, indicating layered attacks.The ecosystem bifurcated into negatively correlated Data Thief and Agent Hijacker archetypes.
  • Conclusion: 54.1% of observed activity came from one industrialized actor using templated brand impersonation.The study also reports increasing evasion with attacker sophistication and platform-native attacks.
  • Conclusion: 157/157 reported skills were removed after responsible disclosure, and the complete dataset and analysis pipeline were released.The paper proposes longitudinal, cross-platform, and external validation work for future research.

Ethical Considerations

The study addresses ethical risks through sandboxed verification, controlled disclosure, anonymization, and safeguards against misuse. It also identifies limitations that constrain the ethical interpretation of its findings, including possible undercounting and single-actor skew.

  • Harms and Mitigations: Static candidates ran in ephemeral Docker sandboxes with no persistent storage, real credentials, or unmonitored egress, and execution was capped at 60 seconds.Crawling respected rate limits and did not bypass access controls, scrape private repositories, or collect telemetry.
  • Deception: Synthetic honeypot credentials were used only in-sandbox to trigger exfiltration; no human, maintainer, real account, key, or token was exposed.
  • End-user exposure during remediation: 157/157 (100%) reported cases are now removed or deindexed after prepublication disclosure with remediation evidence.At submission, 147/157 had been removed; continued maintainer review completed the process.
  • Reputational harm from false positives: 99.6% precision and multiple-rater agreement supported reserving “Malicious” for behaviorally confirmed intentional abuse.The study anonymized identifiers, withheld author names and repository URLs, and deferred ambiguous dual-use cases to maintainers.
  • Adversarial misuse of the artifact: Working exploits, live endpoints, and malicious-repository links were withheld, while aggregate statistics and synthesized snippets were reported.Full malicious samples require controlled access, whereas detection logic, labels, co-occurrence matrices, and scripts are released openly.
  • Limitations with Ethical Implications: Dormant triggers, environment-gated behavior, and the conservative 60-second sandbox window may undercount harm, while smp_170 contributes 85 templated skills (54.1%).The confirmed set is therefore a high-precision lower bound, and per-skill aggregates are partly affected by single-actor skew.

Open Science

The paper follows an open-science policy by releasing its labeled malicious-skill dataset and detection pipeline. Public artifacts support reproducibility while controlled access limits exposure of full malicious samples.

  • The permanent artifact archive is publicly available through the project website and Zenodo.
  • The released dataset includes 632 vulnerability instances across 13 patterns, shadow annotations, sophistication assignments, and attack-chain classifications.
  • The detection pipeline releases the crawler, static-analysis rules, behavioral-verification harness, co-occurrence matrices, and hypothesis-testing scripts.These components support reproduction of the analysis funnel from registry-wide collection to confirmed malicious skills.
  • Aggregate labels and detection logic are openly available, while full malicious samples require a documented controlled-access process.The paper uses synthesized code examples and withholds repository URLs to prevent misuse.

A.1 Complete Pattern Specification

The detection pipeline combines code-level pattern matching, instruction-level analysis, metadata extraction, and dynamic verification to identify potentially malicious skills and observe their runtime behavior.

  • Pattern coverage: 14 detection patterns cover code-level threats, instruction-level manipulation, permissions, and other skill behaviors.Instruction-level detection targets instruction override, hidden instructions, and behavior manipulation.
  • Instruction-level detection: GPT-5.2 analyzes SKILL.md content for instruction override, hidden instructions, and behavior manipulation using structured JSON outputs.The prompt examines attempts to supersede instructions, concealed directives, and coercive language steering agents toward unsafe actions.
  • Dynamic verification: Synthetic testing requests typical-use, edge-case, and adversarial invocations based on each skill’s name, description, and entry points.These generated invocations support dynamic behavioral verification.
  • Dynamic verification: Docker execution is monitored with tcpdump, strace, inotify watches, and fake API keys, with each run timing out after 60 seconds.The monitors capture outbound connections, network and file activity, sensitive-path access, and credential harvesting.

B Statistical Analysis Details

The analysis examines kill-chain co-occurrence, statistical associations, severity differences, and the behavior of the static-analysis pipeline and evaluation dataset.

  • Kill-chain analysis: 89 skills combine Execution with Credential Access, while 82 combine Execution with Impact in kill-chain co-occurrence analysis.The reported combinations correspond to credential harvesting followed by alert suppression and other impact behaviors.
  • Hypothesis tests: E2 and E1 are associated: Fisher’s exact test reports p = 0.020, OR=2.24, 95% CI [1.09, 4.66], and Cramér’s V = 0.19.The contingency table contains 58 skills with both phases, 43 E2-only, 21 E1-only, and 35 with neither.
  • Hypothesis tests: Prompt-injection skills have mean severity 3.33 versus 3.12 without prompt injection, with Mann-Whitney U=2903.0 and p = 0.008.The groups contain n=38 and n=119 skills, respectively; the result is significant at α = 0.01.
  • Severity analysis: Severity is assigned per vulnerability instance, while each skill’s overall severity equals its highest instance severity.The specification categorizes vulnerabilities from CRITICAL through LOW according to attack impact and ambiguity.
  • Evaluation data: The evaluation dataset contains 300 skills split evenly between 150 benign and 150 malicious examples.Benign skills were randomly sampled from 94,093 skills without static-analysis flags, while malicious skills came from the 157 confirmed set.
  • Static-analysis limitations: Regex-based static analysis produces false positives because it lacks semantic understanding and treats documentation and test files like production code.For example, re.compile() can trigger code-injection alerts when a rule matches compile() without understanding context.

F.3 False Negative Analysis

False-negative analysis shows that most unconfirmed candidates may contain dormant malware whose triggers exceed the sandbox’s single-session conditions, while a smaller fraction are legitimate tools or truly missed attacks.

  • False-negative categories: 93.2% of the 413 reviewed unconfirmed candidates were dormant malware missed because conditional triggers did not activate within the sandbox.Triggers included time delays, production-specific environments, root access, and multi-session state persistence.
  • False-negative categories: 4.9% of reviewed candidates were legitimate security tools whose defensive behavior produced suspicious static patterns.Examples include penetration-testing and scanning utilities.
  • False-negative categories: 1.9% were truly missed malicious skills, including second-order attacks that execute payloads in later sessions.Extrapolated to the full candidate set, this represents approximately 78 skills requiring multi-session monitoring.
  • Validation design: Behavioral verification supplies ground truth after static analysis reduces the corpus from 98,380 to 4,287 candidates.An E2 pattern is confirmed only when credential transmission is observed, alongside dual-researcher judgment on all 157 confirmed skills.

G Implementation Details

The implementation uses marketplace crawlers, combined static detectors, metadata extraction, and resource-limited container execution to process the skill corpus.

  • Collection and analysis: Marketplace crawlers collect skills from skills.rest and skillsmp.com using paginated API requests and search enumeration.The pipeline combines regex matching for code-level patterns, GPT-5.2 analysis for instruction-level patterns, and metadata extraction for permissions and triggers.
  • Execution environment: Experiments run on a 64-core AMD EPYC server with 256 GB RAM and 2 TB NVMe storage.Containers are limited to 2 GB memory and 60-second timeouts.
Loading 2602.06547v4…