PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale

Yi Liu, Weizhe Wang, Ruitao Feng, Yao Zhang, Guangquan Xu, Gelei Deng, Yuekang Li, Leo Zhang

arXiv:2601.10338v1cs.CRcs.AIcs.CLcs.SE

TL;DR

Agent skills extend AI agents with instructions and executable code, but implicit trust and limited vetting leave their security risks poorly characterized. The paper collects and analyzes marketplace skills using SkillScan, finding vulnerabilities across the ecosystem and motivating stronger permission and vetting practices.

  • Problem

    The paper addresses the lack of systematic evidence about the prevalence, categories, and distribution of vulnerabilities in real-world agent skills.

  • Method

    The authors collect 42,447 skills from two marketplaces, analyze 31,132 unique skills, and apply SkillScan, combining static analysis with LLM-based semantic classification.

  • Results

    26.1% of analyzed skills contain at least one vulnerability across 14 patterns, while executable-script skills are 2.12x more likely to be vulnerable than instruction-only skills.

  • Takeaways & Limitations

    The findings support capability-based permissions, mandatory security reviews, permission sandboxing, and runtime monitoring for agent skills.

  • Takeaways & Limitations

    The accessible December 2025 ecosystem may understate prevalence because 17.3% of skills were deleted before analysis, potentially creating survivorship bias.

Abstract

from arXiv · show

The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities. While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface. We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces and systematically analyzing 31,132 using SkillScan, a multi-stage detection framework integrating static analysis with LLM-based semantic classification. Our findings reveal pervasive security risks: 26.1% of skills contain at least one vulnerability, spanning 14 distinct patterns across four categories: prompt injection, data exfiltration, privilege escalation, and supply chain risks. Data exfiltration (13.3%) and privilege escalation (11.8%) are most prevalent, while 5.2% of skills exhibit high-severity patterns strongly suggesting malicious intent. We find that skills bundling executable scripts are 2.12x more likely to contain vulnerabilities than instruction-only skills (OR=2.12, p<0.001). Our contributions include: (1) a grounded vulnerability taxonomy derived from 8,126 vulnerable skills, (2) a validated detection methodology achieving 86.7% precision and 82.5% recall, and (3) an open dataset and detection toolkit to support future research. These results demonstrate an urgent need for capability-based permission systems and mandatory security vetting before this attack vector is further exploited.

1 Introduction

Agent skills extend AI agents through dynamically loaded instructions and executable code, but their implicit trust and limited vetting create an uncharacterized security surface. This study addresses that gap with large-scale collection and automated analysis.

  • Security motivation: Skills can bundle executable programs that agents run with high trust, enabling data exfiltration, unauthorized commands, or harmful agent actions.The architecture combines Markdown instructions with scripts and supporting resources loaded on demand.
  • Research gap: The study targets open questions about vulnerability prevalence, categories, and risk differences across real-world agent skills.Prior LLM-security research focused on model behavior rather than implicitly trusted skill code and instructions.
  • Study scope: 42,447 skills were collected from two major marketplaces and 31,132 unique skills were analyzed after filtering and deduplication.The dataset spans eight functional categories and supports the paper’s first systematic characterization of agent skill vulnerabilities.
  • Study contributions: 86.7% precision and 82.5% recall were achieved by SkillScan against manually annotated ground truth.SkillScan integrates static analysis with LLM-based semantic classification.
  • Study contributions: 26.1% of skills contain at least one vulnerability across 14 patterns spanning prompt injection, data exfiltration, privilege escalation, and supply chain risks.The taxonomy and prevalence findings are grounded in the analyzed marketplace population.

2 Background

Agent skills package metadata, instructions, scripts, and resources into dynamically loaded capabilities that share security characteristics with other extensibility and package ecosystems. The threat model centers on malicious, compromised, and negligent skill authors exploiting broad permissions and a consent gap.

  • Agent skills and ecosystem: A skill combines a SKILL.md file containing YAML metadata and Markdown instructions with optional scripts and reference materials.Agents load skills dynamically according to task matching and execute bundled code with the agent’s permissions.
  • Agent skills and ecosystem: Agent skills share risk profiles with MCP and other extensibility ecosystems through code execution, dynamic loading, broad permissions, and LLM attack surfaces.Community registries aggregate third-party skills without mandatory security review.
  • Threat model: The threat model includes malicious authors, supply chain attackers, and negligent developers who introduce harmful behavior or insecure permissions.These adversaries may exploit natural-language instructions, compromised dependencies or repositories, and unsafe coding practices.
  • Threat model: The consent gap is the mismatch between users’ approval of an entire skill and the capabilities it actually exercises.Once approved, a skill may retain permissions to read and write files, download code, and open network connections without further prompts.
  • Threat model: The study focuses on skill-introduced vulnerabilities under a trusted-runtime assumption, excluding attacks on the underlying LLM, side channels, and physical access.Multi-tenant interactions, skill chaining, and dual-use tools are discussed as scope limitations.
  • Related work: Prior research covers extension security, LLM attacks, and package supply chains, but does not systematically characterize code-level vulnerabilities in agent skills.This paper combines large-scale empirical analysis with a detection framework to fill that gap.

3 Methodology

The study builds a systematic pipeline to collect, categorize, and detect vulnerabilities in agent skills, combining static analysis with LLM-based semantic classification. It validates the framework against manually annotated skills while documenting sampling, survivorship, and measurement limitations.

  • Data Collection: The pipeline crawled skills from two marketplaces, merged duplicates, and applied quality filters before analysis.Collection occurred in December 2025; filtering removed duplicates, short instruction files, non-English skills, and deleted repositories.
  • Skill Categorization: Skills were categorized through iterative manual coding of 1,218 stratified samples, followed by automated classification achieving 89.2% agreement with manual labels.The categorized sample intentionally oversampled rare vulnerability types, so its prevalence rates are not population estimates.
  • Vulnerability Definition: The framework defines vulnerability broadly as malicious code, negligent insecure code, or dangerous patterns that could enable harm.Severity is assigned per pattern using an a priori mapping based on attack potential.
  • Detection Framework: SkillScan combines static analysis and LLM-based analysis across prompt injection, data exfiltration, privilege escalation, and supply chain risks.Static patterns prioritize recall, while LLM semantic classification filters context-dependent false positives and catches semantic or obfuscated patterns.
  • Validation and Limitations: Aggregate validation performance was 86.7% precision, 82.5% recall, and 84.6% F1, while per-category comparisons remain uncertain because confidence intervals overlap substantially.Error analysis identified indirect exfiltration, natural-language obfuscation, and delayed execution as false-negative patterns; Security/Red-team skills also confound intent assessment.

4 Evaluation

The evaluation identifies a broad taxonomy of agent-skill vulnerabilities and finds that potentially dangerous patterns are common, though prevalence and intent require careful interpretation. Vulnerabilities cluster across categories and are associated with skill structure and category.

  • Vulnerability Categories: Data exfiltration patterns include external transmission, environment-variable harvesting, file-system enumeration, and context leakage.The reported counts are 89 skills for external transmission, 127 for environment-variable harvesting, 68 for file-system enumeration, and 28 for context leakage.
  • Vulnerability Categories: Privilege-escalation patterns include excessive permission requests, sudo or root execution, and credential access.These patterns were identified in 94, 41, and 52 skills, respectively.
  • Vulnerability Categories: Supply-chain risks include unpinned dependencies, external script fetching, and obfuscated code.The corresponding counts are 156, 67, and 55 skills.
  • Vulnerability Taxonomy: 14 distinct vulnerability patterns span prompt injection, data exfiltration, privilege escalation, and supply chain risks.The taxonomy includes four prompt-injection patterns, four data-exfiltration patterns, three privilege-escalation patterns, and three supply-chain patterns.
  • Overall Prevalence: 26.1% of 31,132 analyzed skills contain at least one potentially dangerous pattern, while 5.2% exhibit high-severity patterns suggesting malicious intent.The raw prevalence estimate adjusts to 26.5%, with an uncertainty range of 23–30%. Most flagged skills exhibit lower-severity patterns associated with insecure development practices.
  • Co-occurrence: 44.2% of vulnerable skills contain vulnerabilities in two or more categories, with Supply Chain → Data Exfiltration showing the strongest reported association at 81%.The co-occurrence matrix is asymmetric because category base rates differ; the authors report that the association may reflect behavioral coupling, while noting alternative explanations.
  • Category Differences: Security and Red-team skills show a 67.4% flagging rate, but adjustment for legitimate security-tool functionality reduces the estimated true vulnerability rate to approximately 21.4%.The category-level estimate is based on a categorized sample and should be interpreted as indicative rather than precise.

5 Discussion

The discussion places agent-skill security risks in the context of rapidly growing, lightly vetted ecosystems and outlines platform, developer, and user responses. It also emphasizes that prevalence estimates mix malicious, negligent, and ambiguous patterns and remain limited by dataset and classifier constraints.

  • Discussion: The agent-skills ecosystem resembles the early browser-extension landscape through implicit trust, limited vetting, and rapid growth outpacing security.The paper contrasts agent skills with browser extensions by noting full system access, semantic prompt-injection attacks, and exposure of conversation context.
  • Implications: Platforms should implement mandatory security reviews, permission sandboxing, and runtime monitoring.The paper also recommends that developers pin dependencies and request minimal permissions, while users prefer official sources and review permissions.
  • Limitations: 26.1% prevalence measures patterns warranting review rather than confirmed malicious skills, because intentional malice, negligence, and ambiguity are conflated.The framework reports 86.7% precision and 82.5% recall; legitimate tools can produce false positives and obfuscation can produce false negatives.
  • Limitations: The December 2025 snapshot may not represent enterprise deployments or the ecosystem’s future state.Full runtime analysis remains future work, and LLM classification shows 5.5% run-to-run and 9.0% prompt-variant variance.

6 Conclusion

The study combines static analysis and LLM-based classification to analyze agent-skill security at scale, finding widespread vulnerabilities and motivating stronger platform safeguards.

  • 26.1% of skills contain at least one vulnerability across 14 distinct patterns, with data exfiltration and privilege escalation most prevalent.Data exfiltration affects 13.3% of skills, while privilege escalation affects 11.8%.
  • Skills bundling executable scripts are 2.12× more likely to be vulnerable than instruction-only skills (OR=2.12, p<0.001).
  • The findings support capability-based permission manifests, mandatory pre-publication security scanning, and runtime sandboxing for agent skills.
  • SkillScan combines static analysis with LLM-based classification and achieved 86.7% precision and 82.5% recall against manually annotated ground truth.The validation used manually annotated ground truth comprising 200 skills.

Ethical Considerations

The study frames its benefits around informing defenses and user decisions while limiting research-related harm through controlled data practices, static analysis, disclosure, and dataset safeguards. Its author analysis also examines participation and vulnerability distribution across the ecosystem.

  • Risks and Benefits: The study aims to help platforms implement defenses, developers adopt secure practices, and users exercise appropriate caution.The authors argue benefits outweigh risks because documented patterns are already known to sophisticated attackers and defenders lack systematic knowledge.
  • Data Collection: The researchers analyzed publicly accessible marketplace data without circumventing access controls, violating terms of service, or collecting private data.Crawlers respected rate limits, used authenticated API access where available, and restricted stored data to the research team.
  • No Harmful Execution: The study used purely static analysis and did not execute potentially malicious skills, contact external services, test exploitability, or weaponize findings.
  • Responsible Disclosure: High-risk skills were disclosed to platforms before publication, and more than a hundred skills were subsequently removed.The released dataset redacts or anonymizes harmful snippets and withholds repository URLs for clearly malicious skills.
  • Author Analysis: Vulnerable skills are distributed broadly: 2,891 authors, or 63.8% of authors with scanned skills, published at least one detected-vulnerable skill.The author population includes 5,326 unique authors contributing 42,447 pre-filtered skill entries.

A.4 Publisher Concentration Analysis

Skill publishing is concentrated among a small group, but volume generally does not track vulnerability; instead, a few publishers and suspicious author profiles show elevated risk requiring closer scrutiny.

  • Publisher Concentration: The top 15 publishers contribute 3,523 skills, representing 11.3% of the analyzed dataset.
  • Publisher Concentration: Most prolific publishers have vulnerability rates near or below the ecosystem average of 26.1%, indicating that publishing volume does not necessarily correlate with poor security practices.
  • Publisher Concentration: Publisher J and Publisher O are outliers with vulnerability rates of 55.8% and 72.0%, respectively.Publisher J’s account was 68 days old at the time of analysis, raising concerns about purpose-built distribution accounts.
  • Suspicious-Author Scoring: The suspicion heuristic adds points for new accounts, minimal community presence, limited repository history, and high-risk skills, flagging authors scoring ≥5.High-risk skills include obfuscated code, confirmed exfiltration, or credential harvesting.
  • Suspicious-Author Scoring: 753 authors, or 14.1% of the author population, meet the suspicion threshold and contribute a disproportionate share of vulnerable skills.
  • Observed Patterns: High-scoring authors include both established accounts with security issues and newer accounts with concentrated risk.The latter combine accounts younger than six months, minimal community presence, and numerous vulnerable skills.

B Threat Model Scope Limitations

The threat model focuses on individual skill vulnerabilities and excludes important interaction and intent scenarios, limiting what automated detection can establish. Comparisons with browser extensions motivate stronger platform controls.

  • Threat Model Boundaries: The threat model does not address multi-tenant interactions in which skills share agent context or system resources.Such interactions could enable cross-skill data leakage even when individual skills appear benign.
  • Threat Model Boundaries: The framework flags credential access and privilege escalation patterns regardless of whether security skills use them legitimately for authorized testing.Distinguishing malicious from legitimate dual-use tools requires contextual information unavailable to the automated analysis.
  • Platform Comparison: The agent-skills ecosystem has structural parallels with the early browser-extension landscape that preceded widespread security incidents.
  • Platform Comparison: Chrome Manifest V3 provides a template by restricting remote code execution and limiting API access.The passage also notes that malicious extensions now target AI chatbot users, indicating converging threat landscapes.

D Detailed Validity Analysis

The study examines threats to validity arising from measurement, detection accuracy, sampling, generalizability, and incomplete runtime or tool validation. Additional pilot validation supports the static findings, but important scope and confirmation limits remain.

  • Construct Validity: The vulnerability definition combines malicious code, negligent practices, and dangerous patterns with ambiguous intent.Only 5.2% of skills exhibit high-severity patterns, illustrating the difference between inclusive and stricter prevalence estimates.
  • Internal Validity: 86.7% precision and 82.5% recall leave false positives, false negatives, and classifier variance as internal-validity concerns.False positives include legitimate credential access, while obfuscation and delayed execution can produce false negatives.
  • External Validity: The dataset is a December 2025 snapshot of public marketplace skills and may not generalize to proprietary, informal, or future ecosystems.The authors call for longitudinal studies as vetting and attacker evasion evolve.
  • Comparative Validation: SkillScan was not benchmarked against Semgrep, Bandit, or Snyk, limiting conclusions about incremental detection value.The paper notes that standard tools may detect code-level vulnerabilities but miss instruction-level threats.

F Comparison with Existing Static Analysis Tools

The paper compares SkillScan with three established static analysis tools on a balanced 100-skill subset. Existing tools remain effective for traditional code vulnerabilities, while SkillScan adds coverage for agent-skill-specific instruction threats.

  • Evaluation Setup: Three established tools—Bandit, Semgrep, and Snyk Code—were evaluated on a 100-skill subset containing 50 vulnerable and 50 benign skills.The comparison used ground-truth labels.
  • Results: Existing tools missed 32–42% of vulnerable skills despite achieving comparable or higher precision.The reported recall gap reflects substantially lower coverage than SkillScan.
  • Results: 0% prompt-injection recall was achieved by every existing tool, indicating that instruction-level threats fall outside their design scope.SkillScan’s advantage derives from agent-skill-specific patterns and LLM-based semantic analysis.
  • Results: Existing tools matched or exceeded SkillScan on code-level supply-chain patterns such as unpinned dependencies and unsafe deserialization.The findings distinguish traditional code-security coverage from agent-skill-specific coverage.
  • Interpretation: The authors recommend combining SkillScan with established SAST tools for defense in depth.The proposed combination addresses both instruction-level and traditional code vulnerabilities.

G Case Study Selection Methodology

The case studies were selected from the highest-risk skills using severity, classifier confidence, category diversity, real-world exposure, and pedagogical clarity. They illustrate possible threats rather than the typical vulnerable skill.

  • Selection Criteria: 87 skills met the highest-risk criteria: high severity, classifier confidence ≥0.8, and at least two vulnerability categories.These comprised 1.1% of 8,126 flagged skills and 0.3% of 31,132 total skills.
  • Selection Criteria: Case studies were chosen for pattern diversity, documented user exposure, and clear attack scenarios.Selected examples represented data exfiltration, prompt injection, and supply-chain categories.
  • Validation: The selected cases were manually reviewed and disclosed to platform maintainers as confirmed vulnerabilities.The authors characterize them as confirmed vulnerabilities rather than worst-case hypotheticals.
  • Selection Bias: The cases are not representative of typical vulnerable skills because selection favored highest risk and pedagogical clarity.Most flagged skills reportedly exhibit less dramatic patterns such as unpinned dependencies or overly broad permissions.
  • Illustrated Patterns: Instruction override matched 23 skills, while exfiltration commands matched 18 skills.Instruction override concerns ignoring user context or system behavior; exfiltration commands direct external transmission of context data.

H.2 Data Exfiltration Examples

The examples span data exfiltration, privilege escalation, supply-chain risks, and detection methodology. They show concrete patterns such as filesystem enumeration, credential access, excessive permissions, unpinned dependencies, and external script fetching.

  • Data Exfiltration: 68 skills contained filesystem-enumeration scripts that scan directories and collect sensitive file paths.Examples target locations including SSH, AWS, and cloud-configuration directories.
  • Privilege Escalation: 94 skills requested permissions beyond their stated functionality, including broad filesystem, network, and execution access.The synthesized example grants read/write access across the filesystem and permits multiple interpreters and package tools.
  • Privilege Escalation: 52 skills contained code accessing credential stores, SSH keys, authentication tokens, or password managers.Examples retrieve tokens and credential files from home-directory locations.
  • Supply Chain: 156 skills depended on packages without version pinning, creating version-drift and supply-chain exposure.The examples include unpinned Python packages and libraries without constraints.
  • Supply Chain: 67 skills downloaded and executed scripts from external URLs at runtime.Examples pipe remote installation scripts into sudo bash, including scripts hosted on GitHub or Google infrastructure.
  • Prevalence Estimation: The Rogan–Gladen correction estimated 26.5% true prevalence from 26.1% observed prevalence.The calculation uses 82.5% sensitivity and 94.2% specificity, with a 95% confidence interval of 23.1%–30.2%.
  • Detection Stability: Classification instability affected 22 unique skills, or 11%, after overlapping run-divergent and prompt-divergent cases were deduplicated.The paper reports 10–15% possible total variance under replication.

L.5 Detection Error Analysis

The error analysis identifies evasion patterns that can produce false negatives and legitimate behaviors that can produce false positives. It also documents sampling and classification caveats relevant to interpreting the reported metrics.

  • False Negatives: False negatives included dynamically constructed URLs, benign-sounding instructions, delayed execution, and code examples that were never executed.These cases defeated regex matching, semantic interpretation, or static analysis under specific conditions.
  • False Positives: Benign HTTP requests to services such as api.github.com and pypi.org triggered exfiltration patterns despite transmitting only non-sensitive metadata.This illustrates how technically matching a pattern can overstate practical risk.
  • False Positives: 50% of false positives came from legitimate Security/Red-team skills whose credential access or file enumeration was intentional and transparent.Excluding these skills increased precision from 86.7% to 90.6%.
  • Sampling and Metric Interpretation: 31.5% of validation skills were vulnerable versus 26.1% in the full dataset because flagged skills were intentionally oversampled.The validation set used stratified sampling across marketplaces, structural types, and size categories.
  • Sampling and Metric Interpretation: Inverse-probability reweighting changed precision from 86.7% to 84.5% and recall from 82.5% to 83.8%, indicating limited sampling bias in reported metrics.The study also excluded 1,203 uncertain candidate skills, representing 9.4% of candidates, from the 26.1% prevalence count.
  • Reproducibility: The released artifacts include the annotated dataset, SkillScan tools, collection pipeline, expert-labeled validation set, and analysis scripts.Repository URLs for confirmed malicious skills were withheld, with anonymized redacted versions provided instead.
Loading 2601.10338v1…