Source-linked AI summary
How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study
Zhihao Chen, Ying Zhang, Yi Liu, Gelei Deng, Yuekang Li, Yanjun Zhang, Jianting Ning, Leo Yu Zhang, Lei Ma, Zhiqiang Li
TL;DR
Credential leakage in LLM agent skills is poorly understood despite skills operating with access to sensitive credentials. The paper analyzes a stratified sample of SkillsMP skills using static extraction, sandbox testing, and intent-behavior comparison. It identifies 520 affected skills with 1,708 issues, showing that leakage is often cross-modal, debug logging is a dominant vector, and exposed credentials are frequently exploitable.
Problem
The study addresses limited evidence about how third-party agent skills leak credentials while operating in privileged execution environments.
Method
The authors analyze 17,022 sampled SkillsMP skills through static extraction, dynamic sandbox validation with mock credentials, and manual comparison of developer intent with runtime behavior.
Results
520 affected skills contain 1,708 security issues, with 76.3% requiring joint analysis of natural-language descriptions and programming logic.
Takeaways & Limitations
Credential leakage in agent skills is fundamentally cross-modal, and agent execution semantics make routine debug output an exploitable exposure channel.
Takeaways & Limitations
AST-based credential-flow analysis covers only Python and JavaScript at the intra-procedural level, while other languages rely on keyword matching.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents increasingly rely on third-party skills that operate within privileged execution environments and routinely handle sensitive credentials, yet how these credentials are leaked remains largely unexplored. To fill this gap, we present the first large-scale empirical study on credential leakage in agent skills. From 170,226 artifacts on SkillsMP, the largest open-source skill marketplace, we sampled 17,022 skills via stratified random sampling and analyzed each through static secret extraction (regex and AST parsing), dynamic sandbox testing with mock credentials, and cross-referencing developer intent against runtime behavior. Our analysis identifies 520 affected skills containing 1,708 security issues, and yields a taxonomy of 10 leakage patterns. Three findings stand out. First, 76.3% of cases require jointly analyzing natural-language descriptions and programming logic, showing that credential exposure in skills is fundamentally cross-modal. Second, debug logging accounts for 73.5% of vulnerabilities because agent frameworks feed stdout into the LLM context window, turning routine debugging into a credential exposure vector. Third, 89.6% of leaked credentials are immediately exploitable -- 92.5% during routine execution without elevated privileges -- and the fork-based distribution model defeats remediation, as secrets removed from 107 upstream repositories persist across 50+ independent forks. Following responsible disclosure, all malicious skills have been removed and 91.6% of hardcoded cases remediated. We release our dataset, taxonomy, and detection pipeline to support future agent security research.
1 Introduction
This study examines credential leakage in rapidly expanding agent skills, focusing on prevalence, leakage patterns, and practical exploitability. It finds that exposure is predominantly cross-modal, debug logging is a major vector, and leaked credentials can remain exploitable after remediation.
- Research findings: 76.3% of leakage cases require jointly analyzing natural-language descriptions and programming logic, while 3.1% exploit natural language alone through prompt injection.Neither modality alone reveals these exposures.
- Research findings: 73.5% of vulnerability issues arise from print or console.log exposure because agent frameworks feed stdout into the LLM context window.This makes logged credentials retrievable through natural-language queries.
- Research findings: Among 83 malicious skills, 37.3% combine multiple attack patterns, commonly pairing defense evasion with remote exploitation.The skill architecture can present benign interfaces while executing multi-objective payloads.
- Research findings: 89.6% of affected skills are exploitable, with 92.5% triggered during routine execution without elevated privileges.Credentials removed from 107 upstream repositories also persist across 50+ independent forks.
- Contributions: The study constructs a dataset of 17,022 skills, proposes 10 leakage patterns, and identifies 1,708 previously unknown security issues across 520 affected skills.Following disclosure, all 83 malicious skills were removed and 91.6% of hardcoded credential cases were remediated.
2 Background
Agent skills combine natural-language workflows with executable programs, allowing LLM agents to invoke external tools and services. Because both modalities can handle credentials, leakage may emerge only from their interaction and includes undeclared exposure rather than legitimate credential use.
- Agent skill architecture: A skill bundles natural-language workflow specifications with Python, Shell, or JavaScript scripts that perform concrete operations at runtime.The framework injects the workflow into the LLM context and uses the model to invoke the scripts.
- Cross-modal risk: NL text enters the LLM reasoning context while PL scripts inherit local credential stores, so either modality can surface credentials that neither reveals alone.This interaction distinguishes agent skills from traditional single-modality packages.
- Operational definition: Credential leakage is exposure of authentication material to any recipient or channel neither declared by nor required for the skill’s stated functionality.The definition includes accidental exposure and deliberate exfiltration but excludes legitimate, declared credential use.
3 Methodology
The study uses a four-phase pipeline combining dataset collection, static analysis of natural-language and code artifacts, dynamic sandbox validation, and manual intent-based classification. This process narrows a representative sample to confirmed vulnerable and malicious cases while testing both ordinary and adversarial execution.
- Dataset collection: The study collects a complete SkillsMP snapshot of 170,226 active and historical skills and randomly samples 17,022 skills for analysis.The sample represents 10% of the population and exceeds the stated requirement for 99% confidence and a 1% margin of error.
- Static filtering: Static filtering combines credential-taxonomy keyword matching, semantic analysis of natural-language descriptions, and AST-based analysis of source code.The code analysis uses tree-sitter for Python and JavaScript and examines potentially unsafe sinks such as network transmission, logging, and file I/O.
- Static filtering: 3,156 unique candidate skills remain after merging 309 natural-language candidates with 2,958 source-code candidates.The merged set includes 198 candidates flagged only by natural language, 2,847 only by source code, and 111 by both streams.
- Dynamic validation: Dynamic validation executes candidates in instrumented Docker sandboxes with mock credentials provisioned through deployment-like channels.Testing uses benign prompts derived from advertised workflows and adversarial payloads embedded in external content consumed during ordinary workflows.
- Manual classification: The pipeline classifies 1,427 dynamically flagged skills through independent review of declared intent and benign/adversarial execution profiles.Reviewers distinguish Benign, Vulnerable, and Malicious outcomes, while direct operator prompts forcing exfiltration are not used.
4 Major Findings
Credential leakage spans both accidental vulnerabilities and deliberate attacks, with exposure emerging from the interaction between natural-language instructions and executable code. Debug logging dominates vulnerable patterns, while malicious skills frequently chain evasion and exploitation techniques.
- Web Scraping, Data Processing, and API Integration account for 44.8% of confirmed leakage cases.Their authentication requirements create frequent opportunities for credential handling; File Management and Automation contribute additional cases.
- Python accounts for 60.0% of leakage cases, compared with 27.5% for JavaScript/Node.js and 7.9% for TypeScript.The paper links this distribution to Python’s prevalence in agent development and leakage-prone practices such as unvalidated environment-variable access.
- 84.0% of affected skills contain unintentional vulnerabilities, while 16.0% exhibit deliberate malicious intent.Across 17,022 sampled skills, 520 contained leakage involving 1,708 security issues.
- 76.3% of leakage cases require jointly analyzing natural-language descriptions and programming logic.The skill architecture allows benign advertised interfaces to coexist with harmful executable behavior, including credential exfiltration.
- Information Exposure accounts for 73.5% of all vulnerability issues, as agent frameworks surface stdout and stderr in the LLM context window.One confirmed example exposed OAuth tokens through console logging and allowed extraction through a natural-language query.
- 37.3% of malicious skills combine multiple attack patterns, commonly pairing defense evasion with remote exploitation.Base64 obfuscation or semantic concealment can precede reverse shells, remote scripts, or RCE backdoors, while some payloads harvest credentials with elevated privileges.
5 Discussion
The findings expose architectural gaps in LLM agent ecosystems because agent skills execute within tightly coupled environments where outputs and credentials cross local execution boundaries. The paper therefore calls for isolation and least-privilege changes by framework designers and skill developers.
- For Agent Framework Designers: Agent frameworks couple natural-language reasoning and code execution, allowing attackers to disguise exfiltration as benign role-play.The execution model differs from traditional software with OS-level permission boundaries.
- For Agent Framework Designers: 80.5% of vulnerable skills involve Information Exposure because frameworks capture stdout and inject it into the LLM context window.The authors recommend credential-pattern extraction before stdout enters conversational memory.
- For Agent Framework Designers: Framework designers should separate the LLM reasoning engine from skill execution through capability-based isolation of memory and network access.This recommendation targets the tightly coupled execution environment described in the discussion.
- For Agent Skills Developers: Agent skill developers must treat stdout, process arguments, and configuration outputs as leakage surfaces visible beyond local execution.The proposed threat model differs from traditional software because skills run in an LLM-mediated runtime.
- For Agent Skills Developers: Developers should minimize credential exposure architecturally through least privilege rather than relying solely on post-deployment fixes.The passage frames least privilege as a fundamental shift in secure skill development.
6 Related Work
Prior work addresses secret leakage, software supply chains, and agent threats, but existing approaches do not capture credential exposure created by skills’ combined natural-language and programming architecture. This study addresses that gap by examining cross-modal leakage patterns in agent skills.
- Secret Leakage Research: Existing secret-leakage detectors primarily target monolingual codebases where credentials appear as detectable string literals.Prior studies used regex, neural pattern matching, or related analyses across conventional software artifacts.
- Secret Leakage Research: Agent skills introduce additional leakage channels through natural-language instructions, stdout ingestion, and environment variables crossing the NL–PL boundary.These channels are outside the assumptions of existing detectors.
- Software Supply Chain Security: Traditional supply-chain studies focus on malicious code injected through dependency resolution in single-language registries.Agent skill registries present a different setting because skills combine natural language with executable behavior.
- Agent Security Research: Prior agent-security research covers prompt injection, autonomy, lifecycle threats, application vulnerabilities, attack techniques, and skill lifecycles.The cited work spans both threat taxonomies and defensive approaches.
- Research Gap: None of the reviewed work examines how NL+PL skill architecture inherently facilitates credential leakage.The paper positions its empirical study as addressing this specific gap.
7 Threats to Validity
The study’s validity is constrained by subjective intent labeling, incomplete static and dynamic analysis, and data drawn from a single platform. These limitations bound confidence in taxonomy coverage, path coverage, and cross-platform prevalence.
- Construct Validity: Intent classification is subjective, although three security-experienced authors independently labeled flagged skills and achieved mean Cohen’s κ=0.88 after discussion.The taxonomy may also omit patterns outside its CWE/CAPEC anchors.
- Internal Validity: The AST-based credential-flow analysis covers only Python and JavaScript intra-procedurally, while other languages fall back to keyword matching.Cross-variable propagation is not tracked.
- Internal Validity: LLM non-determinism and dormant logic may prevent sandbox testing from reaching every conditional path.Three rounds of dual-condition differential testing reduce but do not eliminate this gap.
- External Validity: All data come from SkillsMP, so prevalence rates on other skill platforms may differ.This is the study’s stated external-validity boundary.
8 Conclusion
This paper presents a large-scale empirical study of credential leakage in agent skills, identifying substantial exposure across a sampled SkillsMP corpus. Its findings show that leakage is cross-modal, often surfaced through stdout, and frequently exploitable during routine execution.
- Study Scope and Prevalence: 17,022 skills yielded 1,708 security issues across 520 affected skills, representing 3.1% of the analyzed corpus.Developer negligence accounts for 84.0% of all cases.
- Key Findings: 73.5% of vulnerability patterns involve debug logging because frameworks capture stdout into the LLM context window.Logged credentials thereby become queryable through natural language.
- Key Findings: 76.3% of cases require joint analysis of natural-language descriptions and executable code.This cross-modal property is not addressed by existing static analyzers.
- Key Findings: 89.6% of affected skills are exploitable during routine execution without elevated privileges.The fork-based distribution model also allows leaked credentials to persist across repositories after upstream remediation.
- Remediation and Open Problems: Responsible disclosure led to removal of 83 malicious skills and remediation of 91.6% of identified hardcoded credential cases.The conclusion identifies stdout-to-context redaction and fork-aware remediation as open problems.