Source-linked AI summary
Context Matters: Repository-Aware Security Analysis of the Agent Skill Ecosystem
Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser, Johanna Ullrich
TL;DR
Existing marketplace scanners can label large shares of agent skills as malicious, but it is unclear how meaningful those classifications are without repository context. The paper analyzes 238,180 skills across marketplaces and GitHub, compares scanner signals, and reevaluates flagged skills using surrounding repository evidence. Only 0.52% of 2,887 flagged skill–repository combinations remain associated with suspicious repositories, while the study also identifies repository-hijacking risks and other ecosystem weaknesses.
Problem
Marketplace scanners report widely varying maliciousness rates, creating a need to determine whether classifications remain meaningful when skill and repository context are considered.
Method
The study collects and analyzes skills across three distribution platforms and GitHub, compares scanner outputs, and evaluates flagged skills using surrounding repository documentation, code, and metadata.
Results
0.52% of 2,887 scanner-flagged skill–repository combinations remain associated with suspicious repositories after repository-aware analysis.
Takeaways & Limitations
Scanner results should be treated as contextual risk signals rather than definitive labels, while skill-marketplace security also depends on provisioning, authentication, and archival practices.
Takeaways & Limitations
Repository-aware analysis does not establish definitive maliciousness ground truth, and manual validation is limited to a small sample of visible repository evidence.
Abstract
from arXiv · showhide
Agent skills extend local AI agents, such as Claude Code and OpenClaw, with additional functionality. Their growing popularity has led to dedicated marketplaces resembling mobile app stores, as well as automated scanners that assess whether skills are benign or malicious. However, scanner reports from individual marketplaces classify up to 46.8% of skills as malicious, raising concerns about false positives. We present the largest empirical security analysis of the AI agent skill ecosystem to date. We collect 238,180 unique skills from three major distribution platforms and GitHub, and analyze their contents, behavior, and repository context. Unlike existing scanner-based assessments, which evaluate skills largely in isolation, our repository-aware analysis checks whether a flagged skill is consistent with its surrounding GitHub project. This context substantially reduces the number of suspicious skills: only 0.52% remain suspicious after repository-aware analysis. Our results show that existing scanners can substantially overestimate maliciousness when repository context is ignored. At the same time, we identify previously undocumented real-world attack vectors, including the hijacking of skills hosted in abandoned GitHub repositories. Overall, our findings provide a more robust view of the agent-skill ecosystem's current risk surface and highlight the need for context-aware security evaluation.
1 Introduction
Agent skills extend autonomous agents but introduce supply-chain and behavioral risks, prompting marketplaces to scan skills with widely varying maliciousness rates. This paper studies whether repository context can distinguish genuine threats from false positives while identifying new attack vectors.
- Motivation: Marketplace-reported maliciousness varies from 6.0% to 46.8%, raising questions about whether isolated classifications remain meaningful.The reported rates are 46.8% for ClawHub, 23.0% for Skills.sh, and 6.0% for SkillsDirectory.
- Study scope: 238,180 unique skills from three distribution platforms and GitHub form the paper’s large-scale empirical study of the agent-skill ecosystem.The study examines skill contents, behavior, marketplace classifications, and repository context.
- Approach: The analysis characterizes skills, compares scanner classifications, and reevaluates flagged skills against their surrounding repository documentation and code.This three-part design covers ecosystem measurement, scanner consistency, and repository-aware interpretation.
- Main findings: Among 2,887 scanner-flagged skill–repository combinations, only 15 remain associated with suspicious repositories, corresponding to 0.52%.Repository context substantially reduces the number of likely false positives.
- Main findings: The study identifies repository hijacking risks affecting 121 skills linked to seven abandoned repositories, including one skill with more than 1,000 recorded installations.The attack vector arises when abandoned GitHub namespaces can be re-registered and skill indexes continue referencing them.
2 Background and Related Work
Agent skills are modular extensions distributed through marketplaces and external repositories, but their integration with autonomous execution creates security and supply-chain concerns. Existing marketplace scanners and prior studies report substantial, heterogeneous rates of suspicious skills and vulnerabilities.
- Agent skills: Skills are modular components that add capabilities such as API access, code execution, or data retrieval to autonomous agents.Agents invoke these capabilities within a reasoning-and-action loop.
- Skill marketplaces: ClawHub, Skills.sh, and SkillsDirectory differ in hosting, curation, Git-based distribution, moderation, and security scanning practices.These differences affect how skills are controlled and referenced across platforms.
- Marketplace security: Marketplace scanners classify skills as benign or malicious and expose short explanations, but reported maliciousness can be high enough to suggest substantial false positives.The paper notes a high rate of 46.8% in one marketplace and explicitly frames false-positive concerns.
- Prior empirical studies: Prior analyses report both critical flaws, including malware distribution and prompt injection, and more minor issues such as hard-coded API keys and insecure credential handling.These findings come from studies covering thousands of marketplace skills.
- Agent security: Agent ecosystems also face direct agent vulnerabilities, including localhost control, log manipulation, and prompt-injection-based disclosure of private keys.These examples show that risks extend beyond the skill package itself.
3 Methodology
The study uses a three-stage pipeline to collect and normalize skills, compare scanner assessments, and evaluate flagged skills with repository context. It treats scanner outputs as security signals rather than definitive maliciousness labels.
- Pipeline overview: The three-stage pipeline covers skill collection, malicious classification, and repository-context analysis.The methodology is designed to measure ecosystem breadth, compare tools, and interpret scanner alerts contextually.
- Skill collection: Skills are collected from marketplaces and GitHub, normalized into a common representation, and deduplicated using content hashes.GitHub repositories are searched for SKILL.md files and constrained during collection for scalability.
- Skill analysis: Static analysis records files, directory structure, executable scripts, and embedded secrets, including validation of detected credentials where possible.These measurements characterize skill structure, content, and exposure risks.
- Scanner analysis: Scanner reports, an open-source scanner, and an LLM-based behavioral pipeline are compared for classification frequency and detection overlap.The behavioral analysis extracts indicators such as system interaction, network communication, credential handling, persistence, and abuse potential.
- Repository-aware analysis: Repository-aware analysis evaluates codebase alignment and repository maturity, combining them into a contextual trust and triage score.High scores indicate repository evidence that flagged behavior may be expected or benign, while low scores indicate weak contextual support.
- Validation: The study validates repository-aware judgments through manual inspection of flagged repositories by independent reviewers.The validation checks whether repositories appear benign or suspicious and whether context scores align with human judgments.
4 RQ1: Cross-platform Skill Analysis
The cross-platform analysis shows a rapidly changing, overlapping ecosystem in which GitHub commonly hosts skills while marketplaces act as discovery layers. It also reveals exposed credentials, repository-hijacking opportunities, and platform-level privacy weaknesses.
- Collected dataset: The study analyzes skills collected from ClawHub, Skills.sh, SkillDirectory, and GitHub, including marketplace-listed and repository-extracted artifacts.For Skills.sh, 77,456 additional skill folders from referenced repositories produced 125,928 analyzed skills in total.
- Dataset limitations: Retrieval was constrained by omitted subdirectories, invalid skill paths, missing repository contents, private repositories, and authentication requirements.These issues prevented some indexed skills from being retrieved and complicated reliable archival.
- Cross-platform coverage: 16,755 ClawHub skills, 125,928 Skills.sh skills, 17,611 SkillDirectory skills, and 136,095 GitHub skills were successfully downloaded and analyzed.The platforms overlap substantially, with GitHub serving as the primary hosting platform and marketplaces acting as discovery layers.
- Skill content: Python scripts are most common, followed by shell, JavaScript, and TypeScript, while script prevalence differs across marketplaces.Skills.sh, SkillsDirectory, and GitHub contain scripts in roughly 11.8% to 15.7% of skills, whereas ClawHub is an outlier.
- Secrets: 12 functional credentials were discovered, including NVIDIA, ElevenLabs, Gemini, MongoDB, Amplify, Postgres, and xAI credentials.Attackers could abuse these credentials to access third-party services and act on behalf of credential owners.
- Hijacking risk: 121 skills forward to seven vulnerable repositories that could be recreated under associated GitHub usernames, creating repository-hijacking exposure.The affected skills span Skills.sh and SkillsDirectory, and vulnerable repositories include one with 159 stars.
- Platform weaknesses: The analysis also found that a ClawHub API endpoint exposed GitHub-account email addresses not shown on profiles or the website.This metadata was returned during crawler development and was not expected from the public-facing services.
5 RQ2: Malicious Classification
Scanner assessments vary substantially across marketplaces and tools, with low overlap between detections. These inconsistencies indicate that isolated scanning can classify many skills as suspicious without consistent cross-scanner support.
- Malicious Classification Rates: On ClawHub, OpenClaw flags 41.93% of skills, VirusTotal 36.20%, GPT-5.3 38.8%, and Cisco 16.74%.The comparison illustrates how perceived ecosystem security depends on the scanning approach.
- Malicious Classification Rates: Scanner outputs differ substantially: ClawHub fail rates range from 16.7% to 41.9%, while Skills.sh rates range from 3.79% to 13.76%.On Skills.sh, GPT-5.3 classifies 27.28% as suspicious and Cisco flags 14.04%.
- Cross-Scanner Agreement: Only 33% of Cisco detections overlap with GPT-5.3 detections, while just 10.2% of GPT-5.3 detections overlap with Cisco.The asymmetric overlap demonstrates that scanners frequently identify different suspicious sets.
- Cross-Scanner Agreement: Among 8,402 Skills.sh skills flagged by at least one scanner, 6,032 were flagged by one scanner, compared with only 33 flagged by all five.A further 1,629 were flagged by two scanners, 540 by three, and 168 by four.
- Repository-level Classification Rates: Strict repository aggregation labels 50.76% of repositories with more than 1,000 stars as malicious and 61.40% of repositories associated with highly installed skills.The probability of at least one flagged skill increases as repositories contain more skills.
6 RQ3: Repository-Aware Analysis
Repository-aware analysis combines repository metadata with codebase alignment to reassess scanner-flagged skills. Most evaluated repositories show moderate or strong linkage, while only 0.52% of combinations remain suspicious after contextual analysis.
- Evaluation Scope: The study evaluates 3,000 sampled skill–repository combinations after excluding unavailable repository contexts; cloning failures leave 2,887 combinations for codebase evaluation.ClawHub skills and skills located at repository roots are excluded from the repository-aware analysis.
- Metadata Score: Flagged-skill repositories are often small and inactive in popularity metrics, but 47.4% were updated within the last week and their metadata does not differ distinctly from matched repositories.For example, 47.6% are smaller than 2 MB, 43.4% have no stars, and 66.5% have no forks.
- Codebase Score: Repository context is usually available: 94.1% of repositories contain a README, 65.7% contain code, and 61.9% contain both.Domain alignment is high or medium for roughly 72% of skills, although direct code similarity is lower.
- Repository Context Score: The repository context score weights codebase evidence at 70% and metadata at 30%; its mean is 58.5, with codebase and metadata means of 65.1 and 42.9.The weighting emphasizes alignment between the skill and surrounding repository contents.
- Repository Context Score: Only 121 cases (4.2%) have a repository context score below 40, while 1,373 (47.6%) fall between 40 and below 60 and 1,393 (48.3%) score at least 60.Most scanner-flagged skills therefore show moderate to strong repository linkage.
- Suspicious Repositories: Only 15 skill–repository combinations remain suspicious after repository-aware analysis, representing 0.52% of the 2,887 evaluated combinations.These repositories align with the skill codebase but appear suspicious themselves and are not categorized as security tools.
- Validation: Manual review found all 18 available repositories in a 20-repository sample benign, supporting the interpretation that many scanner alerts disappear under repository context.Two sampled repositories were unavailable for review.
- Validation: Metadata scores correlate positively with human-rated repository maturity, but the metadata component is not a definitive maliciousness label.Reviewer maturity judgments were more heterogeneous than the averaged scores suggest.
7 Discussion
The discussion shows that repository context changes how scanner alerts should be interpreted, while marketplace architecture introduces familiar supply-chain risks. Repository-aware analysis is practical but remains limited by incomplete evidence and the possibility of deceptive repositories.
- Skill Marketplaces: Marketplaces largely serve as discovery layers over GitHub, inheriting risks from broken, renamed, deleted, or private repositories.Referenced skills can also be hijacked when abandoned GitHub namespaces are re-registered.
- Skill Scanners: Scanner outputs vary substantially across tools, with low cross-scanner agreement and most flagged skills detected by only one tool.Aggregating alerts at repository level can further overstate malicious-repository prevalence, including for popular projects.
- Repository Context: Only 15 skill–repository combinations, or 0.52% of the evaluated sample, remained suspicious after repository-aware analysis.Many isolated alerts occurred in repositories whose documentation, code, and stated purpose aligned with the skill functionality.
- New Ecosystem, Familiar Risks: Agent-skill ecosystems reproduce familiar supply-chain risks involving authentication, dependency ownership, abandoned projects, embedded secrets, and repository hijacking.The study also found functional credentials in published skills and marketplace designs that can enable repository hijacking.
- Practicality of Repository-Aware Scanning: Repository-context analysis cost approximately $0.0097 per skill–repository pair without caching and $0.0021 with prompt caching.A 3,000-skill sample cost approximately $24 including retries, supporting periodic marketplace scans.
- Limitations: Repository-aware analysis does not establish definitive maliciousness because attackers may make repositories appear benign while hiding malicious behavior elsewhere.Manual validation covered a small sample and visible evidence, while unavailable repositories, failed clones, and dynamic references limit reproducibility.
8 Conclusion
The study combines large-scale measurement with repository-aware reassessment to characterize scanner inconsistency and marketplace weaknesses. It finds that repository context sharply narrows suspicious cases while exposing concrete hijacking and metadata risks.
- Conclusion: 238,180 unique skills from three marketplaces and GitHub formed the largest empirical security analysis of the agent-skill ecosystem to date.The dataset supported analysis of skill contents, scanner behavior, repository context, and marketplace structure.
- Conclusion: Malicious classification rates ranged from 3.8% to 41.9%, while only 0.12% of commonly analyzed skills were flagged by all five tested scanners.These results demonstrate substantial inconsistency among existing scanners.
- Conclusion: After re-evaluating 2,887 scanner-flagged skill–repository combinations, only 0.52% remained associated with suspicious repositories.Most flagged skills were embedded in repositories whose documentation and code matched the skill functionality; 4.2% showed weak repository linkage.
- Conclusion: The study identified repository-hijacking risks affecting 121 skills and found that ClawHub exposed sensitive developer metadata, including GitHub-associated email addresses.These findings expose structural weaknesses in skill-marketplace design and operation.