PaperlayerView exact revision on arXiv

Source-linked AI summary

ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree

Vincent Koc, Patrick Erichsen, Jacob Tomlinson, Agustin Rivera, Michael Appel, Nir Paz

arXiv:2606.01494v1cs.CRcs.AIcs.SE

TL;DR

Existing studies measure skill prevalence but do not establish whether independent detectors agree on the same skills. This paper analyzes three scanner families across 67,453 skills and finds minimal, attack-surface-structured agreement, supporting layered security decisions.

  • Problem

    Prior studies measure vulnerable or malicious skills but do not establish whether independent detectors agree on which skills warrant trust decisions.

  • Method

    The paper analyzes disagreement among three scanners operating across artifact and tool/MCP layers within ClawScan’s registry verification pipeline.

  • Results

    Only 0.69% of skills receive all-three flags and 81.9% of flags come from one scanner, with scanner dominance reversing across suspicious and malicious verdicts.

  • Takeaways & Limitations

    Skill security is a systems problem requiring complementary, attack-surface-specific components rather than a single allow/block mechanism.

  • Takeaways & Limitations

    SkillSpector advisories and ClawScan verdicts may share LLM mechanisms, so their correlation may reflect circularity rather than independent confirmation.

Abstract

from arXiv · show

Agent skills extend AI agents with reusable instructions, tools, scripts, references, and workflows, establishing a security boundary distinct from both model safety and traditional package-malware detection. ClawHub Security Signals is a sanitized dataset of 67,453 latest public OpenClaw skill versions. Each row pairs redacted SKILL.md content and sanitized bundled files where present with a final ClawScan registry verdict and evidence from three scanner families: VirusTotal, static heuristic analysis, and NVIDIA SkillSpector. Rather than estimating malicious-skill prevalence, we study scanner disagreement. The three scanners rarely flag the same skills: any pair overlaps on at most 10.4% of their combined positives, only 0.69% of skills are flagged by all three, and 81.9% of flagged skills are identified by a single scanner. The disagreement is structured by attack surface. SkillSpector, which raises semantic agentic-risk advisories rather than malware-reputation signals, is positive for 19,209 of 25,504 suspicious rows (75.3%) but only 14 of 206 malicious rows (6.8%). The malicious-verdict region shows the inverse profile: 150 of 206 malicious rows (72.8%) are VirusTotal-positive, consistent with bundled-code malware evidence. These results show that agent-skill security requires layered governance, not single-scanner allow/block decisions. The corpus is released as a sanitized silver-standard dataset: labels are the registry's automated verdicts, not human-annotated ground truth, and the release represents an early, versioned snapshot intended to support the community while a human-annotated subset is developed. Further research is encouraged, including models tailored for skill-security triage.

1 Introduction

This section introduces ClawHub Security Signals as a sanitized, versioned silver-standard corpus for studying scanner disagreement across agent-skill security signals. It shows that disagreement is substantial and structured by attack surface, motivating layered analysis and future human adjudication.

  • Framing: Agent skills are portable execution-layer instruction sets that govern how AI agents use tools, making them distinct security objects.They can encode procedural knowledge, constraints, dependencies, and executable helper code.
  • Dataset and epistemics: 67,453 latest public skill versions are released with analyzed bundle content, a final ClawScan verdict, and evidence from three scanner families.The scanners are VirusTotal malware reputation, static analysis, and NVIDIA SkillSpector semantic agentic-risk analysis.
  • Dataset and epistemics: The corpus is a silver-standard trust dataset, not ground truth or a malware corpus, because its labels are the registry’s automated decisions.Each scanner is treated as a weak-supervision source rather than definitive adjudication.
  • Scanner disagreement: 0.69% of skills are flagged by all three scanners, while 81.9% of flags originate from a single scanner without corroboration.Any two scanners agree on fewer than one in ten of their combined flags.
  • Structured disagreement: 75.3% of suspicious-verdict skills are SkillSpector-positive, whereas 72.8% of malicious-verdict skills are VirusTotal-positive and only 6.8% are SkillSpector-positive.The section presents semantic agentic-risk and bundled-code malware evidence as distinct signals tracking different inspected surfaces.
  • Limitations and next steps: The versioned release uses automated silver labels and plans human annotation focused on disputed cases to support improved skill-security tooling.The early release is intended to let the community examine disagreement directly and develop models for skill-security triage.

2 Background and Threat Model

Agent skills become part of an agent’s effective program and can expose risks across artifact, tool/MCP, and runtime layers. The scanners observe different layers—without fully observing runtime—so their disagreement reflects different security perspectives on the same skill.

  • Skill Model: A skill bundles SKILL.md instructions and triggers with optional scripts, references, and capability declarations, becoming part of the agent’s effective program at installation.At runtime, the agent may load the full skill when its description appears useful for the task.
  • Threat Layers: Agent-skill risk spans artifact, tool/MCP, and runtime layers, which should not be conflated.The artifact layer includes instructions, bundled scripts, shell construction, secrets, install sources, and behavior-purpose mismatches; the tool/MCP layer concerns delegated authority and reachable external systems.
  • Scanner Scope: VirusTotal and static analysis inspect bundled artifacts, while SkillSpector reasons about instructions and declared capabilities in the tool/MCP layer.None of the three scanners fully observes the runtime layer, making their outputs samples of different layers of the same object.
  • Threat Model: Tool-enabled agents can autonomously exploit websites and vulnerabilities, while prompt injection can manipulate tool invocation and data movement through embedded or retrieved instructions.The supplied passage states that these risks have also appeared in ClawHub itself, citing the ClawHavoc report.

3 Related Work

Prior work spans skill-security datasets and detectors, MCP and agent attacks, extension-marketplace trust failures, package-ecosystem malware, and security-tool disagreement. This study is complementary: it combines a deployed registry verdict with independent scanner signals and releases the resulting disagreement for direct observation.

  • Skill-security datasets and detectors: Skill-security studies establish large-scale vulnerability analysis, behavioral confirmation of malicious samples, multi-agent auditing, and threat taxonomies with configuration-injection cases.These works primarily support prevalence estimation and detection-method development, whereas this study examines disagreement across detectors.
  • MCP and agent security: MCP and agent-security research documents tool poisoning, tool-description injection, rug-pull updates, client-privilege exploits, indirect prompt injection, and broader agent attacks and defenses.These literatures motivate security analysis for skills that route agents toward tools.
  • Extension marketplaces: Extension-marketplace studies report weak vetting, arbitrary names, post-approval backend changes, incomplete permission disclosure, skill-squatting, and third-party platform, privacy, and safety concerns.The findings show how marketplace trust infrastructure can lag behind ecosystem growth.
  • Package ecosystems: Package-ecosystem research covers malicious publication, dependency confusion, typo-squatting, install-time execution, maintainer compromise, multi-behavior malware, and persistence through mirrors.These attack patterns provide adjacent context for bundled-code and supply-chain risks.
  • Security-tool disagreement: Security-tool research emphasizes false positives, developer trust, warning suppression, and aggregating multiple noisy detectors as weak-supervision sources rather than trusting any single tool.This study extends the established “tool vs. user” problem to “tool vs. tool.”
  • Corpus positioning: The corpus’s distinctive contribution is pairing a deployed-registry moderation verdict with multiple independent scanner signals and releasing the data publicly so disagreement is directly observable.Two concurrent agent-skill studies exceed the corpus in raw scale, but not this combination of registry verdicts and scanner signals.

4 The ClawScan Verification Pipeline

The ClawScan pipeline verifies skills before catalog publication by combining three scanner outputs with provenance, metadata, and moderation context into one registry verdict and Skill Card. Its scanner positives are evidence to weigh, not confirmed findings, because the scanner families inspect different signals and SkillSpector advisories are not install-blocking verdicts.

  • Verification pipeline: Before publication, skills pass a pre-catalog Scan → Evaluate → Skill Card → Sign (proposed) verification gate.Skills may enter ClawHub as linked source artifacts or uploaded bundles through the publisher UI.
  • Verification pipeline: At Evaluate, ClawScan combines three scanner outputs with provenance, metadata, and moderation context to emit a single registry verdict and Skill Card.
  • Scanner roles: The scanner families inspect different signals: static analysis examines code and text patterns, while VirusTotal contributes traditional malware and reputation evidence.Static findings include dangerous execution, credential access, exposed secret literals, dynamic code execution, and untrusted install sources.
  • Scanner roles: SkillSpector findings are advisory risk signals, not accusations or install-blocking verdicts by themselves.An issue can indicate a meaningful blast radius rather than abuse.
  • Positive definition: A scanner is positive when its status is suspicious or malicious; clean, stale, error, and missing statuses are non-positive.The conservative definition applies to overlap and agreement statistics, and “positive” denotes evidence to weigh rather than a confirmed finding.

5 Dataset Construction

The dataset is a versioned snapshot of latest public OpenClaw skills, pairing sanitized skill content and bundled files with automated ClawScan verdicts and scanner signals. Its labels are operational silver labels rather than human-adjudicated ground truth.

  • Snapshot and splits: 31 May 2026: the snapshot contains 67,453 latest public skill rows with ClawScan verdicts, deterministically split into training, validation, and test sets.The source includes 187,423 public source-artifact rows and 67,478 normalized latest public skill artifacts.
  • Released content: 19.65% of rows include at least one exported bundle file, including 10.06% with code files, totaling 58,516 files and 278.9 MB of sanitized content.The release includes redacted SKILL.md content and sanitized bundle-file content where present, rather than SKILL.md text alone.
  • Label provenance: ClawScan assigns clean, suspicious, or malicious verdicts through automated registry review, with GPT-5.5 high used for 99.6% of rows.Confidence is high for 87.1% of rows, medium for 12.5%, and low for 0.4%; the verdicts are treated as silver labels, not human-adjudicated ground truth.
  • Scanner coverage: 97.3% of rows have resolved VirusTotal statuses, including 7.75% positive rows, while SkillSpector resolves statuses for 98.2% and reports 48.71% advisory-positive rows.The scanner families have sharply different resolved-status distributions, reflecting different observed security surfaces.

6 Scanner Disagreement

Scanner positives cover 35,600 of 67,453 rows but overlap weakly, with most flagged rows identified by one scanner. Disagreement is structured by verdict region: SkillSpector dominates suspicious cases, while VirusTotal dominates malicious ones, supporting layered triage rather than single-scanner authority.

  • Cross-scanner overlap: 35,600 of 67,453 rows (52.8%) carry at least one positive scanner signal, yet 29,153 positive rows (81.9%) are positive on exactly one scanner.Only 468 positive rows (1.31% of positive rows; 0.69% of all rows) are positive on all three.
  • Cross-scanner overlap: No scanner pair exceeds 0.104 Jaccard agreement, while Cohen’s κ remains slight at 0.045–0.082.These agreement values treat stale, error, and missing statuses as non-positive.
  • Verdict-structured disagreement: SkillSpector raises advisories for 75.3% of suspicious skills and is the only positive scanner for 56.3%, whereas VirusTotal flags 72.8% of malicious skills.SkillSpector advisories are semantic agentic-risk signals rather than maliciousness labels; disclosure and authority risk are determined by its LLM stage.
  • Malicious-verdict blind spots: 192 of 206 malicious-verdict rows carry no SkillSpector issues, and 24.3% of malicious verdicts have no positive scanner signal of any kind.The paragraph attributes this silence to bundled-code or provenance evidence, while ClawScan reached malicious verdicts despite missing scanner positives.
  • Triage implications: Mean SkillSpector score rises from 22.1 for clean skills to 59.3 for suspicious skills, separating those two classes for learned triage.Mean issue count likewise rises from 1.9 to 6.5, and median redacted SKILL.md length from 3,955 to 5,562 characters.

7 Verdict Structure and Risk Categories

Verdicts are deliberately non-binary: most skills are clean, suspicious skills require review before trust, and malicious skills are rare. Risk categories emphasize authority, scope, tool semantics, disclosure, and sharper static indicators such as dangerous execution and credential access.

  • Verdict distribution: 61.9% of skills are clean, 37.8% suspicious, and 0.3% malicious.The suspicious class is a review-before-trusting posture rather than an abuse label.
  • Verdict distribution: 32.7% of clean skills still carry a SkillSpector advisory without contradicting the clean registry verdict.This indicates that advisory findings can coexist with a clean final verdict.
  • SkillSpector risk categories: SkillSpector’s common categories describe authority, scope, tool semantics, and disclosure rather than classic malware indicators.Data Exfiltration appears in 1,196 clean skills and 996 suspicious skills, while Dangerous Code Execution skews toward suspicious skills.
  • Static findings: 1,428 rows contain dangerous-execution static findings, followed by environment-credential access in 1,298 and exposed secret literals in 1,219.Other listed findings include dynamic code execution, prompt-injection instructions, untrusted install sources, destructive deletes, potential exfiltration, insecure TLS verification, and secret exposure via command arguments.

8 Illustrative Cases · 9 OWASP-Aligned Risk Lens

Illustrative cases show that clean skills can still carry substantial agentic-risk advisories, while trust dimensions and summary verdicts may diverge. The paper uses OWASP GenAI categories as a shared vocabulary for grouping observable skill evidence without treating them as official dataset labels.

  • 8 Illustrative Cases: Illustrative cases are context-dependent and are not presented as a formal qualitative analysis.Aggregate statistics can understate this context dependence.
  • 8 Illustrative Cases: Policy blocking can address abuse, evasion, or under-disclosed control even when the installer is not the direct victim.The paper compares this malware–moderation distinction with the potentially unwanted application grey zone.
  • 8 Illustrative Cases: Clean skills can still receive strong agentic-risk advisories when they transparently sync meeting notes or handle sensitive biometric data and OAuth tokens.The cited examples, scald/granola and 4xiomdev/whoop-central, each received a SkillSpector score of 100.
  • 8 Illustrative Cases: The advisory describes what users accept rather than asserting wrongdoing by the skill.Both cited examples are characterized as correctly clean while carrying strong advisories.
  • 8 Illustrative Cases: Skill trust has multiple facets, including malware reputation, static code risk, semantic agentic risk, disclosure, and registry posture.These facets can diverge across illustrative cases.
  • 8 Illustrative Cases: Summary verdicts should therefore be interpreted with underlying evidence rather than treated as standalone ground truth.The cases demonstrate divergence among trust dimensions and registry posture.
  • 9 OWASP-Aligned Risk Lens: OWASP’s GenAI Security Project separates risks for LLM applications, agentic applications, and skills.The paper adopts these categories as a shared vocabulary for grouping observable evidence in Table 7.
  • 9 OWASP-Aligned Risk Lens: The dataset categories are not claimed to be official OWASP labels.The OWASP-aligned framework is used for organizing evidence rather than establishing official nomenclature.

10 Toward Human Adjudication · 11 Threats to Validity

Because scanner outputs are substantially disjoint, disputed skills often require human judgment, while the study’s automated labels, constructs, data, coverage, and temporal scope impose important validity limits. The authors therefore position the release as a basis for human-annotated adjudication and disagreement analysis rather than definitive ground truth.

  • 10 Toward Human Adjudication: 56.3% of review-needed skills rely on a single semantic agent-risk signal, while 24.3% of malicious registry verdicts have no scanner support.These patterns make trust decisions about disclosure, intent, and agentic risk genuinely uncertain and frequently dependent on human judgment.
  • 10 Toward Human Adjudication: A subsequent dataset version could over-sample single-scanner positives, scanner conflicts, clean-but-advised skills, high-agentic-risk categories, and code-bearing bundle files for human annotation.The proposed adjudication would record separate dimensions rather than forcing one opaque label per skill.
  • 11 Threats to Validity: 99.6% of rows received registry verdicts from OpenAI GPT-5.5 high, so labels are silver labels rather than human ground truth and model configuration could shift the clean/suspicious boundary.The registry’s automated review supplies the label provenance for the corpus.
  • 11 Threats to Validity: Positive scanner status is evidence rather than confirmed vulnerability; the study measures detector agreement, whose disagreement claims are more robust to verdict error than prevalence claims.The authors deliberately avoid prevalence claims because correctness is not established.
  • 11 Threats to Validity: SkillSpector and ClawScan both partly rely on LLMs, so their correlation may reflect shared mechanisms and LLM biases rather than independent confirmation.This motivates studying disagreement instead of agreement with the verdict.
  • 11 Threats to Validity: The sanitized corpus redacts skill content, exported bundle files, secret-like values, private identifiers, and private artifacts, so exact scanner reproduction may require released metadata.The release is a research corpus, not a byte-for-byte registry mirror.
  • 11 Threats to Validity: 97.3% of rows have resolved VirusTotal results, including 233 stale rows and 1,580 rows without results; “not positive” is not human-confirmed clean.The earlier pending-queue caveat no longer drives agreement statistics, and scanner pairs remain near-zero after chance correction.
  • 11 Threats to Validity: The corpus covers one registry, latest versions, public skills, an English-heavy sample, and one dated snapshot, while skills, scanner versions, and asynchronous VirusTotal results can drift.These selection and temporal constraints limit how broadly the findings can be generalized.

12 Data Availability, Licensing, and Maintenance · 13 Discussion

The dataset is released as a documented, permissively licensed, living resource, while the discussion argues that agent-skill security requires layered evidence and specialized triage rather than single allow/block decisions. Advisories and suspicious labels should not be treated as accusations or proof of maliciousness.

  • 12 Data Availability, Licensing, and Maintenance: The dataset is released on Hugging Face with a datasheet and machine-readable metadata describing composition, collection, intended use, and limitations.The sanitized signals and analyzed public skill content are covered by the permissive MIT license in effect at publication.
  • 12 Data Availability, Licensing, and Maintenance: The corpus is treated as a living dataset, following the model of living systematic reviews.
  • 13 Discussion: No single scanner can comprehensively secure agent skills because the scanners examine distinct attack surfaces.The proposed defense integrates complementary components mapped to specific surfaces, including reputation and signature scanning for bundled-code malware and static analysis for code-pattern risks.
  • 13 Discussion: Skill security is best addressed as a layered pipeline that transparently presents evidence underlying each verdict rather than relying on a single allow/block mechanism.
  • 13 Discussion: Advisories are not accusations, and suspicious labels are not equivalent to maliciousness.One-third of clean skills carry an advisory, while most suspicious skills have no static or VirusTotal positive; advised skills should be evaluated for disclosed, purpose-aligned, least-privileged, and bounded capabilities.
  • 13 Discussion: Structured, large-scale scanner disagreement creates an opportunity for specialized skill-security triage models.Potential tasks include predicting when semantic advisories should trigger review, require documentation, affect registry posture, or support drafting a Skill Card summary.
  • 13 Discussion: The score separations reported earlier indicate that these triage models have meaningful signal to learn from.

14 Ethics and Responsible Disclosure · 15 Conclusion

The paper frames sanitized skill data as suitable for scanner, trust, and moderation research rather than offensive use, while emphasizing that agent-skill security spans malware detection, contextual trust, and repository governance.

  • 14 Ethics and Responsible Disclosure: Sanitized skill content is intended for scanner, trust, and moderation research, not offensive use or exploit reproduction.Case-study identifiers and rationales are paraphrased and redacted to illustrate trust categories without attributing wrongdoing.
  • 14 Ethics and Responsible Disclosure: Public case-study slugs are used to illustrate trust categories, not to accuse clean or suspicious skills of wrongdoing.A positive signal is explicitly not treated as an accusation.
  • 15 Conclusion: Agent skills distribute malware and potentially unwanted application detection evidence across prose, configuration, tool wiring, and executable code.This differs from conventional detection settings where evidence is concentrated in a binary.
  • 15 Conclusion: Most skills are benign, while a small fraction are clearly malicious.The conclusion distinguishes these groups from a consequential context-dependent middle ground.
  • 15 Conclusion: Identical capabilities may be legitimate or unacceptable depending on authorship, disclosure, and the authority granted to the agent.This context-dependent middle ground makes simple capability-based judgments insufficient.
  • 15 Conclusion: Trust is commonly established through review, creating a governance challenge similar to malicious-package moderation in repositories such as PyPI.The passage notes that such repositories may need to remove malicious packages.
Loading 2606.01494v1…