PaperlayerView exact revision on arXiv

Source-linked AI summary

MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

Wenbo Guo, Wei Zeng, Chengwei Liu, Xiaojun Jia, Yijia Xu, Lei Tang, Yong Fang, Yang Liu

arXiv:2606.07131v3cs.CRcs.SE

TL;DR

Malicious agent skills combine code and instructions, but existing evaluations lack verified ground truth spanning this hybrid attack surface. MalSkillBench builds a runtime-verified benchmark and evaluates detectors across it, finding that no single-domain approach reliably captures maliciousness expressed through the relationship between task, code, and instructions.

  • Problem

    Detector effectiveness is difficult to assess because existing data lacks verified ground truth spanning skills’ code-level and instruction-level attack surface.

  • Method

    MalSkillBench uses a closed-loop Generate-Verify-Feedback pipeline, runtime behavior checks, and a 108-cell taxonomy to construct and evaluate malicious and benign skills.

  • Results

    No single-domain tool or naive combination reliably captures the hybrid attack surface, while wild-only scoring can reorder detector rankings.

  • Takeaways & Limitations

    Detecting malicious skills requires joint reasoning over the advertised task, executable code, and agent-facing instructions.

  • Takeaways & Limitations

    Wild samples remain narrow and dominated by dependency impersonation, leaving much of the attack surface underrepresented without generated coverage.

Abstract

from arXiv · show

AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves with third-party skills: markdown packages bundling natural-language instructions, executable scripts, and tool permissions. Because a skill is at once code and agent-facing instruction, it introduces a supply chain dependency whose risk is neither pure code nor pure prompt. Detection tools have never been measured against verified ground truth spanning this hybrid space, leaving their effectiveness unknown and wild-only evaluations biased. We present MalSkillBench, the first runtime-verified benchmark of malicious agent skills: 3,944 malicious skills labeled along a three-dimensional taxonomy of 108 cells. Of these, 3,214 come from a closed-loop Generate-Verify-Feedback pipeline admitting only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge; we add 703 in-the-wild and 4,000 matched benign skills. Our measurements are consistent: code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect; the wild sample is narrow, dominated by one cryptocurrency-theft campaign (86.6% one behavior, 81% from two accounts) with a small but architecturally new tail attacking the agent control plane; the strongest skill-specific detector reaches 98.4% recall on code injection yet collapses on prompt-injection and agent-control attacks, and wild-only scoring swings the ranking by up to 66 recall points; supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship. Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions. We release the dataset, pipeline, baselines, and results.

1 Introduction

AI coding-agent skills combine executable code, natural-language instructions, and tool configurations, creating a hybrid attack surface that existing detector evaluations cannot reliably measure. MalSkillBench addresses this gap with runtime-verified coverage across the attack space and systematic detector evaluation.

  • The hybrid skill attack surface: Skills function as third-party software dependencies by bundling instructions, scripts, and tool configurations that agents load and execute when requests match.Their hybrid nature combines code-layer and instruction-layer threats within one artifact.
  • The hybrid skill attack surface: Code injection embeds executable payloads, whereas prompt injection manipulates markdown instructions to alter agent behavior or bypass safeguards.The two vectors draw on established package-malware and social-engineering patterns, respectively.
  • Evaluation gap: Existing public data lacks verified, broad ground truth: prior academic benchmarks contain only 157 wild samples, while industry reports withhold underlying samples.This prevents reliable comparison of detector capability across the broader attack surface.
  • Evaluation design: The benchmark organizes evaluation around attack realizability, wild attack patterns, skill-specific detection, and transferability from existing security tools.These four research questions cover both attack construction and detector performance.
  • Benchmark construction: MalSkillBench constructs 3,944 malicious skills and 4,000 paired benign samples across 108 taxonomy cells using a closed-loop pipeline with sandbox execution and system-call evidence.Only samples whose observed behavior matches their declared indicators enter the dataset.

2 Motivation

Existing wild datasets are too narrow to represent the malicious-skill attack space, motivating systematic expansion from established attack patterns. The paper shows that both package malware and phishing-style social engineering migrate into skills through different carriers and triggers.

  • Limitations of existing data: Existing public and wild datasets leave most of the attack space empirically invisible, even when combined.This limits their use as standalone detector benchmarks.
  • Limitations of existing data: The 157-sample wild benchmark is highly concentrated in commercial-brand impersonation, remote code execution, and credential theft, with pure instruction-layer attacks rare.Its concentration prevents claims of coverage across the broader attack space.
  • Limitations of existing data: 86.3% of the 703 curated wild skills use dependency impersonation, while prompt-injection and control-plane attacks appear only as low-frequency outliers.The dominant outcome is host-level malware delivery through a fake trusted-looking prerequisite.
  • Attacks migrate into skills: Attackers reuse core behaviors such as exfiltration, credential theft, backdoors, and social engineering while changing the carrier used to deliver them.This recurring structure supports mining established attack knowledge bases for systematic benchmark expansion.
  • Attacks migrate into skills: A malicious skill reproduces a PyPI reverse shell with the same code structure, system calls, and attack chain, changing only its carrier and activation trigger.The package runs on installation, whereas the skill runs on agent activation.
  • Attacks migrate into skills: A phishing-style prerequisite in SKILL.md can socially engineer an agent into downloading and executing an external dependency, achieving remote code execution without a code vulnerability.This illustrates an agent-mediated prompt-injection pathway distinct from package-malware transplantation.

3 Methodology

MalSkillBench constructs a runtime-verified benchmark by generating malicious skills across a three-dimensional taxonomy, verifying observed behavior, and combining verified, wild, test-collected, and benign samples.

  • Benchmark Construction: The framework retrieves attack patterns, selects benign facades, synthesizes skills for taxonomy cells, verifies candidates, and feeds rejected samples back for regeneration.Verified candidates are combined with wild-collected skills and existing tool test samples in the final dataset.
  • Skill Model: A skill bundles markdown instructions, executable scripts, and optional resources, exposing distinct frontmatter, instruction, and code attack surfaces.The taxonomy organizes attacks across code, instructions, and cross-tier combinations.
  • Attack Taxonomy: The taxonomy spans attack vector, malicious behavior, and insertion strategy, covering CI, PI, and MIXED attacks across 108 valid cells.B1–B9 are deliverable through CI and PI, while B10–B15 target agent reasoning and are PI-only.
  • Knowledge Base: Knowledge bases provide 3,026 malicious PyPI packages for code-injection patterns and 20,961 prompt-injection payloads for prompt-level behaviors.The pipeline labels heterogeneous source entries with an LLM so they can be retrieved by taxonomy coordinates.
  • Verification: Runtime verification matches collected evidence against declared indicators, while an LLM judge classifies prompt-injection outcomes and assigns confidence.For CI and MIXED skills, matching examines network, file, and process evidence; triggered and detected outcomes both count as positive.
  • Dataset Composition: The released benchmark contains 3,944 malicious skills—3,214 generated, 703 wild, and 27 test-collected—plus 4,000 benign skills.Generated skills provide approximately balanced coverage, while wild and test-collected samples add real-world grounding and current tool threat models.

4 Empirical Study

The empirical study evaluates benchmark construction and malicious-skill detection through four research questions covering realizability, wild attacks, skill-specific detectors, and tool transferability.

  • Research Questions: The study asks which hybrid attacks can be reliably realized, what wild skills reveal, whether skill-specific detectors cover CI and PI, and whether existing tools transfer.These questions jointly assess benchmark quality and current detection capability.
  • Experimental Setup: The generation agent uses an abliterated Qwen3.5-35B model served locally through Ollama to produce attack content without safety-trained refusal behavior.The implementation targets reverse shells, credential theft, and prompt-injection payloads.
  • Experimental Setup: Generation runs at T=0 and top-p=0.3, executes candidates for up to 360 seconds, and allows up to r=3 regeneration rounds after rejection.Verification uses threshold θ=0.7.
  • Baselines: Table 3 evaluates baseline detection tools on MalSkillBench.The baselines are part of the empirical comparison of malicious-skill detection approaches.

4.2 Baseline Selection and Configuration

Baseline selection prioritizes reproducibility, approach diversity, and full-dataset evaluation so observed coverage gaps are not attributable to one weak detector design.

  • Selection Criteria: Baselines must be open-source or expose a documented public interface, enabling independent reproduction of every reported result.This is the first baseline-selection criterion.
  • Selection Criteria: The selection spans rule-based static analysis, hybrid static-plus-LLM analysis, and LLM-native prompt scanning for RQ3.RQ4 includes both static-analysis and LLM-assisted designs.
  • Configuration: All baselines are pinned to their latest public release at benchmark freeze time and applied to the full dataset.Rule-based and static tools use bundled rules, while model-based tools use published defaults except where an explicit backend is required.

4.3 RQ1: Attack Realizability

RQ1 finds uneven attack realizability: code-bearing attacks verify reliably, while instruction-level prompt injection is harder to realize, especially under covert insertion. Runtime evidence is deterministic for CI and MIXED but judge-based for PI.

  • Runtime verification: 3,214 samples were realized across all 108 taxonomy cells, using runtime verification to confirm intended malicious behavior.The pipeline admits a candidate only when its intended behavior triggers during agent execution.
  • Vector-level realizability: 94.5% CI and 91.9% MIXED realizability exceeded 75.8% for PI.PI-only reasoning behaviors were among the least realizable, including Instruction Override at 63.3%.
  • Insertion strategies: 62.5% steganographic PI realizability was substantially below Full Camouflage at 89.7% and Partial Injection at 74.3%.CI and MIXED remained comparatively stable across their insertion strategies.
  • Runtime evidence: 99.7% of accepted PI samples exceeded the judge threshold θ=0.7, with median confidence 0.97.CI and MIXED instead produced concrete file, process, and network evidence, with median matched indicators of 7 and 6.
  • Implication: Instruction-level attacks leave fewer concrete traces and are correspondingly harder to detect.This links PI’s weaker observability during verification to its later detection difficulty.

4.4 RQ2: Real-World Analysis

RQ2 shows that the wild sample is dominated by conventional malware delivered through fake prerequisites, while a small agent-native tail targets the agent control plane. The concentration means wild data alone cannot cover the broader taxonomy.

  • Campaign concentration: 81% of the wild skills came from two accounts, while one cryptocurrency-theft campaign contributed 35% of the sample.The campaign targeted wallets, seed phrases, or chains such as Solana and Ethereum.
  • Coverage limitation: Wild-only data would leave most of the taxonomy untested, so the benchmark pairs it with generated coverage and analyzes the sources separately.This scope boundary follows from the wild sample’s concentration on a few campaigns and behaviors.
  • Conventional payloads: The dominant wild pattern translates conventional supply-chain malware into an agent-task delivery frame.The harmful dependency is installed or run while the agent performs an apparently benign user task.
  • Agent-native tail: 1.7% of wild skills targeted the agent control plane through goals, identity, session lifecycle, or instruction priority.These agent-native behaviors have no counterpart in conventional package malware and motivate taxonomy cells B10–B15.
  • Wild-sample concentration: 86.3% of wild skills used fake-prerequisite dependency impersonation, and 86.6% mapped to Malware Delivery (B4).The sample was also concentrated by authorship: two accounts contributed 81% of the skills.

4.5 RQ3: Skill-Specific Detection

Skill-specific detectors are not uniformly reliable: aggregate recall can be high while instruction-carried and insertion-obfuscated attacks remain difficult, and wild-only evaluation can reverse detector rankings.

  • Overall detector effectiveness: 98.4% recall is achieved by Sentry Skill Scanner in full mode, but it flags 937 benign skills.AI-Infra-Guard is more balanced at 85.6% F1, 86.6% recall, and 620 false positives.
  • Overall detector effectiveness: Maliciousness is usually a semantic relationship between requested authority, actions, and the advertised task rather than a localized artifact.LLM-based scanners therefore gain recall by reading instructions and scripts together, while signature scanners trade recall for fewer false positives.
  • Detection gaps by behavior and insertion strategy: 35.8% and 32.1% recall on Content Manipulation show that high aggregate recall does not imply uniform behavioral coverage.AI-Infra-Guard and Cisco LLM recover code-backed behaviors more reliably than instruction-layer attacks.
  • Detection gaps by behavior and insertion strategy: Sentry Full reaches 99.8%, 97.6%, and 99.8% recall on CI, PI, and MIXED, while Cisco Static falls to 18.7% on PI.Instruction-only attacks provide fewer concrete artifacts for detector verdicts than executable code or download chains.
  • Detection gaps by behavior and insertion strategy: 5.1% and 8.5% are Sentry Static’s recall on Partial Injection and Full Camouflage, where operative content is hidden in surrounding text or covert encodings.Explicit source-to-sink chains are easier for static and LLM scanners because code and intent reinforce each other.
  • Wild-only detection performance: 66 points separate some detectors’ wild-only and full-benchmark recall, and the ranking can reverse because the wild subset is narrow.VirusTotal rises from 21.6% to 87.9%, while AI-Infra-Guard falls to 74.0% on the 703 wild skills.
  • Wild-only detection performance: 99.3% recall on wild Malware Delivery but only 16.7%, 9.1%, and 6.7% on three other behaviors shows that aggregate wild recall reflects behavior prevalence.LLM-based detectors maintain at least 60% recall across the four compared behaviors.

4.6 RQ4: Tool Transferability

Transferred supply-chain scanners and prompt-injection defenses detect only partial signals from malicious skills. Combining their verdicts does not recover the code-instruction relationship and instead trades false positives against false negatives.

  • Transferred tool performance: 97.3% and 99.5% recall from OSSGadget and SAP-DT comes with 3,587 and 3,858 false positives.GuardDog and MalGuard-MLP produce only 49 and 4 false positives but recall 15.4% and 15.1%.
  • Why transfer fails: Supply-chain scanners and prompt-injection defenses each read only one half of a hybrid skill, making the limitation structural rather than a tuning problem.The former focus on package-level evidence, while the latter target jailbreak-shaped prompts.
  • Naive combinations: 100.0% recall paired with 100.0% false-positive rate illustrates the cost of OR-combining high-recall transferred tools.OSSGadget or NeMo Guardrails reaches 99.2% recall with 96.4% false-positive rate.
  • Naive combinations: AND-combination can reduce false positives to zero, but recall collapses to 5.0%, 3.3%, and 2.8%.The few AND pairs retaining 94–96% recall still carry 80–84% false-positive rates.
  • Why transfer fails: No set operation over independent verdicts recovers whether a skill’s requested dependency or action is justified by its advertised task.That relationship is absent from both transferred tool families’ views.
  • Conclusion: Up to 4,000 false positives on 4,000 benign skills and AND recall below 6% summarize why transferred tools do not provide a usable solution.The supported conclusion is that malicious-skill detection requires joint reasoning over code and instructions.

5 Discussion

The studies identify a shared weakness across attack realization, detection, and tool transfer: maliciousness often resides in the relationship between a skill’s intent, code, and instructions. Reliable evaluation must therefore cover attack regions that wild data misses and require detectors to reason across the whole artifact.

  • Instruction-level attacks are both the least reliably realized and the least reliably detected across the studies.The same hard cases recur in attack realizability, skill-specific detection, and tool transfer.
  • Combining supply-chain scanners with prompt-injection defenses does not recover the missing relationship between a skill’s code and instructions.Skill-specific detectors perform better by reading both together, but still fail when maliciousness appears as a plausible instruction.
  • Effective detection must jointly judge advertised task, requested authority, executable behavior, and agent-control instructions.Static evidence remains necessary for code, file, or network traces, while runtime evidence tests whether the malicious path is followed.
  • Wild-only evaluation can flatter payload-centric scanners and reorder detector rankings by up to 66 points.Runtime-verified generated samples fill instruction-level and agent-control regions absent from the narrow wild sample.

6 Related Work

Prior work treats malicious skills as a hybrid supply-chain and agent-security problem, but existing studies and tools cover only parts of that surface. MalSkillBench differs by providing runtime-verified, taxonomy-labeled ground truth spanning code-injection and prompt-injection skills.

  • Skills bundle SKILL.md instructions with optional Python or JavaScript helpers and are distributed through public channels, creating a low-barrier agent supply-chain target.Malicious content can reside in metadata, natural-language instructions, or bundled scripts.
  • Existing skill studies emphasize instruction attacks, while code-bearing threats in bundled scripts receive less coverage.Skill-Inject measures agent vulnerability and SkillJect automates stealthy injection for coding agents.
  • Public skill platforms provide only lightweight moderation and user-side review, leaving detection to external auditing and agent-side defenses.Relevant defenses include real-time monitors and platform-level mitigations.
  • Academic and industry tools include auditing agents, dependency-graph reasoning, constraint-guided synthesis, semantic fuzzing, and skill-specific scanners.The related systems span both research detectors and commercial AI-security platforms.
  • Existing agent-security benchmarks assess agents or agent vulnerability rather than malicious skill artifacts with runtime-verified labels across code and prompt injection.MalSkillBench targets this unprovided combination of artifact-level scope, runtime verification, taxonomy labels, and hybrid attack coverage.

7 Conclusion

MalSkillBench is presented as a runtime-verified benchmark spanning malicious skill vectors and the full taxonomy. Its results show that detection quality depends on where malice is hidden and requires joint reasoning over task, code, and instructions.

  • MalSkillBench contains 3,944 malicious and 4,000 benign skills covering code-injection, prompt-injection, mixed vectors, and all 108 taxonomy cells.
  • The strongest detector reaches 98.4% recall yet collapses on instruction-level attacks, while transferred single-domain tools fail to cover the hybrid surface.
  • Wild-only scoring reorders detector rankings, showing that measured quality depends on where a skill hides its malice.
  • Detecting malicious skills therefore requires joint reasoning over a skill’s task, code, and instructions.

8 Open Source

The dataset, baselines, and evaluation pipeline are publicly available. The section also contains non-substantive conference metadata.

  • The dataset, baselines, and evaluation pipeline are publicly available at the project repository.
Loading 2606.07131v3…