PaperlayerView exact revision on arXiv

Source-linked AI summary

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Yue Wang, Yi Liu, Gelei Deng, Ying Zhang, Yuekang Li, Zhenyu Chen, Leo Zhang

arXiv:2608.19901v1cs.CRcs.AI

TL;DR

Existing malicious-Skill resources differ in artifacts, evidence, and benign coverage, making them difficult to combine reliably. MALICIOUSSKILLBENCH consolidates public sources and evaluates learned detectors and scanners across random, structural-disjoint, and held-out-source settings. The results show distinct recall and false-positive trade-offs, with no evaluated detector achieving both high malicious recall and low benign over-flagging across held-out sources.

  • Problem

    Malicious-Skill datasets vary in artifact types, evidence regimes, and benign coverage, while duplicated and structurally related content complicates aggregation and evaluation.

    Source: §3 · p. 4

  • Method

    The paper consolidates 13 public sources with evidence-preserving canonicalization, identity and structural-reuse controls, conflict exclusion, and source-native attack characterization.

    Source: §1 · p. 2

  • Results

    Source-Disjoint Macro-F1 is 0.653–0.665 versus Random Macro-F1 of 0.882–0.932, while held-out-source tests show learned detectors over-flag benign Skills and scanners sharply reduce recall.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Reliable malicious Agent Skill detection remains open and requires diverse malicious and benign coverage together with source-aware, two-sided evaluation.

    Source: §9 · p. 9

    The evaluated learned baselines and scanner track use static primary Skill artifacts and exclude package-level and runtime behavior.

    Source: §8 · p. 9

Abstract

from arXiv · show

Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a direct distribution channel for malicious behavior, yet existing malicious-Skill datasets are fragmented across sources, artifact formats, evidence regimes, and benign coverage; duplicated and structurally related content further complicates direct aggregation and evaluation. We present MaliciousSkillBench, a comprehensive benchmark for malicious Agent Skill detection. We consolidate 13 public sources, 11 of which contribute Core malicious artifacts, and reduce 8,414 raw malicious records to 7,539 normalized-unique identities in 4,588 operational structural families. After conservative cross-label conflict exclusion, the primary benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign. To characterize its coverage, we harmonize 11 attack categories for 4,983 malicious identities with supported source-native mappings and find substantial differences in threat composition across sources. We then evaluate three learned text detectors and three off-the-shelf Skill scanners. Learned detectors achieve 0.882-0.932 Random Macro-F1 but only 0.653-0.665 under Source-Disjoint evaluation; the strongest word TF-IDF SVM scores 0.932/0.916/0.665 on Random/structural-disjoint/Source-Disjoint while retaining 95.6% malicious recall but producing 62.4% benign FPR on held-out sources. Off-the-shelf scanners occupy different but also unsatisfactory operating regimes, reducing false positives only at the cost of sharply lower malicious recall. Together, these results show that reliable malicious-Skill detection requires both broader cross-source benchmark coverage and evaluation that jointly measures attack detection and benign over-flagging.

1 INTRODUCTION

Malicious-Skill research lacks a unified basis because sources differ in artifacts, evidence, and benign coverage, with duplicates and conflicting labels complicating aggregation. MALICIOUSSKILLBENCH consolidates these resources and finds both heterogeneous attack coverage and a persistent detection trade-off under held-out-source evaluation.

Motivation

  • Existing datasets differ in artifact types, evidence regimes, and benign coverage, while overlapping content and conflicting labels make direct aggregation unreliable.The sources include complete artifacts, constructed variants, runtime-verified cases, vulnerability records, marketplace observations, and scanner-derived signals.Source: §1 · p. 1

Benchmark contribution

  • The benchmark consolidates 13 public sources into 9,740 Skills—7,505 malicious and 2,235 benign—after identity control, structural grouping, benign curation, and conflict filtering.Its malicious side reduces 8,414 raw Core artifacts to 7,539 normalized-unique identities organized into 4,588 operational structural families.Source: §1 · p. 2

Threat characterization

  • Harmonized mappings cover 11 attack categories for 4,983 malicious identities, and their distributions differ sharply across sources.The mapping uses documented source-native labels and deterministic strong-semantic mappings.Source: §1 · pp. 2–3

Detector evaluation

  • The strongest word TF–IDF SVM scores 0.932/0.916/0.665 under Random, malicious-structural-disjoint, and Source-Disjoint evaluation, retaining 95.6% malicious recall but yielding 62.4% benign FPR on held-out sources.Off-the-shelf scanners occupy a different regime: lower false positives come with sharply lower malicious recall, and no evaluated detector combines high recall with low benign FPR across held-out sources.Source: §1 · pp. 2–3

2 BACKGROUND AND RELATED WORK

The related work positions malicious Skills at a trust boundary distinct from broader tool-mediated attacks and surveys complementary Skill threats, datasets, and defenses. It also motivates benchmark controls for duplication and source shift, which this work addresses with source-aware evaluation.

2.1 AGENT AND TOOL SECURITY

2.2 SKILL-SPECIFIC THREATS, DATASETS, AND DEFENSES

  • Prior work spans Skill-file injection, triggered backdoors, unsafe behavior, marketplace vulnerabilities, artifact datasets, scanner resources, and defenses from static screening to runtime audit.MALICIOUSSKILLBENCH complements these proposals by consolidating heterogeneous resources into an artifact-level benchmark and evaluating learned and off-the-shelf detectors under one frozen data contract.Source: §2.2 · p. 3

2.3 BENCHMARK ARTIFACTS AND DISTRIBUTION SHIFT

  • Annotation artifacts and distribution shift motivate controlling duplication and standardizing evaluation; source identity and construction provenance are audited but excluded from detector features.The evaluation separates random, malicious structural-family-disjoint, and held-out-source tests, with the last measuring source-conditioned stress under coupled factors.Source: §2.3 · pp. 3–4

3 WHY A COMPREHENSIVE BENCHMARK?

The section argues that a useful malicious-Skill benchmark must reconcile non-equivalent sources, control reused content and label conflicts, include credible benign examples, and test beyond random splits. MALICIOUSSKILLBENCH is designed around those requirements.

3 WHY A COMPREHENSIVE BENCHMARK?

  • Published resources cover complementary threat slices but release non-equivalent artifacts and labels, so consolidation must preserve source, artifact, provenance, and evidence semantics.A runtime-supported complete Skill artifact is not the same observation as a vulnerability record or an automated scanner verdict.Source: §3 · p. 4
  • The frozen collection’s 8,414 Core malicious records reduce to 7,539 normalized-unique contents, 4,588 structural families, and 34 identities with conflicting malicious and benign labels.Identity, scaffold reuse, and label inconsistency are distinct phenomena; concatenation can inflate scale and leak reused content across splits.Source: §3 · p. 4
  • The primary benchmark has 7,505 malicious and 2,235 Main-benign Skills, addressing uneven source support and varying benign-label strength with conservative eligibility and conflict exclusion.Because source, construction procedure, and class composition are coupled, broad flagging of unfamiliar artifacts can look like high attack coverage.Source: §3 · p. 4
  • Random splits can distribute source conventions and related variants across train and test, motivating complementary random, structural-disjoint, and held-out-source evaluations.Held-out-source evaluation measures source-conditioned stress under coupled source factors.Source: §3 · p. 4
  • The benchmark design combines a common artifact-level unit, evidence-preserving canonicalization, identity/reuse/conflict controls, high-confidence benign data, and distinct generalization protocols.These controls translate the section’s four deficiencies into a shared benchmark design.Source: §3 · p. 4

4 BUILDING MALICIOUSSKILLBENCH

MALICIOUSSKILLBENCH is built through conservative source qualification, evidence-preserving canonicalization, separate identity and structural-reuse controls, and careful benign-pool curation. Cross-label conflicts are excluded to produce one frozen primary benchmark for evaluation.

4.1 SOURCE COLLECTION AND ARTIFACT ELIGIBILITY

  • Eleven of 13 frozen public sources meet the Core-malicious artifact rule and contribute 8,414 raw Skill artifacts spanning wild, test-fixture, injected/backdoored, and synthetic collections.The source heterogeneity is retained so later analyses can distinguish observed, constructed, and inferred content.Source: §4.1 · p. 5

4.2 CANONICALIZATION WITHOUT ERASING EVIDENCE

  • Each acquired record receives a stable canonical ID and retains source, provenance, label-strength, and evidence metadata; unsupported vulnerable, suspicious, or scanner-flagged cases are not upgraded to malicious ground truth.Potentially malicious artifacts are handled as inert static data; Skill code, helpers, payloads, URLs, installers, and embedded instructions are not executed.Source: §4.2 · p. 5

4.3 DEDUPLICATION AND STRUCTURAL REUSE CONTROL

  • Exact and normalized identity are separated from structural reuse: 8,414 raw Core artifacts yield 7,562 exact-unique and 7,539 normalized-unique identities, then 4,588 operational structural families.Families support reuse audits and malicious-side structural-disjoint splitting, not attack, campaign, actor, or threat-class semantics.Source: §4.3 · p. 6

4.4 BENIGN POOL, LABEL CONFLICTS, AND FINAL BENCHMARK

  • Main benign records require an actual Skill artifact plus strong or moderate evidence; weaker negatives remain Auxiliary and are excluded from primary benign ground truth.The pools contain 2,251 raw Main-benign artifacts and 45,966 raw Auxiliary-benign artifacts before conflict exclusion.Source: §4.4 · p. 6
  • The frozen primary benchmark contains 9,740 normalized-unique Skills: 7,505 malicious and 2,235 Main benign, after excluding all 34 cross-label-conflicting identities.Conflicting identities are retained for consistency auditing, and all evaluation protocols start from the same master table without split-specific relabeling or synthetic oversampling.Source: §4.4 · p. 6

5 THREAT LANDSCAPE OF MALICIOUSSKILLBENCH

The benchmark maps 11 attack categories for a supported subset of malicious Skills and finds that threat profiles vary substantially across sources. Conservative impact mappings characterize a smaller subset, where several attack–impact pairs co-occur descriptively.

5 THREAT LANDSCAPE OF MALICIOUSSKILLBENCH

  • Attack characterization uses direct labels and deterministic strong-semantic mappings for 4,983 of 7,505 malicious identities (66.4%) across 11 categories, without inferring labels from Skill text.Nine of the 11 Core-contributing sources provide mappable per-unit annotations; unsupported units remain unannotated.Source: §5 · p. 6

5.1 ATTACK COVERAGE IS BROAD BUT SOURCE-DEPENDENT

  • Execution / Code Delivery appears in 3,320 identities (66.6% of the mapped subset), followed by Instruction / Goal / Memory Manipulation in 1,671 (33.5%) and Privilege / Tool / Authority Abuse in 1,013 (20.3%).The taxonomy is multi-label, so category percentages do not sum to 100%.Source: §5.1 · p. 6, Figure 2 · p. 7
  • SkillTrustBench contributes 2,755 of 3,320 execution-coded identities and 1,258 of 1,671 instruction-manipulation identities, while MalSkillBench contributes 262 of 330 resource/availability-abuse identities.The sources provide complementary, source-specific threat coverage, with different mixtures of wild, synthetic, injected, backdoored, and test-fixture sources.Source: §5.1 · p. 6

5.2 DERIVED IMPACTS PROVIDE A MORE CONSERVATIVE VIEW

  • Among 1,888 identities with both mappings, execution/code delivery most often co-occurs with unauthorized code execution or system control (489), data exfiltration/disclosure with sensitive-data disclosure (345), and credential access with credential compromise (293).These are descriptive co-occurrences, not causal transitions or benchmark-wide ground truth; conservative impact mappings cover 2,128 identities (28.4%).Source: §5.2 · p. 7, Figure 2 · p. 7
  • The consolidated benchmark broadens observed threat coverage while showing that no single source represents the full observed malicious-Skill landscape.This heterogeneity motivates multi-source construction and source-aware evaluation.Source: §5.2 · p. 7

6 BENCHMARKING MALICIOUS AGENT SKILL DETECTION

Across held-out sources, learned detectors and fixed scanners occupy different but unsatisfactory operating regimes: learned models retain malicious recall but over-flag benign Skills, while scanners reduce false positives with much lower recall. Random-split results overstate cross-source performance, and the evaluation uses static primary Skill artifacts.

6.1 EVALUATION DESIGN

  • Evaluation uses a frozen 9,740-unit table and Random, malicious-structural-disjoint, and Source-Disjoint protocols; Source-Disjoint holds out SRC009, SRC011, and SRC012 after excluding eight cross-provenance identities.Learned baselines use inert primary Skill instruction text, while scanners use fixed pre-registered gates; the operational comparison does not isolate model capacity.Source: §6.1 · pp. 7–8

6.2 FINDING 1: RANDOM EVALUATION OVERSTATES ROBUSTNESS

  • Learned-baseline Macro-F1 is 0.882–0.932 on Random but 0.653–0.665 on Source-Disjoint; the strongest word TF–IDF SVM scores 0.932/0.916/0.665 across Random, Malicious-Structural-Disjoint, and Source-Disjoint.Structural-family disjointness reduces Random Macro-F1 by only 0.016–0.038, modestly relative to the source-holdout gap.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 FINDING 2: THE MAIN CROSS-SOURCE FAILURE IS BENIGN OVER-FLAGGING

  • On Source-Disjoint, learned-detector malicious recall remains 94.4–95.7%, while benign FPR reaches 62.0–64.4%.The word-SVM flags 340/545 benign Skills while missing 37/839 malicious Skills; SRC011 contributes 293 false positives.Source: §6.3 · p. 8

6.4 FINDING 3: EXISTING SCANNERS DO NOT RESOLVE THE TRADE-OFF

  • On Source-Disjoint, Cisco and SkillSpector keep benign FPR at 1.1% and 0.55% but detect only 2.5% and 0% of malicious Skills; SkillFortify reaches 25.3% recall at 49.9% FPR, while word-SVM reaches 95.6% recall at 62.4% FPR.Across methods, no detector occupies the desired high-recall, low-FPR regime on held-out sources.Source: Table 3 · p. 8, §6.4 · pp. 8–9
  • Excluding SRC012 lowers SkillFortify’s Source-Disjoint Macro-F1 from 0.349 to 0.254 and worsens both recall and FPR; this is source-overlap sensitivity, not evidence of memorization.On SRC011’s 455 benign Skills, the word-SVM produces 293 false positives, SkillFortify 272, Cisco 6, and SkillSpector 3.Source: §6.4 · pp. 8–9

7 DISCUSSION

The discussion argues that benchmark comprehensiveness depends on diversity and control, not simply adding malicious rows. Detector quality must also be assessed on both malicious detection and benign over-flagging.

7 DISCUSSION

  • A realistic benchmark must report malicious detection and benign over-flagging together because random-split Macro-F1, attack recall, or false alarms alone provide incomplete views of detector utility.Source-Disjoint couples provenance, construction, labeling, and class composition; learned models preserve 94.4–95.7% malicious recall while benign FPR reaches 62.0–64.4%.Source: §7 · p. 9
  • Simply adding malicious rows is insufficient when they duplicate existing content, concentrate in a few construction styles, or are paired with narrow benign data.Identity/conflict control precedes splitting, while source/evidence metadata support audit and stratification and are excluded from detector features.Source: §7 · p. 9

8 LIMITATIONS AND THREATS TO VALIDITY

The paper bounds its coverage claims to 13 frozen public sources and its source-disjoint results to a setting where several dataset properties remain coupled. Characterization and detector conclusions are further bounded by mapped subsets, narrower benign-source coverage, and static-artifact evaluation.

8 LIMITATIONS AND THREATS TO VALIDITY

  • Benchmark coverage is bounded by the 13 frozen public sources, and Source-Disjoint measures source-conditioned generalization with provenance, construction, labeling, and class composition coupled.Attack mappings cover 4,983/7,505 malicious identities, derived impacts 2,128, and their intersection 1,888.Source: §8 · p. 9
  • The Main benign pool favors label confidence over ecological balance, so its source coverage is narrower than the malicious side.Learned baselines and the common scanner track use static primary Skill artifacts and exclude package-level/runtime behavior, bounding claims to the evaluated benchmark and configurations.Source: §8 · p. 9

9 CONCLUSION

MALICIOUSSKILLBENCH consolidates 13 public sources into 9,740 conflict-clean normalized-unique Skills and characterizes source-specific attack profiles. Evaluation reveals a persistent trade-off between malicious detection and benign over-flagging, leaving reliable detection open.

9 CONCLUSION

  • Learned models retain high malicious recall by over-flagging unfamiliar benign Skills, while fixed scanners reduce false alarms only at much lower recall.Reliable detection remains open and requires diverse malicious and benign coverage together with source-aware, two-sided evaluation.Source: §9 · p. 9
  • The benchmark consolidates 13 public sources into 9,740 conflict-clean normalized-unique Skills with identity, reuse, evidence, and threat-coverage accounting.It includes an 11-category threat characterization showing complementary, source-specific attack profiles across existing datasets.Source: §9 · p. 9

REPRODUCIBILITY STATEMENT

The statement records frozen inputs, rules, manifests, and configurations for the benchmark and evaluations. Public materials represent all identities, but exact frozen text is withheld for five sensitive malicious records.

REPRODUCIBILITY STATEMENT

  • Frozen source revisions, metadata, eligibility rules, hashes, family assignments, tables, split manifests, leakage checks, mappings, preprocessing rules, verdict contracts, and configurations underpin the reported results.These materials are documented in the public project resources and support evaluation without executing untrusted Skill payloads.Source: REPRODUCIBILITY STATEMENT · pp. 9–10
  • All 9,740 benchmark identities are publicly represented, with exact frozen static Skill text available for 9,735 identities.The five exceptions are malicious records containing sensitive credential material; sanitized representations are supplied instead and differ from the frozen experiment inputs.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement treats Skill contents as untrusted static data and excludes execution of malicious payloads during construction and experiments. It also limits redistribution and withholds exact text for five sensitive records.

ETHICS STATEMENT

  • Skill contents are treated as untrusted static data; malicious payloads, helper scripts, and embedded commands are not executed during construction or experiments.Full-package redistribution remains subject to source-specific licenses and constraints.Source: ETHICS STATEMENT · p. 10
  • Exact frozen text is withheld for five malicious records containing sensitive credential material, with sanitized public representations released instead.The benchmark is intended for defensive detection and auditing, and documentation should avoid operationalizing harmful payloads beyond what reproducible research requires.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

This appendix documents the benchmark’s frozen, evidence-preserving construction and reuse audits, source-supported threat characterization, and static evaluation and reproducibility protocols. It also describes detector-independent robustness controls, with claims bounded by contributing-source evidence and the disjointness guaranteed by each protocol; the controls leave the benchmark definition and primary splits unchanged.

AI USE STATEMENT

  • Generative AI assisted literature search, scripting, drafting and editing, LaTeX organization, consistency checks, and blind structural-template review; authors reviewed the final claims and manuscript.An independent LLM reviewer judged all 72 blind pairs to share a structural template at high confidence; the paper does not treat this review as human ground truth.Source: AI USE STATEMENT · p. 10, §C.6 · p. 22

A.1 SCOPE AND ACQUISITION PRINCIPLES

  • The registry freezes public revisions, preserves reported and acquired counts, and inspects artifacts statically without executing code, payloads, URLs, installers, or tests.Where package redistribution is restricted or unclear, the release provides per-identity static text, source adapters, revision identifiers, metadata, and hashes.Source: §A.1 · p. 12, §A.3 · p. 13

A.2 CORE, MAIN-BENIGN, AND AUXILIARY ROLES

  • Eleven sources contributed Core malicious artifacts under the Skill-artifact eligibility rule; five also supplied 2,251 raw Main-benign records.Those benign records represented 2,238 exact- and normalized-unique units before conflict exclusion.Source: §A.2 · p. 12

B CANONICAL TAXONOMY AND DATA MODEL

  • Canonical intent, provenance, confidence, evidence, artifact identity, and source-published lineage remain separate, preserving upstream labels and disagreements rather than forcing binary labels before deduplication.Official lineage is fully resolved for 1,380 Core records and family-level partially resolved for 35; 6,999 remain unresolved, and structural families are not lineage claims.Source: B CANONICAL TAXONOMY AND DATA MODEL · p. 14, §B.6 · pp. 18–19, §B.7 · p. 19

B.4 LABEL STRENGTH AND EVIDENCE ARE ORTHOGONAL

  • Core malicious artifacts carry only strong or moderate labels, with constructed, human+runtime, static, or runtime evidence; scanner-only evidence never enters Core ground truth.The 8,414 Core artifacts comprise 8,237 strong and 177 moderate labels.Source: §B.4 · p. 17

C.1 EXACT AND NORMALIZED CONTENT IDENTITY

  • 8,414 Core malicious records yielded 7,562 exact-unique and 7,539 normalized-unique contents, grouped into 4,588 operational structural families.Exact deduplication removed 852 records; normalization merged 23 additional global identities.Source: Table 14 · p. 20, §C.1 · p. 20, §C.4 · p. 21

C.2 CROSS-SOURCE CONTENT REUSE

  • 105 exact and 119 normalized hashes recur across sources, involving 830 raw Core records at exact identity and 871 at normalized identity.Pairwise overlaps are non-additive, and the analysis makes no inference about source ancestry, actors, or campaigns.Source: §C.2 · p. 20

C.4 FAMILY DISTRIBUTION AND CROSS-SOURCE STRUCTURAL REUSE

  • SRC009 and SRC013 share 26 structural families despite no exact or normalized hash overlap; those families cover 42.35% and 62.99% of their normalized units, respectively.The audit flags 15 of 4,588 families (0.33%) for possible chaining; structural overlap does not establish copying, shared actors, or campaigns.Source: §C.4 · p. 22, §C.5 · p. 22

C.7 INTERPRETATION BOUNDARIES

  • Structural families are operational grouping variables, not ground-truth attack labels, lineage, or actor identities; validation covers only sampled inherited pairs.The check leaves recovered identities, new families, clustering recall, and threshold semantics unvalidated. Official lineage is incomplete, so it remains an audit axis rather than a corpus-wide split contract.Source: §C.6 · p. 22, §C.7 · p. 22, §F.5 · p. 31

D.3 BENIGN DEDUPLICATION AND STATIC HANDLING

  • 2,251 raw Main-benign artifacts yield 2,238 normalized-unique candidates before conflict exclusion, while the 45,966-record Auxiliary pool remains separate.Main eligibility requires an inspectable artifact, supported canonical benign intent, strong or moderate label evidence, available static content, and a traceable frozen source revision; five sources contribute Main candidates.Source: §D.1 · p. 23, §D.2 · p. 23, §D.3 · p. 24

D.4 CROSS-LABEL CONFLICT DEFINITION AND ACCOUNTING

  • 9,740 primary benchmark units remain after conflict exclusion: 7,505 malicious and 2,235 benign.The audit found 34 exact- and normalized-hash conflict groups; all 34 conflicting malicious identities and three otherwise eligible Main-benign units were excluded rather than relabeled.Source: Table 22 · p. 24, §D.4 · pp. 24–25

E.2 FINAL HARMONIZED ATTACK TAXONOMY

  • 4,983 of 7,505 malicious identities receive at least one of 11 harmonized attack categories, led by Execution / Code Delivery (3,320).Categories are multi-label and use documented direct or strong-semantic source-native mappings; 2,522 identities remain attack-unmapped, and the counts do not represent populations beyond collected sources.Source: §E.1 · p. 25, Table 24 · p. 26, §E.2 · p. 26

E.3 PROVENANCE COMPOSITION VARIES BY ATTACK CATEGORY

  • Provenance composition varies substantially by attack category, while SRC001-specific vector labels and attack-category co-occurrences remain descriptive rather than generalized or causal.Execution, instruction manipulation, privilege/tool abuse, and supply-chain abuse have large injected components; credential access, defense evasion, and discovery have larger wild components. SRC001 vector labels cover 1,466 identities, and the largest co-occurrence is Execution / Code Delivery with Instruction / Goal / Memory Manipulation (1,244).Source: §E.3 · p. 27, §E.4 · p. 27, §E.5 · p. 27

E.6 DERIVED IMPACT TAXONOMY AND INTERSECTION BOUNDARY

  • 2,128 identities receive derived impact labels, but only 1,888 have both impact and attack mappings.The largest derived impacts are unauthorized code execution / system control (489), sensitive data disclosure (423), and credential compromise (297); the intersection pairs are descriptive, not causal.Source: §E.6 · pp. 28–29, Table 25 · p. 29

F.2 PROTOCOL CONSTRUCTION CONTRACTS

  • The frozen protocols enforce different contracts: structural-disjoint splits separate malicious structural families, while Source-Disjoint holds out SRC009, SRC011, and SRC012.Random and Source-Balanced Random require content-identity disjointness; Source-Disjoint excludes eight cross-boundary identities, and overlap is a failure only when it violates a protocol’s declared constraint.Source: §F.1 · p. 29, §F.2 · p. 30, §F.4 · p. 31

F.3 HELD-OUT SOURCE AND LABEL COMPOSITION

  • Source-Disjoint tests 1,384 units with 839 malicious and 545 benign, but source-specific label skew prevents interpreting its gap as a pure source-identity effect.SRC009 contributes only malicious test units, SRC011 is overwhelmingly benign, and SRC012 is balanced; balancing class prevalence does not remove source-linked differences in provenance, construction, or documentation.Source: §F.3 · p. 30, Table 27 · p. 30

G.1 DETECTOR INPUT AND FEATURE FITTING

  • Detector inputs use each Skill’s inert primary instruction document, excluding provenance and structural-family fields, helper code, and runtime artifacts.Text vectorizers and the format-only scaler are fit on training partitions only; the same seed-42 configurations are reused across four protocols without split-specific hyperparameter changes.Source: §G.1 · pp. 31–32

G.2 PRIMARY CLASSICAL ESTIMATORS

  • The classical text baselines and format-only diagnostic produce identical primary scores across seeds 42, 43, and 44, with standard deviation 0.Split assignments remain frozen, and this deterministic-seed check does not assess resampling uncertainty across alternative dataset constructions; class weighting is used for the word logistic regression and both LinearSVC baselines.Source: §G.2 · p. 32

G.3 FORMAT-ONLY SANITY BASELINE

  • The format-only sanity baseline uses five standardized document-format counts, receives no lexical tokens, and is excluded from competitive detector claims.Its inputs are character length, line count, Markdown heading count, fenced-code-block count, and URL count; it tests whether coarse formatting alone can explain text-model performance.Source: §G.3 · p. 33

G.4 METRICS AND DECISION SCORES

  • Macro-F1 is primary because the malicious-heavy benchmark can make an always-malicious classifier appear strong on malicious-class F1 while flagging every benign Skill.Benign FPR and Macro-F1 expose this failure mode; continuous decision scores support AUROC/AUPRC, with no probability calibration or validation-threshold tuning for primary comparisons.Source: §G.4 · p. 33

G.5 AUXILIARY MINILM EMBEDDING BASELINE

  • The auxiliary MiniLM baseline uses a locally cached, offline-only encoder with frozen 384-dimensional mean-pooled, L2-normalized embeddings and class-weighted logistic regression.It is not fine-tuned on MaliciousSkillBench; only seed 42 is reported for its Random, Malicious-Structural-Disjoint, and Source-Disjoint runs.Source: §G.5 · p. 33

G.6 SOFTWARE-VISIBLE REPRODUCIBILITY ARTIFACTS

  • The frozen package retains split CSVs, per-seed metrics, serialized estimators, MiniLM embeddings, and a methodology manifest; scikit-learn 1.7.1 is the safest compatibility target.The exact host Python, operating system, and hardware are unrecorded; a public release should add an environment lockfile or container specification.Source: §G.6 · pp. 33–34

H.1 CLASS-BALANCED SUBSETS

  • Exact and normalized overlap remain zero after class balancing; Malicious-Structural-Disjoint retains zero structural-family overlap, and Source-Disjoint retains zero source overlap.Balancing deterministically downsamples malicious units within frozen partitions to 2,235 malicious and 2,235 benign units per protocol, without moving units or changing labels.Source: §H.1 · p. 34, Table 33 · p. 35

H.2 AUDITED SCAFFOLD SANITIZATION

  • 1,771 benchmark documents trigger scaffold-sanitization rules, affecting 1,977 unique lines; a separate post hoc audit finds zero explicit source, benchmark, or dataset-name markers among top linear features.The fixed sanitizer targets audited construction wrappers, preserves attack-semantic content, and records hashes, lengths, and rule IDs so transformed documents can be regenerated.Source: §H.2 · p. 35

H.3 CONTROL RESULTS

Across the frozen protocols, source-disjoint evaluation remains substantially worse than random evaluation, even after balancing or scaffold sanitization. The full detector results show that this gap is dominated by benign false alarms, while source-level transfer varies widely and the controls do not establish a single cause.

H.3 CONTROL RESULTS

  • The word-SVM’s Source-Disjoint Macro-F1 rises from 0.665 to 0.710 with balancing, but benign FPR remains 0.433 and the source gap persists.Scaffold sanitization leaves Sanitized Random Macro-F1 at 0.860–0.914 and Sanitized Source-Disjoint at 0.643–0.662, with held-out benign FPR of 0.624–0.640.Source: Table 35 · p. 36, §H.3 · p. 36

H.4 WHAT THE CONTROLS ESTABLISH—AND WHAT THEY DO NOT

  • The controls rule out class imbalance or audited construction markers as sole explanations, but do not establish a unique cause for the source gap.Source remains entangled with provenance, attack construction, benign population, labeling policy, and documentation style; sanitization removes seven fixed wrapper classes while preserving attack semantics and executable-looking text.Source: §H.4 · p. 36

I.1 FULL PRIMARY DETECTOR METRICS

  • On Source-Disjoint, Macro-F1 is 0.653–0.665, balanced accuracy 0.657–0.666, and MCC 0.409–0.427, while malicious F1 remains 0.804–0.810.Random AUROC is 0.965–0.986, compared with 0.738–0.821 on Source-Disjoint; AUPRC also decreases, though it remains numerically high because malicious Skills remain prevalent in the held-out test.Source: Table 36 · p. 37, §I.1 · p. 37

I FULL LEARNED-DETECTOR RESULTS AND ERROR ANALYSIS

I.2 CONFUSION MATRICES AND DIAGNOSTIC REFERENCES

  • 340 benign false positives versus 37 malicious false negatives for the word-SVM on Source-Disjoint show that held-out-source errors are dominated by false alarms.The always-malicious baseline has Random malicious F1 of 0.870 but Macro-F1 of 0.435 and benign FPR of 1.0; format-only Random Macro-F1 is 0.486.Source: Table 37 · p. 37, §I.2 · pp. 37–38, Table 38 · p. 38

I.3 AUXILIARY MINILM EMBEDDING BASELINE

  • MiniLM reaches Macro-F1 0.554 on Source-Disjoint, with malicious recall 0.895 and benign FPR 0.749; the frozen encoder is used only as a compact representation baseline.It does not remove the cross-source error regime observed with sparse text features, and the authors make no state-of-the-art neural comparison.Source: Table 39 · p. 38, §I.3 · p. 38

I.4 LEAVE-ONE-SOURCE-OUT TRANSFER DIAGNOSTIC

  • Across four eligible held-out sources, word-SVM Macro-F1 ranges from 0.329 on SRC011 to 0.733 on SRC012, while benign FPR ranges from 0.500 to 0.933.Label mixtures differ substantially, and many Core-contributing sources are single-label; the spread is a source-specific transfer diagnostic, not an estimate of a universal unseen-source distribution.Source: §I.4 · pp. 38–39, Table 40 · p. 39

I.5 SOURCE-CONDITIONED ERROR CONCENTRATION

Across the frozen static benchmark, scanners occupy distinct operating regimes, while held-out-source errors and attack-stratified recall reveal concentrated and uneven coverage. The appendices bound these findings with source-sensitivity caveats, representative identity cases, and a release contract for reproducibility.

I.5 SOURCE-CONDITIONED ERROR CONCENTRATION

  • 293 of 340 benign false positives come from SRC011 and 47 from SRC012; overall benign FPR is 0.624 while malicious recall is 802/839 = 0.956.The analysis reports frozen counts and source concentration without assigning semantic error-cause labels.Source: Table 41 · p. 39, §I.5 · p. 39

I.6 LINEAR-FEATURE INSPECTION AND INTERPRETATION LIMITS; J.5 COMPARABILITY BOUNDARY

  • The feature audit finds zero explicit source or dataset markers in exported top features, but does not establish source invariance; wrapper removal leaves the Source-Disjoint gap.The bounded conclusion is source sensitivity, not a single causal mechanism; scanner results are not interchangeable with runtime, full-package, cloud, or LLM-backed evaluations.Source: §I.6 · p. 40, §J.5 · p. 43

J.1 COMMON ARTIFACT TRACK AND FROZEN VERDICT CONTRACTS

  • Cisco and SkillSpector have very low benign FPR but only 6.4% and 3.3% malicious recall, while SkillFortify reaches 56.6% recall with a 45.1% benign FPR.These are fixed scanner configurations tested on isolated static Skill text, not full-package or runtime behavior.Source: Table 43 · p. 40, §J.1 · p. 40, §J.5 · p. 43

J.2 PROTOCOL BEHAVIOR AND OPERATING-POINT SENSITIVITY

  • On Source-Disjoint, all three scanners have 100% coverage, but Cisco and SkillSpector miss nearly all held-out malicious Skills while SkillFortify incurs larger benign FPR.Prespecified secondary gates shift the false-positive/false-negative balance but do not produce a uniformly strong operating point.Source: §J.2 · pp. 40–41, Table 44 · p. 41, Table 45 · p. 41

J.3 SOURCE OVERLAP, TECHNICAL ABSTENTIONS, AND UNCERTAINTY

  • Excluding SRC012 changes SkillFortify Source-Disjoint Macro-F1 from 0.349 to 0.254, malicious recall from 25.3% to 16.7%, and benign FPR from 49.9% to 59.8%.Three technical abstentions occur in the Malicious-Structural-Disjoint test and none in Source-Disjoint; Cisco results also use a local compatibility configuration that limits exact replication.Source: Table 46 · p. 41, §J.3 · pp. 41–42, Table 47 · p. 42

J.4 THREAT-STRATIFIED MALICIOUS RECALL

  • Among 4,983 attack-mapped malicious identities, SkillFortify has higher recall than the other primary scanners across mapped categories, with 81.5% for Credential Access and 25.8% for Integrity/Output Manipulation.Categories are unevenly distributed across sources, so this analysis describes coverage gaps rather than source-adjusted scanner rankings.Source: §J.4 · pp. 42–43, Figure 6 · p. 43

K REPRESENTATIVE EXAMPLES

  • Structural families capture nonduplicate template reuse, while cross-label conflict groups are excluded rather than assigned one unsupported static-artifact label.A benign Skill is predicted correctly under Random but malicious under Source-Disjoint; this illustrates source-conditioned sensitivity without identifying its cause.Source: §K.2 · p. 44, Table 50 · p. 45, §K.3 · p. 45

L BENCHMARK RELEASE, SAFETY, AND GOVERNANCE

  • Every identity has a public readable representation, but five malicious texts are sanitized and package redistribution remains source-governed; the frozen snapshot is maintained separately from future releases.The benchmark is intended for defensive static research, not payload execution or wider-ecosystem prevalence estimates; acquired Skill contents are handled as untrusted data and not executed.Source: §L.1 · p. 46, L BENCHMARK RELEASE, SAFETY, AND… · p. 46, §L.3 · p. 47

Related papers

Loading 2608.19901v1…