PaperlayerView exact revision on arXiv

Source-linked AI summary

SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs

Haoran Ou, Gelei Deng, Xuanye Zhang, Wenbo Guo, Tianwei Zhang, Kwok-Yan Lam

arXiv:2609.36879v1cs.CRcs.AI

TL;DR

Compact LLMs struggle to identify malicious behaviors hidden in complex Skill packages, despite the need for auditing in privacy-sensitive and resource-constrained settings. SKILLLITE structures observed behaviors with functional context for compact-LLM risk assessment. It improves detection across compact-model backbones, generalizes to behaviorally confirmed in-the-wild threats, and maintains practical inference efficiency.

  • Problem

    Compact LLMs struggle to identify malicious behaviors hidden in complex Skill packages, while effective auditing in privacy-sensitive and resource-constrained settings remains underexplored.

    Source: §1 · pp. 1, 3

  • Method

    SKILLLITE extracts security-relevant behaviors, infers Skill functionality, grounds findings in source and functional context, and uses a compact LLM for risk adjudication.

    Source: §1 · p. 2, §3.2 · p. 5, §3.4 · p. 6

  • Results

    SKILLLITE consistently improves malicious-Skill detection across benchmarks and compact LLM backbones, generalizes to behaviorally confirmed in-the-wild threats, and maintains practical inference efficiency.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SKILLLITE supports practical malicious-Skill auditing with compact, locally deployable LLMs while maintaining practical inference efficiency.

    Source: §5 · p. 9

    The audit covers pre-execution behaviors within Skill packages, not runtime-only behaviors or attacks on the underlying agent infrastructure, models, or execution environment.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

As LLM-based agents perform increasingly complex tasks, Agent Skills have emerged as a flexible mechanism for extending their capabilities. An Agent Skill packages task-specific instructions with executable components and auxiliary resources to provide specialized functionalities. However, the growing adoption of third-party Skills introduces a new supply-chain attack surface. Malicious Skills can embed harmful behaviors that abuse agent privileges and compromise the agent execution environment or accessible resources. Although recent LLM-based malicious Skill auditing approaches have achieved promising performance, they often rely on capable commercial LLMs. How to achieve effective auditing with compact, locally deployable LLMs in security-sensitive and resource-constrained settings remains largely unexplored. Our investigation reveals that compact LLMs struggle to identify malicious behaviors hidden in complex Skill packages. This difficulty arises from both the implicit nature of such behaviors and the limited reasoning capacity of compact LLMs. To address these challenges, we propose SKILLLITE, an evidence-guided agentic framework for malicious Skill detection. SKILLLITE effectively extracts security-relevant behaviors and infers the intended functionality from complex Skill packages. It then employs a compact LLM to assess the maliciousness of the Skill based on the observed behaviors and their functional context. Experiments show that SKILLLITE improves malicious Skill detection across different compact LLM backbones and outperforms existing representative auditing baselines. Its effectiveness generalizes to behaviorally confirmed in-the-wild malicious Skills. Meanwhile, SKILLLITE maintains a low inference latency, supporting its practical deployment.

1 INTRODUCTION

Third-party Skills expand agents’ capabilities but create a supply-chain attack surface, while effective auditing with compact, locally deployable LLMs remains underexplored. SKILLLITE structures evidence and functional context for compact-LLM risk assessment, achieving strong benchmark results and practical latency.

1 INTRODUCTION

  • Third-party Skills can hide malicious behavior among legitimate operations, potentially abusing agent privileges to access sensitive resources, manipulate execution, or communicate externally.Such threats motivate auditing before Skills are deployed or executed.Source: §1 · p. 1
  • Prior LLM-based auditing approaches often rely on commercial or large models, while compact LLM auditing for complex packages remains largely unexplored.External-service requirements, local hosting overhead, privacy needs, and limited compute motivate compact local auditing.Source: §1 · p. 2
  • SKILLLITE coordinates behavior discovery, intent analysis, evidence synthesis, and risk adjudication, grounding observed behaviors in the Skill’s intended functionality.Analyzers inspect instructions, code, configurations, and auxiliary resources; compact LLM roles infer intent and assess structured evidence.Source: §1 · p. 2
  • SKILLLITE achieves the best overall F1-scores on MalSkillBench (0.905), SkillTrustBench (0.966), and behaviorally confirmed in-the-wild MASB (0.807).The paper reports that these results outperform representative baselines.Source: §1 · pp. 2–3
  • Inference takes 27.4 seconds per Skill on MalSkillBench, 28.5 on SkillTrustBench, and 19.4 on MASB; SKILLLITE is 3.5× and 4.9× faster than the strongest competitor on MalSkillBench and MASB.Across 15 compact-model–benchmark settings, it improves F1 by up to 73.6 percentage points; removing key components drops F1 as low as 0.451.Source: §1 · pp. 2–3

2 RELATED WORK

Agent Skills package instructions and optional resources to extend agent capabilities, and third-party distribution broadens the supply chain. Malicious behaviors can be embedded across artifacts, while existing detection approaches face contextual and deployment challenges that motivate study of compact models.

2.1 AGENT SKILLS

  • Skills package task-specific instructions and optional scripts, references, templates, or other resources, extending agent capabilities without retraining the underlying model.Their adoption spans coding, document processing, data analysis, web automation, and domain-specific execution.Source: §2.1 · p. 3

2.2 MALICIOUS SKILLS

  • Third-party distribution expands the agent software supply chain, and malicious Skills can hide across instructions and implementation artifacts to steal credentials or data or execute commands without authorization.Reported incidents include over 800 malicious skills—roughly 20% of ClawHub’s registry—and a separate deceptive installation campaign.Source: §2.1 · p. 3, §2.2 · pp. 3–4

2.3 MALICIOUS SKILLS DETECTION

  • Rule-based scanners lack context to distinguish malicious behavior from behavior required by a Skill, while LLM-based detectors often rely on powerful commercial or large models.Compact models may suit limited-compute or privacy-constrained settings, but their auditing potential remains largely unexplored.Source: §2.3 · p. 4

3 METHODOLOGY

SKILLLITE audits packages before execution through complementary stages that extract security-relevant behaviors, infer intended functionality, ground findings in context, and use a compact LLM to adjudicate risk. The framework distinguishes observed behavior from maliciousness and produces an evidence-grounded rationale.

3.1 OVERVIEW

  • Given a complete Skill package, SKILLLITE extracts security-relevant behaviors across artifacts; runtime-only behavior and attacks on the underlying agent infrastructure, models, or execution environment are outside its scope.The threat model covers third-party Skills that may conceal malicious logic in instructions, code, configurations, or auxiliary resources.Source: §3.1 · pp. 4–5

3.2 SECURITY EVIDENCE EXTRACTION

  • Deterministic analyzers cover general security patterns, code, concealed payloads, and agent-control signals, recording each finding with its source artifact and supporting evidence.The behavior classes include sensitive-resource access, external communication, system execution, persistence, concealed behavior, and agent-control manipulation; these behaviors alone do not establish maliciousness.Source: §3.2 · p. 5

3.3 INTENT ANALYSIS

  • The compact-LLM Intent Analyst derives a functional goal and expected capabilities from the Skill specification, which provide a reference for interpreting findings.An unexpected behavior is not necessarily malicious and is examined further during risk adjudication.Source: §3.3 · pp. 5–6

3.4 EVIDENCE SYNTHESIS

  • Evidence synthesis grounds each finding in its originating artifact and organizes grounded observations with functional and policy context into a unified report.The report combines evidence of what behaviors occur and where with risk context for assessing them.Source: §3.4 · p. 6

3.5 RISK ADJUDICATION

  • The Risk Adjudicator assesses functional necessity, credible benign counterevidence, and security risk, considering both individual behaviors and harmful combinations.For critical agentic risks, generic functional justification is insufficient; the compact LLM returns a maliciousness judgment with a traceable, evidence-grounded rationale.Source: §3.5 · p. 6

4 EXPERIMENTS

Across five experimental subsections, SKILLLITE is evaluated on benchmark detection, in-the-wild generalization, compact-model backbones, efficiency, and component ablations. The reported results favor evidence-guided auditing for detection effectiveness while retaining practical inference efficiency.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers three benchmarks, five representative scanners, binary-classification metrics, and average processing time per Skill.MaliciousAgentSkillsBench includes behaviorally confirmed in-the-wild malicious Skills; LLM-assisted baselines use the same underlying model as SKILLLITE whenever supported.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • SKILLLITE achieves the strongest overall detection performance on MalSkillBench and SkillTrustBench while maintaining strong precision and recall.It also offers low latency comparable to several LLM-based baselines, though purely static scanners are faster and weaker in detection.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 GENERALIZATION TO IN-THE-WILD MALICIOUS SKILLS

  • SKILLLITE achieves the best overall accuracy and F1-score on MaliciousAgentSkillsBench while maintaining a low false-positive rate.The remaining errors are mainly false negatives; the authors note that static auditing cannot observe risks that emerge only at runtime.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 ENHANCING COMPACT LLMS

  • 9.2–73.6 percentage-point F1 gains over direct zero-shot auditing occur across all 15 model–benchmark configurations.SKILLLITE adds inference latency, but runtime remains practical across evaluated backbones; on MalSkillBench, DeepSeek-R1:8B improves from 59.2% to 80.5% F1 with 11.5 seconds per Skill.Source: §4.4 · pp. 8–9, Figure 3 · p. 8

4.5 ABLATION STUDY

  • Removing Security Evidence Extraction, Intent Analysis, or Evidence Synthesis degrades detection performance on SkillTrustBench.Extraction’s largest impact is on malicious-skill recall; without intent analysis, the model misses most malicious Skills, while removing source-grounded context makes security patterns harder to interpret.Source: §4.5 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion presents SKILLLITE as evidence-guided auditing for compact, locally deployable LLMs, reporting improved detection across benchmarks and backbones, in-the-wild generalization, and practical efficiency. The appendix notes that extracted analyzer signals serve as evidence rather than directly deciding maliciousness.

5 CONCLUSION

  • Across multiple benchmarks and compact LLM backbones, SKILLLITE improves detection, generalizes to behaviorally confirmed in-the-wild threats, and maintains practical inference efficiency.The framework externalizes security-evidence discovery and grounding while focusing the compact LLM on intent-conditioned risk adjudication.Source: §5 · p. 9
  • The experiments were conducted in controlled research environments and did not target real users or production systems.The authors state they did not deploy or intentionally distribute malicious Skills to public ecosystems or attack third-party systems.Source: ETHICS STATEMENT · p. 10

A.1 SECURITY EVIDENCE EXTRACTION DETAILS

  • Deterministic analyzers expose security-relevant behaviors across heterogeneous Skill artifacts, but their signals do not directly determine the final maliciousness judgment.Table 4 summarizes analyzer categories and representative security signals.Source: §A.1 · p. 12

A.2 DATASET DETAILS

The dataset details describe three benchmarks with different construction and coverage: diverse attack combinations, complete real-world Skill packages, and behaviorally confirmed in-the-wild samples.

A.2 DATASET DETAILS

  • MalSkillBench contains 3,944 malicious and 4,000 benign Skills, covering code injection, prompt injection, and mixed attacks across 15 malicious behaviors.Its construction spans 108 combinations of attack vectors, behaviors, and insertion strategies.Source: §A.2 · p. 12
  • SkillTrustBench is built from more than 62,000 real-world Skills and covers nine security categories across complete Skill packages.For binary evaluation, the 1,014 suspicious samples are excluded, leaving 2,863 malicious and 1,643 benign Skills.Source: §A.2 · p. 13
  • MaliciousAgentSkillsBench contains 157 behaviorally confirmed malicious Skills from public registries and 299 verified benign Skills for binary evaluation.Malicious samples were screened through static analysis and validated by behavioral execution in isolated environments.Source: §A.2 · p. 13

A.3 BASELINE DETAILS

The baseline details cover static and LLM-assisted scanners evaluated on complete Skill packages using official implementations and recommended configurations when applicable. SkillSpector’s LLM configuration required a lightweight model substitution to complete evaluation reliably.

A.3 BASELINE DETAILS

  • SkillSpector’s static and LLM configurations are evaluated separately, with GPT-4.1-nano replacing Gemma4:e4b in the LLM configuration after high timeout-related failure rates.The original SkillSpector workflow is unchanged.Source: §A.3 · p. 13
  • Cisco Skill Scanner is evaluated in static and LLM configurations, with the LLM version using SKILLLITE’s underlying compact LLM.The evaluation also includes Tencent AI-Infra-Guard’s Skill-Scan component.Source: §A.3 · p. 13
  • SkillWard and Skill-Vetter use their official implementations, with UNSAFE and warning verdicts mapped to the malicious class, respectively.Skill-Vetter uses the same underlying compact LLM as SKILLLITE.Source: §A.3 · p. 14

A.4 IMPLEMENTATION AND EXPERIMENTAL SETUP

The experimental setup uses compact LLMs deployed locally, and the detailed analysis examines attack categories and Skill package characteristics.

A.4 IMPLEMENTATION AND EXPERIMENTAL SETUP

  • Gemma4:e4b is the default backbone; the study also evaluates Qwen3.5:9B, DeepSeek-R1:8B, and Mixtral-8x7B.DeepSeek-R1:8B is tested with thinking both enabled and disabled.Source: §A.4 · p. 14
  • Open-source models run locally through Ollama without fine-tuning, with each model instance on one GPU in a Linux server with NVIDIA RTX A6000 GPUs.Each GPU has 48 GB of memory.Source: §A.4 · p. 14

B DETAILED ANALYSIS

  • The detailed analysis covers attack categories and package characteristics.Package characteristics include package size and programming-language characteristics.Source: B DETAILED ANALYSIS · p. 14

B.1 PERFORMANCE ACROSS ATTACK CATEGORIES

SKILLLITE shows strong recall across the three benchmarks’ fine-grained malicious-behavior categories, with some challenging cases and a trade-off between sensitivity and false alarms.

B.1 PERFORMANCE ACROSS ATTACK CATEGORIES

  • SkillTrustBench and MASB categories are multi-label and non-exclusive, and recall is calculated only over malicious samples.MASB includes fine-grained vulnerability patterns derived from audit annotations.Source: §B.1 · pp. 14–15
  • On MalSkillBench, recall reaches 0.993 on B6 (Reverse Shell) and 0.982 on B8 (Resource Abuse), while B10 (Role Hijack) and B14 (Goal Hijacking) remain challenging.Some baselines achieve higher recall on the challenging categories but incur substantially higher false-positive rates on the complete benchmark.Source: §B.1 · p. 14
  • On SkillTrustBench, recall reaches 1.000 on T06 (Persistence) and 0.984 on T07 (Tool Hijacking), while SKILLLITE maintains high recall across diverse security categories.The categories are non-exclusive; baselines with high recall across more categories tend to have considerably higher false-positive rates.Source: §B.1 · p. 14
  • On MalSkillBench, SKILLLITE achieves a better overall balance between category-level detection and false alarms than high-sensitivity baselines, resulting in higher overall F1.Figure 4 compares category-level recall across detection methods; its categories include only malicious samples.Source: §B.1 · p. 14, Figure 4 · p. 15
  • On MASB, recall exceeds 0.78 on SC2, E1, E2, and P4, while P1, E3, E4, and SC3 are more challenging for SKILLLITE.Skill-Vetter and AIG achieve high recall across many patterns but have substantially higher false-positive rates; other scanners miss more malicious Skills.Source: §B.1 · p. 15

B.2 PERFORMANCE ACROSS PACKAGE CHARACTERISTICS

SKILLLITE performs strongly across several package-size and programming-language groups, while results vary on MASB and packages without detected code are more challenging.

B.2 PERFORMANCE ACROSS PACKAGE CHARACTERISTICS

  • On MalSkillBench and SkillTrustBench, SKILLLITE outperforms all baselines on both single- and multi-language packages and remains stable across those groups.Packages without detected code are more challenging, suggesting fewer explicit implementation signals for security analysis.Source: Table 6 · p. 16, §B.2 · p. 16
  • Across most language groups in MalSkillBench and SkillTrustBench, SKILLLITE achieves the highest F1; on MASB, it achieves the highest F1 for Python and Shell, with greater variation across languages.The evaluated groups cover languages including Python, Shell, JavaScript, and YAML; groups may overlap when a Skill contains multiple languages.Source: Table 7 · p. 16, §B.2 · pp. 16–17
  • On SkillTrustBench, SKILLLITE outperforms all baselines across three package-size groups; on MalSkillBench, it leads in the two dominant groups.On MASB, performance is strong on single-file packages and competitive for packages with two to five files, but decreases for larger packages.Source: §B.2 · p. 16

C COMPLETE CROSS-MODEL RESULTS

Across the evaluated compact LLM backbones, direct zero-shot auditing has high precision but lower recall, while SKILLLITE improves detection with additional but practical inference latency. Its prompts guide intent analysis and evidence-based risk adjudication.

C COMPLETE CROSS-MODEL RESULTS

  • Across three benchmarks, direct zero-shot auditing has high precision but lower recall, and SKILLLITE improves detection across evaluated compact LLM backbones.Improvement magnitude varies with backbone capability; the added inference latency leaves overall runtime practical.Source: C COMPLETE CROSS-MODEL RESULTS · p. 17

D PROMPT TEMPLATES

  • The prompt templates specify two semantic reasoning tasks: intent analysis and evidence-guided risk adjudication.They provide the core instructions and output structures used in the implementation.Source: D PROMPT TEMPLATES · p. 17

Intent Analysis

  • Intent analysis asks the compact LLM to infer a Skill’s declared purpose and the capabilities necessary and proportionate to that purpose.The prompt takes the Skill name and specification as input.Source: Intent Analysis · p. 18
  • Risk adjudication reviews suspicious evidence and its source, checks for benign counterevidence, and assesses whether the behavior forms an abuse chain.The policy treats security-sensitive capability alone as insufficient for maliciousness and calls for direct, risk-specific benign counterevidence for critical agentic risks.Source: Intent Analysis · p. 18
Loading 2609.36879v1…