SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

Zihan Guo; Zhiyu Chen; Xiaohang Nie; Jianghao Lin; Yuanjian Zhou; Weinan Zhang

Source-linked AI summary

SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

Zihan Guo, Zhiyu Chen, Xiaohang Nie, Jianghao Lin, Yuanjian Zhou, Weinan Zhang

arXiv:2603.21019v1Mar 22, 2026cs.CRcs.SE

TL;DR

Open skill marketplaces are vulnerable to semantic-behavioral inconsistency and risks that emerge when individually benign skills are composed. SkillProbe uses modular multi-agent auditing across admission, alignment, and composition phases, and its evaluation finds popularity is not a reliable security proxy while high-risk skills form a giant connected component. The framework’s scope remains limited by obfuscated or black-box code, predefined zero-day policies, and computational overhead at very large influxes.

Problem
Open agent skill marketplaces lack reliable protection against documentation–behavior mismatches and inter-skill risks that can remain invisible to atomic auditing.
Method
SkillProbe packages admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation into standardized skills coordinated by collaborating agents.
Results
Evaluation shows that high-download skills harbor latent risks comparable to long-tail modules, while high-risk skills form a single giant connected risk component.
Takeaways & Limitations
The findings support proactive, intent-aware pre-distribution auditing because traditional atomic-level auditing cannot perceive cascaded risks in multi-skill orchestration.
Takeaways & Limitations
Detection precision remains constrained for highly obfuscated code or black-box APIs, predefined policies may miss novel zero-day patterns, and auditing faces overhead during influxes exceeding 10,000 skills.

Abstract

from arXiv · show

With the rapid evolution of Large Language Model (LLM) agent ecosystems, centralized skill marketplaces have emerged as pivotal infrastructure for augmenting agent capabilities. However, these marketplaces face unprecedented security challenges, primarily stemming from semantic-behavioral inconsistency and inter-skill combinatorial risks, where individually benign skills induce malicious behaviors during collaborative invocation. To address these vulnerabilities, we propose SkillProbe, a multi-stage security auditing framework driven by multi-agent collaboration. SkillProbe introduces a "Skills-for-Skills" design paradigm, encapsulating auditing processes into standardized skill modules to drive specialized agents through a rigorous pipeline, including admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation. We conducted a large-scale evaluation using 8 mainstream LLM series across 2,500 real-world skills from ClawHub. Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy for security quality, as over 90% of high-popularity skills failed to pass rigorous auditing. Crucially, we discovered that high-risk skills form a single giant connected component within the risk-link dimension, demonstrating that cascaded risks are systemic rather than isolated occurrences. We hope that SkillProbe will inspire researchers to provide a scalable governance infrastructure for constructing a trustworthy Agentic Web. SkillProbe is accessible for public experience at skillhub.holosai.io.

1 Introduction

LLM agent skill marketplaces face risks because natural-language descriptions mediate execution, enabling semantic-behavioral inconsistencies and attacks that emerge through skill collaboration. SkillProbe addresses these gaps with a modular, multi-agent auditing pipeline spanning admission filtering, alignment detection, and combinatorial risk analysis.

Security Challenges: Inter-skill collaboration creates combinatorial risks when individually acceptable skills interact through untrusted data flows and multi-step sequences.These risks arise beyond isolated skill auditing.
Motivation: Natural-language descriptions guide agent invocation while underlying implementations operate on external systems, creating a trust propagation chain vulnerable to semantically compliant but harmful skills.The risk grows as agent marketplaces scale.
Security Challenges: Semantic-behavioral inconsistency lets implementations conceal shadow functions or over-privileged behavior because agents and users anchor reasoning in the semantic layer.The execution-layer deviation can remain imperceptible during invocation.
Approach: SkillProbe encapsulates auditing phases as standardized skill modules in a three-phase pipeline covering admission filtering, semantic consistency detection, and combinatorial risk analysis.The framework evaluates both static features and dynamic behaviors.
Contributions: The framework automates compliance filtering, semantic analysis, and cross-skill link-attack detection while supporting modular third-party tools and customized rule sets.Its multi-dimensional risk model represents capabilities through alignment classes and a standardized label graph.

2 Related Work

Prior work documents vulnerabilities in open skill ecosystems, description-driven attacks, documentation–code inconsistency, and multi-tool attack chains. SkillProbe builds on this literature by treating documentation as a possible attack surface and addressing risks that atomic auditing misses during composition.

Marketplace Security: Open skill marketplaces expose a large security surface: 26.1% of 31,132 audited skills contained at least one vulnerability, and script-combining skills were 2.12× more likely to be vulnerable.A follow-up study also identified Data Thieves and Agent Hijackers across 98,380 skills.
Description-Driven Threats: Description-driven attacks can redirect agent behavior through adversarial external data or misleading tool descriptors without modifying model weights.These attacks exploit LLM reliance on natural-language context.
Documentation Alignment: Earlier documentation-consistency methods generally assume good-faith documentation and seek to preserve code–documentation consistency.Their threat model differs from settings where documentation itself is malicious.
Documentation Alignment: SkillProbe adopts an inverse threat model in which benign-looking descriptions can induce unintended invocations while code exfiltrates data or escalates privileges.This behavior is termed semantic-behavioral inconsistency.
Compositional Risks: Atomic auditing is insufficient for composed chains because individually harmless tools can be orchestrated into attacks exceeding 90% success rates.Prompt payloads can traverse multiple tool-call boundaries through the agent’s reasoning chain.
Multi-Agent Foundations: Multi-agent frameworks provide orchestration primitives such as structured message passing, delegated tool invocation, and stateful workflow graphs for collaborative auditing pipelines.These systems support role specialization and parallel task decomposition.

3.1 System Overview

SkillProbe is a hierarchical, modular multi-agent system organized into five functional layers and driven by a Security Auditor. Its standardized skill packages and adaptive execution modes support both rapid screening and comprehensive, high-throughput assessment.

Architecture: SkillProbe comprises Input, Orchestration, Skill, Output, and Infrastructure layers for ingestion, collaboration, auditing, reporting, and foundational storage and computation.The Skill Layer contains the core audit modules.
Architecture: The Security Auditor dispatches tasks, maintains audit state, and schedules phase-wise execution across low-level executors.It acts as the central orchestrator.
Audit Modules: Auditing modules include Gatekeeper for admission filtering, Alignment Detector for semantic consistency, and Flow Simulator for combinatorial risk analysis.The Output Layer produces structured reports and a multi-dimensional scorecard.
Modularity: Each auditing phase is packaged with natural-language specifications, workflow configurations, system prompts, and scripts, representing the process as an agentic DAG.This Specification-as-Code design decouples audit logic from execution topology.
Execution: SkillProbe supports sequential, dynamic-concurrent, and pre-calculated parallel execution modes through reusable atomic capabilities and on-demand tool loading.The execution topology is decoupled from task logic.
Auditing Modes: Quick mode serially runs Gatekeeper and Alignment Detector for preliminary screening, whereas standard mode parallelizes all phases for comprehensive assessment.The modes trade analysis depth and throughput differently.

3.2 Core Auditing Phases

SkillProbe audits skills through admission filtering, declared-versus-implemented capability alignment, and compositional risk simulation. These phases combine autonomous inspection, semantic normalization, deterministic rules, and risk fingerprints to identify both direct and chained threats.

Workflow: The workflow contains three phases: malicious-pattern inspection, capability alignment detection, and composition-risk simulation.The phases are coordinated as a sequential auditing workflow.
Phase 1: Gatekeeper: Gatekeeper evaluates compliance, malicious code patterns, hazardous dependencies, and permission rationality during skill ingestion.It combines parallel autonomous inspection with centralized aggregation and deterministic detection.
Phase 1: Gatekeeper: A skill proceeds only after passing every Gatekeeper dimension; any failed dimension produces a BLOCK status.This creates an admission gate before later analysis.
Phase 2: Alignment Detector: Alignment Detector compares documented capabilities D(s) with implemented capabilities C(s), representing each capability as a resource and access tuple.The detector focuses on intersection and union relationships to expose high-risk deviations.
Phase 2: Alignment Detector: Document and Code Extractors operate in parallel, after which semantic normalization maps capabilities to canonical keys for alignment classification.Code analysis combines pluggable tools, AST analysis, I/O binding analysis, and LLM reasoning.
Phase 2: Alignment Detector: The alignment matrix distinguishes Match, Over-declaration, Under-declaration, and Mixed states.Under-declaration corresponds to shadow functions, while Mixed captures complex deviations.
Phase 3: Flow Simulator: Flow Simulator reduces combinatorial search from O(2^N) brute-force enumeration to O(N × |Rules|) rule matching through risk fingerprint tagging.Risk labels describe output features and downstream input sensitivities.

3.3 Auditing Verdict

SkillProbe evaluates skills across malicious patterns, semantic consistency, and composition safety, then applies a strict one-vote veto to assign the final verdict.

Auditing dimensions: The security scorecard evaluates malicious patterns, semantic consistency, and composition safety as three auditing dimensions.These dimensions correspond respectively to Gatekeeper findings, Alignment Detector findings, and Flow Simulator findings.
Auditing dimensions: Malicious patterns cover active threats including backdoors, data exfiltration, and dependencies with known CVEs.
Auditing dimensions: Semantic consistency identifies shadow functions and over-declarations between documentation and executable logic.
Verdict policy: The final verdict uses a strict one-vote veto policy based on the severity of detected findings.
Verdict levels: REJECTED denotes critical vulnerabilities or severe semantic conflicts, CONDITIONAL denotes low-risk findings requiring manual review, and APPROVED requires all metrics to meet security baselines.

3.4 System Implementation

Holos-SkillHub operationalizes SkillProbe through a backend, web interface, CLI, and LLM integration plugins for end-to-end auditing and report delivery.

Platform components: Holos-SkillHub comprises a high-performance backend, reactive web interface, CLI, and LLM integration plugins.
Platform components: The platform provides a closed-loop service from asynchronous task scheduling to multidimensional security visualization.Its backend uses FastAPI and asyncio, while the frontend uses Vue 3.
Platform components: Embedding models generate 1,536-dimensional semantic vector indexes, while SHA256 fingerprinting supports global deduplication and rapid retrieval.
Governance tools: The CLI manages authentication, batch submission, streaming downloads, and multi-format auditing reports.
Governance tools: The OpenClaw plugin enables asynchronous auditing workflows and structured result retrieval within LLM environments through lightweight scripts.This integration reduces the need for complex host-environment configuration.

4 Evaluation

SkillProbe was evaluated on ClawHub skills using eight LLM series, with analyses spanning cross-model consistency, popularity-ranked auditing, and risk-link structure. Results show substantial model heterogeneity, widespread findings among popular skills, and interconnected high-risk capabilities.

Experimental setup: The evaluation used eight LLM series and a benchmark of 20 most-downloaded skills, producing 160 auditing samples.The broader production scan covered the top 2,500 most-downloaded skills using nex-n1.
Cross-model audit applicability: Audit performance varied substantially: Gemini-Flash averaged 18.5 seconds, while Nex-N1.1 averaged 282.0 seconds.Sonnet-4.6 averaged 228.6 seconds and was especially sensitive to high-complexity scripts.
Cross-model audit applicability: Cross-model heatmaps showed consistent Approved verdicts for several high-profile skills, while nano-banana-pro was Conditional across all eight models.Localized fluctuations for agent-browser and self-improving indicate greater sensitivity in edge cases.
Large-scale empirical audit: Among 2,500 skills, 9.9% were clean, 59.8% passed with minor findings, and 30.4% were Conditional.The conditional proportion remained broadly stable across download tiers, so download rank was not a reliable security proxy.
Large-scale empirical audit: The risk network contained 75,373 edges, with Data Exfiltration and Fact Poisoning accounting for 80% of edges.The high-risk subset was fully interconnected, supporting combinatorial simulation for cascaded cross-skill risks.

5 Discussion

SkillProbe’s discussion identifies a security-governance gap: popularity does not predict safety, semantic inconsistency enables stealth threats, and interconnected risks require combinatorial auditing. The framework reports a highly connected risk network while acknowledging limitations in obfuscated-code analysis, zero-day coverage, and large-scale latency.

Popularity Paradox: Download popularity is not a reliable proxy for security, with high-download skills retaining risk levels comparable to long-tail modules.The discussion frames this mismatch as a structural governance deficit in current marketplaces.
Semantic Gap: Semantic-behavioral inconsistency creates a stealth path because agents select skills from natural-language descriptions but execute their underlying scripts.This discrepancy can hide harmful behavior from both users and agents.
Combinatorial Defense: The giant connected risk component indicates that individual vulnerabilities can propagate through multi-skill chains, motivating simulation of entire execution chains rather than atomic inspection.The Flow Simulator is presented as enabling combinatorial risk prediction under linear complexity.
Limitations: SkillProbe’s detection depth remains constrained for highly obfuscated code or black-box APIs, while predefined policies may miss novel zero-day patterns and massive influxes create latency challenges.The stated scale boundary includes instantaneous influxes exceeding 10,000 skills.

6 Conclusion

SkillProbe is a modular, collaborative multi-agent framework for proactive security auditing of agent skills. Its evaluation finds popularity-independent latent risks and a giant connected risk component, while future work targets obfuscated code and zero-day combinatorial attacks.

Framework: SkillProbe formalizes semantic gaps and risk combinatorial explosion through proactive, intent-aware pre-distribution auditing.Its “Skills-for-Skills” design supports extensibility and integration of third-party security utilities.
Evaluation Results: ClawHub evaluation found high-download skills harbor latent risks comparable to long-tail modules, establishing the reported popularity paradox.The result is presented as systematic empirical evidence from the evaluation.
Evaluation Results: A single giant connected risk component exposes limitations of traditional atomic auditing for perceiving cascaded risks in multi-skill orchestration.The finding underscores multiplicative threat effects within composed skill environments.
Future Work: Future research will examine dynamic sandboxing for obfuscated code and reinforcement learning for discovering zero-day combinatorial attack patterns.These directions address limitations identified for code scrutiny and predefined risk-link policies.