Source-linked AI summary
SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement
Zenghao Duan, Yuxin Tian, Zhiyi Yin, Liang Pang, Jingcheng Deng, Zihao Wei, Shicheng Xu, Yuyao Ge, Xueqi Cheng
TL;DR
Agent skills from open registries are difficult to vet, and static auditing cannot establish whether latent vulnerabilities in non-malicious skills are exploitable. SkillAttack dynamically tests exploitability through vulnerability analysis, parallel attack generation, and feedback-driven refinement without modifying skills. Across 10 LLMs, it substantially outperforms baselines on adversarial and real-world skills, showing that well-intended skills can pose serious risks under realistic interactions.
Problem
Static auditing can miss whether latent vulnerabilities embedded in legitimate, non-malicious skills are exploitable through adversarial prompting without modifying the skill.
Method
SkillAttack uses vulnerability analysis, surface-parallel attack generation, and feedback-driven exploit refinement in a closed-loop red-teaming search.
Results
SkillAttack substantially outperforms baselines across 10 LLMs, achieving ASR of 0.73–0.93 on adversarial skills and up to 0.26 on real-world skills.
Takeaways & Limitations
Even well-intended skills pose serious security risks under realistic interactions, supporting dynamic exploit verification beyond static auditing.
Takeaways & Limitations
The evaluation covers 171 skills from two benchmarks, uses a single judge model, and considers only prompt-level attacks.
Abstract
from arXiv · showhide
LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities, yet the openness of such ecosystems makes skills difficult to thoroughly vet. Existing attacks rely on injecting malicious instructions into skills, making them easily detectable by static auditing. However, non-malicious skills may also harbor latent vulnerabilities that an attacker can exploit solely through adversarial prompting, without modifying the skill itself. We introduce SkillAttack, a red-teaming framework that dynamically verifies skill vulnerability exploitability through adversarial prompting. SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven exploit refinement into a closed-loop search that progressively converges toward successful exploitation. Experiments across 10 LLMs on 71 adversarial and 100 real-world skills show that SkillAttack outperforms all baselines by a wide margin (ASR 0.73--0.93 on adversarial skills, up to 0.26 on real-world skills), revealing that even well-intended skills pose serious security risks under realistic agent interactions.
1 Introduction
Open skill registries make agent skills difficult to vet, while static auditing misses whether latent vulnerabilities in non-malicious skills are exploitable. SkillAttack addresses this gap through iterative adversarial prompting, vulnerability analysis, parallel attack generation, and feedback-driven refinement, achieving strong results across models and skill types.
- Motivation: Open registries make agent skills difficult to vet because they combine executable code, domain knowledge, and instructions from potentially broad contributors.Skills support specialized agent workflows and are increasingly published, versioned, and shared through open ecosystems.
- Motivation: Existing attacks inject malicious instructions into skill files, but conspicuous code or execution patterns make them detectable by static auditing.Non-malicious skills pose a subtler risk because vulnerabilities can be embedded in legitimate functionality.
- Motivation: Static analysis can identify vulnerabilities in non-malicious skills but cannot confirm whether attackers can exploit them through agent interactions.The central question is whether adversarial prompting alone can exploit latent vulnerabilities without modifying the skill.
- Approach: SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven refinement into a closed-loop search without modifying the target skill.The pipeline extracts attacker-controllable inputs and sensitive operations, generates paths across vulnerability candidates, then refines prompts using execution feedback.
- Evaluation: SkillAttack evaluates 10 LLMs on 71 adversarial skills and 100 real-world skills, with ASR of 0.73–0.93 on adversarial skills and up to 0.26 on real-world skills.The evaluation judges attacks from execution trajectories, intermediate artifacts, and final responses, and finds distinct threat profiles by skill type.
2 Related Work
Prior work studies skill ecosystems, vulnerabilities, skill-file attacks, and automated red teaming, but existing skill attacks often depend on manually constructed scenarios or direct modification of skill artifacts.
- Agent Skills and Tool Ecosystems: Research on agent skill ecosystems examines orchestration, third-party marketplaces, and skill generalization across diverse tasks.These ecosystems resemble distributed software supply chains and support complex agent workflows.
- Security Risks of Agent Skills: Large-scale analyses report pervasive vulnerabilities in skill ecosystems, including prompt injection, data exfiltration, and privilege escalation.Related studies also identify malicious skills capable of multi-stage attacks such as data theft and agent hijacking.
- Security Risks of Agent Skills: Prior skill attacks use embedded malicious instructions, benchmarked skill-file injections, or closed-loop generation of poisoned skill packages.These approaches target skill mechanisms through modified or malicious artifacts.
- Security Risks of Agent Skills: Existing skill-attack methods rely on manually constructed attack scenarios with limited scalability or assume attackers can directly modify skill artifacts.This leaves a methodological gap for attacks that exploit unmodified skills through prompting alone.
- Automated Red Teaming for LLMs and Agents: Automated red teaming for LLMs and agents uses gradient optimization, iterative interaction, structured search, and semantic perturbation to evaluate safety.This literature provides methodological context for automated adversarial evaluation beyond skill-specific attacks.
3 SkillAttack
SkillAttack models exploit discovery as a path-search problem over agent skills, using vulnerability analysis, parallel attack generation, and feedback-driven refinement. It evaluates candidate exploits through agent trajectories, execution artifacts, and final responses against predefined unsafe behaviors.
- Overview: SkillAttack treats exploit discovery as iterative path search from an adversarial prompt through the skill interface to unsafe behavior.Each attack path captures the expected execution flow, and refinement seeks a viable exploit without modifying the target skill.
- Problem Formulation: The framework models skills as natural-language instructions paired with executable implementations and auxiliary resources that agents may load.This representation covers both the instruction interface and the implementation used during agent execution.
- Problem Formulation: SkillAttack assumes attackers can craft arbitrary user prompts but cannot modify the skill, system prompt, or runtime environment.The agent produces an execution trajectory consisting of actions and corresponding environment observations.
- Problem Formulation: The objective is to find a prompt that triggers a predefined unsafe behavior, evidenced by execution trajectories, artifacts, or the final response.The risk taxonomy includes data exfiltration, destruction, backdoors, malware or ransomware, denial of service, phishing, manipulation, and poisoning.
- Skill Vulnerability Analysis: Vulnerability analysis extracts attacker-controllable inputs, sensitive operations, and triggering conditions, representing each candidate as structured metadata.The metadata tuple v = (t, I, O, C) supports downstream attack generation across identified vulnerabilities.
- Surface-Parallel Attack Generation: For each vulnerability, parallel generation constructs plausible prompts and attack paths, while feedback refinement uses execution signals to improve unsuccessful attempts for up to B rounds.The loop judges trajectories, artifacts, and final responses; successful prompts are returned as exploits, while unsuccessful trajectories provide structured feedback.
4 Experiment
SkillAttack is evaluated across diverse models, skill sets, baselines, and iterative attack rounds. It substantially outperforms static and direct attacks, while revealing skill-dependent threat profiles and latent exploits that often require feedback-driven refinement.
- Experimental Setup: The evaluation covers 71 adversarial skills, 100 real-world ClawHub skills, and 10 LLMs spanning diverse providers and capability tiers.Adversarial skills include 30 Obvious and 41 Contextual Injection instances.
- Main Results: SkillAttack achieves ASR above 0.73 for every model on Obvious skills and up to 0.93, versus at most 0.43 for SKILL-INJECT and near-zero for Direct Attack.These results show a wide advantage over both baselines on overtly malicious skills.
- Main Results: On Contextual skills, SkillAttack reaches 0.56–0.88 ASR, while real-world skills reach up to 0.26 compared with Direct Attack never exceeding 0.04.SkillAttack therefore extends beyond benchmark-crafted injections to organic skills.
- Main Results: Only about 24% of successes occur within the first two rounds, while roughly 65% first emerge in rounds three or four.Round 3 is consistently the most frequent breakthrough point, and Hot100 concentrates 70% of successes in rounds three and four.
- Threat Analysis: Threat profiles vary by skill type: Obvious is fragmented, Contextual is manipulation-dominated, and Hot100 concentrates on operational threats.Data Exfiltration and Malware / Ransomware together exceed 70% of Hot100 successes.
- Case Study: A job-posting case study shows feedback overcoming tool avoidance and hallucinated execution to expose a hardcoded credential without modifying legitimate skill code.The first two rounds failed; subsequent feedback produced tool-call attempts and credential exposure classified as attack_success.
5 Conclusion
SkillAttack dynamically tests whether agent skills can be exploited through adversarial prompting alone, using closed-loop refinement across vulnerability analysis, parallel attack generation, and feedback. Across adversarial and real-world skills, it outperforms baselines, with iterative exploits and skill-specific threat profiles revealing risks that static auditing can miss.
- Conclusion: SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven refinement into a closed-loop search.The framework probes exploitability without modifying the target skill.
- Conclusion: Across 10 LLMs, SkillAttack substantially outperforms all baselines on adversarial and real-world skills.The evaluation also finds that most exploits require iterative refinement and that threat profiles differ by skill type.
- Conclusion: Even well-intended skills pose serious risks under realistic interactions, motivating dynamic exploit verification beyond static auditing.The conclusion emphasizes latent vulnerabilities that adversarial prompting can expose without skill modification.
6 Limitations
The evaluation has several scope and reliability limitations, including reliance on one judge model, prompt-level attacks only, limited skill coverage, and no proposed defenses.
- A single judge model was used, so multiple judges or human annotation would strengthen evaluation reliability.
- The framework covers prompt-level attacks but not multi-agent collusion or environment-level interventions.
- The 171 evaluated skills span two benchmarks and represent only a fraction of real-world ecosystems.
- The framework identifies vulnerabilities but does not propose defenses; input sanitization and runtime monitoring remain future work.