PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

Yuting Ning, Zhehao Zhang, Yash Kumar Lal, Boyu Gou, Junyi Li, Weitong Ruan, Chentao Ye, Rahul Gupta, Diyi Yang, Yu Su, Huan Sun

arXiv:2606.02540v1cs.CL

TL;DR

Existing skill-attack benchmarks largely miss persistent cross-session skill use and lack systematic risk coverage. SkillHarm addresses these gaps with lifecycle-aware attack scenarios and automated construction, finding attack success rates up to 86.3% in FPP and 69.3% in SMP.

  • Problem

    Existing benchmarks mainly evaluate poisoned skills within one task execution and use ad-hoc risk lists, limiting lifecycle and systematic risk coverage.

  • Method

    SkillHarm evaluates fixed-payload and self-mutating poisoning across skill lifecycles and organizes attacks into 12 workflow-component risk types.

  • Results

    Attack success rates reach 86.3% for FPP and 69.3% for SMP, with current agents remaining highly vulnerable across the benchmark.

  • Takeaways & Limitations

    Many apparent failures reflect agents not accessing poisoned files, while scanners and defensive prompts do not reliably mitigate the threat.

  • Takeaways & Limitations

    The threat model assumes attackers control package files but lack post-installation access to victim environments or knowledge of downstream prompts and configurations.

Abstract

from arXiv · show

Agent skills occupy a privileged position in the agent workflow, as agents are expected to implicitly follow and execute them, rendering third-party skills a vulnerable attack surface. Existing studies have revealed unsafe agent behaviors induced by skill-based attacks, but they primarily evaluate poisoned skills within a single task execution and enumerate harms through ad-hoc risk lists. To bridge these gaps, we introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle, paired with a systematic taxonomy of skill-relevant risks. SkillHarm evaluates two attack scenarios: Fixed-Payload Poisoning (FPP), where a fixed poisoned skill package directly compromises any task session that invokes it, and Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content, deferring harm until a subsequent reuse. It further defines 12 risk types based on the agent workflow component targeted by the harm: data pipelines, system environments, and agent autonomy. To instantiate these attacks at scale, we build AutoSkillHarm, an automated construction pipeline with coding agents driven by natural-language harnesses. The resulting benchmark contains 879 attack samples across 71 skills. Experiments show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP. Our analysis further reveals a latent risk: many apparent attack failures stem from the agent failing to engage with the poisoned file rather than genuine resistance, and current defenses still fail to reliably mitigate the threat.

1 Introduction

SkillHarm extends skill-attack evaluation from isolated task runs to the installed-skill lifecycle, covering both direct and deferred cross-session compromise. Its benchmark scales automated attack construction across heterogeneous skills and shows that agents and standard defenses remain vulnerable.

  • Motivation: SkillHarm targets lifecycle gaps in prior benchmarks that largely evaluated poisoned skills within a single task execution.This misses persistence and reuse across sessions.
  • Attack scenarios: FPP compromises every task session invoking a fixed poisoned package, whereas SMP mutates persistent content and triggers harm when a later task reuses it.Together, the scenarios cover direct single-session and deferred cross-session compromise.
  • Benchmark and evaluation: 879 self-contained attack samples span 71 skills and 12 risk types, covering both single-session and cross-session attacks.The benchmark evaluates six model-harness configurations across Claude Code, Codex, Gemini CLI, and OpenCode.
  • Results: Current agents remain highly vulnerable to SkillHarm attacks across the evaluated harness configurations.The benchmark reports attack success rates up to 86.3% for FPP and 69.3% for SMP.
  • Defenses: Skill scanners miss many contextualized payloads, while defensive system prompts do not reliably reduce overall attack success rates.Explicit refusal is rare; only Claude-family models show noticeable refusal behavior, and their refusal rates remain low.

2 Related Work

Prior work has studied prompt injection across diverse agent attack surfaces, while agent skills introduce a distinct surface through reusable procedural artifacts with a privileged workflow position. Automated attack construction is motivated by the cost of manual crafting, but skill-based heterogeneity makes existing approaches only partially adequate.

  • Skill-Based Attacks: Agent skills constitute a distinct attack surface because they are reusable procedural artifacts occupying a privileged position in agent workflows.
  • Skill-Based Attacks: Prompt injection research has examined tools, webpages, MCP servers, and operating-system environments as agent attack surfaces.
  • Automated Attack Construction: Manual attack crafting is costly and difficult to scale, motivating automated attack-construction methods.
  • Automated Attack Construction: Skill-based attack construction is amplified by heterogeneity across skills, risk types, and attack scenarios, leaving existing automated approaches only partially effective.Template-based or seed-mutation pipelines are described as existing approaches that address this challenge only partially.

3 Threat Model and Attack Formulation

SkillHarm models publicly distributed poisoned skills under a constrained attacker–victim setting, and evaluates attacks by when harm occurs and which agent-workflow component it compromises. Its formulation distinguishes direct same-session compromise from deferred harm caused by persistent skill mutation and organizes 12 risks into three workflow categories.

  • 3.1 Threat Model: Attackers publish poisoned skill packages and control shipped files, but lack post-installation access to victims’ environments, executions, prompts, files, and configurations.The attacker may control SKILL.md, references, scripts, and auxiliary resources while knowing the skill’s advertised purpose and supported task class.
  • 3.1 Threat Model: The victim follows the standard skill-use workflow, making the poisoned package’s distributed contents the attacker’s primary control surface.The supplied threat model states that attackers cannot observe or adapt to a particular victim environment after installation.
  • 3.2 Attack Scenarios: SkillHarm evaluates harm across the skill-use lifecycle, distinguishing outcomes realized during the current execution from outcomes deferred until later skill reuse.Persistent procedural artifacts can enable either immediate harm or later harm after modification of reusable skill content.
  • 3.2 Attack Scenarios: Fixed-Payload Poisoning (FPP) embeds a malicious payload at installation, enabling harmful outcomes within the same execution when an agent invokes the skill for a relevant task.Payloads may reside in instructions, reference documents, helper scripts, or attacker-added artifacts referenced by the original skill files.
  • 3.2 Attack Scenarios: Self-Mutating Poisoning (SMP) makes the first session appear harmless while silently modifying persistent skill content, deferring harm until a later reuse.This scenario exposes failures that single-session evaluations cannot observe because the initial task may complete successfully while leaving a modified reusable skill pack.
  • 3.3 Risk Taxonomy: The evaluation space also varies by which agent-workflow component is compromised: task-data processing, local-environment code execution, or delegated decision-making.These components specify where the harmful outcome materializes during skill use.
  • 3.3 Risk Taxonomy: SkillHarm groups 12 skill-related risk types into three workflow-based categories, organized by the component through which harm materializes.The taxonomy is summarized in Table 2 and covers data pipelines, system environments, and agent autonomy.

4 Automated Skill-Attack Construction

AutoSkillHarm is a three-stage coding-agent pipeline that constructs lifecycle-aware skill attacks across FPP, SMP, and 12 risk types. The resulting benchmark contains 879 runnable attack samples selected, designed, and quality-filtered through grounded procedures.

  • Pipeline overview: AutoSkillHarm constructs SkillHarm attacks across two attack scenarios and 12 risk types.The pipeline uses coding agents driven by natural-language harnesses to address heterogeneous construction requirements.
  • Pipeline overview: The pipeline has three stages: attack target selection, attack design, and quality filtering.Implementation details are provided in Appendix B.
  • Attack target selection: For FPP, the pipeline retains skill-file injection points reached in at least 30% of benign baseline trajectories, capping each task at three documentation files and three scripts.An FPP target is defined by a user task and a skill file that agents are likely to read, consult, or execute.
  • Attack target selection: For SMP, the pipeline selects task pairs that reuse a shared skill, with an earlier routine task triggering mutation and a later task enabling a security-relevant outcome.Three coding agents independently propose candidate task pairs to improve coverage and reduce selection bias.
  • Quality filtering: Quality filtering validates candidates through end-to-end executions on Claude Code with Sonnet 4.6 and Codex with GPT-5.4, followed by trajectory-grounded reviewer analysis.Reviewers check environmental feasibility, payload plausibility, and whether the evaluator captures the intended harmful end state.
  • Benchmark composition: 879 final attack samples comprise 687 FPP and 192 SMP samples spanning 12 risk types.Each sample is a runnable evaluation unit containing an environment, task or task pairs, poisoned skill files, and a deterministic evaluator.

5 Experiments

Experiments show that frontier agents remain highly vulnerable to both fixed-payload and self-mutating skill attacks, while engagement-conditioned results expose substantial latent risk. Existing scanners and defensive prompts provide unreliable protection, and poisoned skills often preserve plausible task success despite harmful side effects.

  • Attack Effectiveness: FPP ASR reaches up to 86.3%, while SMP ASR reaches up to 69.3% despite requiring persistent mutation and cross-session reuse.Codex agents are most vulnerable in both scenarios, whereas Claude Code with Opus 4.7 has the lowest ASR among evaluated configurations.
  • Latent Risk: SMP cASR increases attack success by up to 32.1% for Claude Opus 4.7, showing that many apparent failures reflect non-engagement with poisoned files.cASR conditions success on target-payload engagement inferred from agent trajectories using LLM-as-a-Judge.
  • Attack Refusal Behavior: Claude-family agents show noticeable explicit refusals, but Sonnet 4.6 ARR drops from 25.3% in FPP to 2.1% in SMP.Temporal separation between preparation and harm reduces malicious-intent visibility, making SMP attacks harder for agents to refuse.
  • Risk and Payload Breakdown: Script-mediated payloads often achieve high ASR because agents commonly execute helper scripts without inspecting their contents, while overt data-exfiltration harms face greater resistance.Attacks succeed across all structural risk types, but payload realization form materially affects outcomes.
  • Defense: The strongest skill-scanner configuration detects only 55.6% of FPP injections and 68.8% of SMP injections, while defensive prompts leave nearly all configurations above 70% ASR.These results support the conclusion that existing defenses provide insufficient protection against skill-based attacks.
  • Task Utility Under Attack: Poisoned skills preserve similar task success for most configurations, although Claude Code with Opus 4.7 shows a 13.8% drop consistent with stronger refusal behavior.Users may therefore receive plausible outputs while harmful side effects occur silently.

6 Conclusion · Appendix · A Comparison with Related Benchmarks

SkillHarm evaluates skill-based attacks across the skill-use lifecycle and 12 workflow-component risk types, while AutoSkillHarm automates scalable attack construction. Its benchmark contains 879 diverse attack samples, and evaluation finds current agents highly vulnerable, with many failures caused by not accessing the injected file.

  • 6 Conclusion: SkillHarm evaluates agent vulnerabilities to skill-based attacks across the skill-use lifecycle.
  • 6 Conclusion: The benchmark covers single-session and cross-session attack scenarios across 12 risk types organized by affected workflow component.
  • 6 Conclusion: AutoSkillHarm is an automated, scalable, and extensible skill-attack construction pipeline driven by coding agents with natural language harnesses.
  • 6 Conclusion: 879 diverse attack samples are produced by the AutoSkillHarm construction pipeline.
  • 6 Conclusion: All current agents remain highly vulnerable according to the evaluation.
  • 6 Conclusion: Most attack failures stem from agents never accessing the injected file.

B Benchmark Construction Details

The benchmark construction details cover target selection, attack design, and quality filtering.

  • B.1 Target Selection Details: Target selection details are presented in Section B.1.
  • B.2 Attack Design Details: Attack design details are presented in Section B.2.
  • B.3 Quality Filter Details: Quality filter details are presented in Section B.3.

C Detailed Benchmark Statistics … F Analysis of Refinement Iterations

The paper organizes its supplementary analysis around benchmark statistics, experimental details, additional analyses, and refinement-iteration behavior. These sections separately cover fixed-payload and self-mutating poisoning, evaluation breakdowns, defenses, harness–model relationships, designer-agent ablations, and evasion patterns.

  • C Detailed Benchmark Statistics: Fixed-Payload Poisoning is presented as a dedicated subsection of the detailed benchmark statistics.
  • C Detailed Benchmark Statistics: Self-Mutating Poisoning is presented as a dedicated subsection of the detailed benchmark statistics.
  • D Experimental Details: The experimental-details section covers evaluation metrics and result breakdowns by risk and realization.
  • E Additional Analysis: Additional analyses examine defensive system prompts, harness–model decoupling, and ablation of the designer agent.
  • F Analysis of Refinement Iterations: The refinement-iteration analysis begins with a quantitative overview.
  • F Analysis of Refinement Iterations: A separate subsection analyzes surface-level evasion during refinement iterations.
  • F Analysis of Refinement Iterations: The refinement-iteration analysis also considers structural evasion.
  • F Analysis of Refinement Iterations: The final refinement subsection addresses intrinsically undetectable constructions.

A Comparison with Related Benchmarks … D Experimental Details

SkillHarm extends prior benchmarks with cross-session poisoning, a skill-specific 12-risk taxonomy, diverse payload realizations, and deterministic evaluation. AutoSkillHarm constructs and filters attacks through exposure-based targeting, lifecycle-aware mutation, environment grounding, and trajectory review, yielding 687 FPP and 192 SMP samples.

  • A Comparison with Related Benchmarks: SkillHarm uniquely models cross-session attacks in which a skill mutates during one session and persists to compromise later sessions.Prior benchmarks evaluate attacks within a single agent session, leaving this temporal failure mode invisible.
  • A Comparison with Related Benchmarks: SkillHarm organizes 12 risk types into 3 workflow categories: data pipeline, system environment, and agent autonomy.This skill-specific taxonomy enables systematic coverage and per-category robustness analysis.
  • A Comparison with Related Benchmarks: Each attack sample is independently designed for its target skill and risk type, prioritizing attack diversity over raw sample count.SkillHarm also varies payload delivery mechanisms, enabling analysis of effectiveness and stealth.
  • B.1 Target Selection Details: FPP target selection retains skill files with read rate r_t,f ≥ 0.30, computed from released SkillsBench agent trajectories.For task t, read rate is r_t,f = n_t,f/T_t.
  • B.1 Target Selection Details: SMP construction uses three selector agents to propose task pairs scored by feasibility and impact.The feasibility axis measures normal shared-skill execution, while impact ranges from low-stakes harm to critical operations.
  • B.2 Attack Design Details: FPP payloads use in-place or routing-based realizations, while SMP adds an atexit callback that rewrites persistent skill files after Task A for activation in Task B.Environment grounding seeds required preconditions, and evaluators check only concrete harmful end states deterministically and independently of realization.
  • B.3 Quality Filter Details: 47% of filtered-out designs failed because of weak cover stories, followed by 32% with infrastructure failures and 9% with improper payload placement.Additional failures involved incorrect evaluators (6%), dependency mismatches (4%), and other issues (2%).
  • C Detailed Benchmark Statistics; C.1 Fixed-Payload Poisoning; C.2 Self-Mutating Poisoning; D Experimental Details: 687 FPP samples cover 57 tasks, 71 skills, and 126 unique task–poisoned-file combinations, while 192 SMP samples cover 12 task pairs, 6 skills, and 15 combinations.SkillHarm’s primary ASR uses deterministic scripts checking concrete end-state artifacts without LLM involvement.

D.1 Evaluation Metrics … E.2 Harness–model Decoupling

SkillHarm evaluates attack success alongside whether agents engage with poisoned files or refuse suspicious instructions, then analyzes risk, payload form, defenses, and harness–model effects. The results show that data-flow harms are more resistant, script-mediated payloads are more effective, defensive prompts are unreliable, and both model and harness contribute to vulnerability.

  • D.1 Evaluation Metrics: The engagement judge classifies whether the agent inspected the entry-point file, identified suspicious content, and declined to follow it.Inspection includes reads, shell commands processing file contents, and native skill-loading tools for SKILL.md files.
  • D.1 Evaluation Metrics: Conditional ASR aggregates attack success over samples judged to use the target file, while Attack Refusal Rate measures samples judged to involve refusal.ASR uses deterministic evaluators over the full sample set, whereas cASR and ARR aggregate over all samples.
  • D.2 Result Breakdown by Risk and Realization: Data pipeline exploitation has the lowest ASR in both settings, indicating greater resistance to overt data-flow harms such as explicit data exfiltration.Explicit external-endpoint patterns are more likely to be noticed by agents.
  • D.2 Result Breakdown by Risk and Realization: Script-mediated payloads achieve substantially higher ASR than document-based payloads because agents often treat helper scripts as trusted and execute them without inspection.This exposes a gap between defenses against natural-language indirect prompt injection and defenses against skill-based attacks.
  • E Additional Analysis: The defense evaluation uses a high-ASR subset stratified across risk types, selecting the top 30% of retained samples within each type by mean ASR across six victim agents.This design prevents the subset from concentrating only on naturally high-ASR risk categories.
  • E.1 Defense with Defensive System Prompt: Defensive system prompts reduce ASR in some settings but remain unreliable across agents, with nearly all evaluated agents above 70% ASR in both settings.The defense warns agents to inspect skill documents and scripts, follow the user’s task, and disregard malicious or conflicting content.
  • E.2 Harness–model Decoupling: Both model and harness contribute to vulnerability: with GPT-5.4, switching from Codex to OpenCode reduces ASR and cASR in both settings.The comparison fixes the harness to OpenCode while varying between Claude Opus 4.7 and GPT-5.4, the lowest- and highest-ASR models in the main results.

E.3 Ablation of Designer Agent · F Analysis of Refinement Iterations · F.1 Quantitative Overview

The ablation compares Claude Code (Opus 4.7) with Codex (GPT-5.4), finding FPP largely designer-robust but SMP more designer-sensitive. Refinement analysis shows repeated scan–revise activity, while most evasion reasoning occurs before detector-triggered revisions.

  • E.3 Ablation of Designer Agent: The study compares Claude Code (Opus 4.7) and Codex (GPT-5.4) to test whether attack effectiveness depends on the designer agent.FPP is rerun on a randomly sampled 30% subset, whereas SMP uses the full set because it is smaller.
  • E.3 Ablation of Designer Agent: FPP is largely designer-robust: both designers achieve the same average ASR, with slight cross-family transfer differences.Each designer’s attacks transfer slightly better to the other model family.
  • E.3 Ablation of Designer Agent: SMP is more sensitive to the choice of designer agent than FPP.The supplied passage identifies greater SMP sensitivity but does not provide the completed comparative values.
  • F Analysis of Refinement Iterations: The refinement analysis organizes designer behavior into reframing the payload’s presentation, redesigning its mechanism, and constructing intrinsically legitimate-looking payloads.These levels emerge from interactions with an LLM-based detector, without specifying detector-evasion tactics.
  • F.1 Quantitative Overview: 3.9 detector invocations occur per fixed-payload poisoning design and 3.7 per self-mutating poisoning design on average.These figures summarize the refinement workload across the two attack settings.
  • F.1 Quantitative Overview: 82–88% of designs undergo at least two scan–revise cycles.The range is reported across settings in the refinement workload summary.
  • F.1 Quantitative Overview: 12% of fixed-payload poisoning designs and 14% of self-mutating poisoning designs require active evasion revisions after a critical detector flag.The remaining designs evade on early attempts, indicating that evasion reasoning is front-loaded rather than discovered reactively.

F.2 Surface-Level Evasion · F.3 Structural Evasion · F.4 Intrinsically Undetectable Constructions

Evasion progresses from surface-level reframing to structural redesign, while some constructions make detection inapplicable by using operations indistinguishable from legitimate behavior. Across these strategies, attacks preserve harmful outcomes while shrinking or eliminating detectable signals.

  • F.2 Surface-Level Evasion: Surface refinements preserve payload behavior while replacing imperative language, attack-indicative terms, or domain-specific triggers with descriptive, domain-native, or generic presentation.Detectors flag explicit directive patterns such as “REQUIRED” and “MUST” more readily than documentation framed as reference behavior or system description.
  • F.2 Surface-Level Evasion: Zero detection across all realizations followed imperative-to-descriptive reframing of an identical DOI-allowlist poisoning payload.The successful framing described cache validation behavior rather than prescribing attacker-chosen registrant prefixes as pre-verified.
  • F.2 Surface-Level Evasion: 100% evasion across five consecutive scans resulted from embedding goal-hijacking behavior inside a reference pdfplumber code block that victim agents routinely copy and execute.The detector treated code blocks as inert illustrative material despite applying directive heuristics to textual instructions.
  • F.2 Surface-Level Evasion: A generic pdfplumber calibration-hygiene framing passed with only a non-critical warning while preserving the identical forensic-log-truncation end state.The designer reached this result after pivots from mandatory NERC CIP-008 framing and conditional financial-PDF activation.
  • F.3 Structural Evasion: Structural evasion is required when harmful semantics intrinsically overlap detector knowledge, including document-content exfiltration, attacker-directed file creation, and executable-content retrieval.These risk types remain flagged regardless of cover story because removing the core mandate would break the attack.
  • F.3 Structural Evasion: A content–metadata split shrank exfiltration’s detectable footprint by sending page_count, chars, and fingerprint[:16] over HTTP while staging complete text locally.The resulting call resembled benign telemetry despite preserving the attack’s access to full extracted content.
  • F.3 Structural Evasion: Mechanism-level redesign replaced detected SSH persistence with a PYTHONSTARTUP hook framed as GaMMA worker telemetry, demonstrating reasoning about alternative detection surfaces.In another SMP case, exfiltrated content survived cross-container transfer in scripts/.preflight_cache/ as a skill-cache artifact invisible to detector and victim agent.
  • F.4 Intrinsically Undetectable Constructions: Intrinsically undetectable constructions use legitimate operations as malicious steps, including one-sided text normalization and natural content-type gates.A single .title() call created phantom department changes, while detecting embedded .xlsx files activated financial-advisory content without an inherently malicious signal.
Loading 2606.02540v1…