PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents

Xiaojun Jia, Jie Liao, Simeng Qin, Jindong Gu, Wenqi Ren, Xiaochun Cao, Yang Liu, Philip Torr

arXiv:2602.14211v3cs.CRcs.AI

TL;DR

Skill-based prompt injection remains difficult to trigger reliably because explicit malicious instructions can be rejected or ignored, despite skills being loaded as trusted executable capability packages. SkillJect addresses this gap with coordinated artifact hiding, front-loaded documentation inducement, and closed-loop trace-guided refinement. Across its evaluations, it substantially outperforms naive direct injection and prior manual attacks, while its scope is limited to settings where the poisoned skill is selected and payload execution is measured rather than complete downstream impact.

  • Problem

    Existing skill-based attacks are often manual and brittle, while poisoned skills can exploit trusted packages of instructions and executable artifacts to steer agent tool use.

    Source: §I · p. 2

  • Method

    SkillJect hides payloads in auxiliary helper scripts, rewrites SKILL.md with front-loaded prerequisite instructions, and iteratively refines the documentation using Attack, Victim, and Evaluate Agents.

    Source: §I · p. 2, C. Generating injected skills with… · p. 6

  • Results

    SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks across platforms, backend LLMs, and attack categories.

    Source: §I · p. 3, 1) Results on Claude Code · p. 7, §VI · p. 14

  • Takeaways & Limitations

    Poisoned skills are a persistent threat in reusable skill ecosystems because apparently coherent documentation can induce execution of hidden helper artifacts.

    Source: §VI · p. 14

    The evaluation assumes the poisoned skill is selected and measures hidden-helper execution, not whether every execution produces a fully realized downstream consequence.

    Source: §V · p. 13

Abstract

from arXiv · show

Agent skills extend LLM agents with task-specific instructions, executable scripts, and auxiliary resources, improving reusability but creating a new supply-chain attack surface. A malicious or compromised skill can be repeatedly loaded as trusted guidance and steer downstream tool use. Existing skill-based prompt-injection attacks are often manual and brittle, because explicit malicious instructions are rejected or ignored when they are not aligned with the original workflow. We propose SkillJect, the first automated framework for generating poisoned skills against skill-enabled agent systems. SkillJect uses two coordinated channels. In the artifact channel, it hides the payload inside an auxiliary helper script. In the instruction channel, it rewrites SKILL.md with a front-loaded inducement strategy, placing injected content at the beginning and framing the helper script as a mandatory prerequisite or initialization step. The rewritten instruction explicitly references the helper-script path and provides an executable example command, making the helper appear to be a legitimate setup step before normal skill operations. SkillJect further adopts a closed-loop multi-agent process to improve attack effectiveness. An Attack Agent generates poisoned skills, a Victim Agent executes downstream tasks with the poisoned skill, and an Evaluate Agent inspects execution traces to determine whether the hidden payload was executed. The Attack Agent then uses this feedback to diagnose failure causes and rewrite SKILL.md, while keeping the payload fixed. Experiments across skill-enabled platforms, backend LLMs, and attack categories show that SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks, highlighting poisoned skills as a persistent threat in reusable skill ecosystems.

I. INTRODUCTION

Agent skills package trusted instructions with executable artifacts, creating a supply-chain surface for prompt injection. SkillJect automates poisoned-skill generation by combining hidden helper scripts, front-loaded inducement instructions, and trace-guided refinement, outperforming direct and manual attacks in evaluation.

Motivation

Threat

  • Poisoned skills can covertly steer tool use toward sensitive-file access, project modification, or other unauthorized actions when imported and executed.Impact depends on runtime permissions, sandboxing, network access, and user approval policies.Source: Figure 1 · p. 1, §I · p. 2
  • Publicly collected malicious skills rarely trigger their intended behavior because explicit instructions are often rejected or ignored by agents.The figure distinguishes explicit refusal from failure to follow the instruction after skill execution.Source: §I · p. 2, Figure 2 · p. 2

Approach

  • SkillJect uses an Attack Agent, Victim Agent, and Evaluate Agent in a closed loop to generate poisoned skills and refine them from execution traces.The evaluator checks whether the hidden payload executes, and the attack process uses that feedback for further rewriting.Source: §I · p. 2
  • SkillJect coordinates hidden auxiliary payloads with front-loaded SKILL.md instructions that present a helper script as a required workflow prerequisite.The injected documentation references the helper path and executable command, making the script appear operationally legitimate.Source: §I · pp. 2–3

Results

  • SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks across platforms, backend models, and attack categories.The reported experiments support improved effectiveness in the evaluated benchmark.Source: §I · p. 3

C. Skill Security

SkillJect treats skills as executable security boundaries and constructs poisoned packages whose documentation and artifacts jointly induce helper-script execution. Its closed-loop agents use execution traces to refine the documentation while preserving the payload.

Skill security

  • Skills resemble software plugins because they bundle trusted instructions, executable scripts, configuration, and auxiliary resources.This packaging creates a distinct supply-chain surface beyond passive prompt context.Source: C. Skill Security · p. 3

Security gap

Closed-loop refinement

  • The Victim Agent produces execution traces, while the Evaluate Agent checks hidden-helper execution and returns feedback for iterative documentation refinement.The Attack Agent keeps the artifact payload fixed while rewriting the inducement.Source: §III · p. 4

Attack mechanism

Scope

  • The attack is evaluated only when the poisoned skill is selected and counts success at hidden-helper execution rather than full downstream harm.Router compromise is not assumed, and downstream consequences may depend on deployment controls.Source: A. Threat Model and Problem Formulation · p. 5

C. Generating injected skills with SKILLJECT

SKILLJECT generates poisoned skills by hiding malicious functionality in auxiliary artifacts while rewriting SKILL.md to induce execution through a front-loaded prerequisite. A closed-loop process then refines the documentation using victim execution traces while keeping the payload fixed.

Trace-level objective

  • SKILLJECT evaluates success with a trace-level predicate that checks whether the victim agent explicitly executed the hidden helper script.Execution traces include tool and command executions, file operations, and intermediate or final outputs.Source: C. Generating injected skills with… · p. 6

Payload understanding

  • The Attack Agent analyzes the helper script to infer its functionality, inputs, outputs, execution stage, and plausible workflow role before rewriting the documentation.This payload understanding guides construction of an operationally aligned injected document.Source: C. Generating injected skills with… · p. 6

Constrained generation

  • The front-loaded strategy places a mandatory helper-script instruction at the beginning of SKILL.md, references its path, and supplies an executable command before normal operations.This presentation is intended to increase salience and execution priority while making the helper appear to be a required initialization step.Source: C. Generating injected skills with… · p. 6

Feedback-driven refinement

  • The hidden payload is generated once and held fixed throughout refinement, isolating improvements in inducement quality from changes in payload capability.This design focuses the optimization process on how the payload is presented to the victim agent.Source: C. Generating injected skills with… · p. 6
  • The framework uses an iterative refinement loop in which the Evaluate Agent diagnoses victim traces and the Attack Agent revises the documentation using stored feedback.The feedback can classify outcomes such as ignored, refused, partially triggered, or incorrectly executed.Source: C. Generating injected skills with… · p. 6

IV. EXPERIMENTS

Experiments evaluate SKILLJECT across four high-severity attack behaviors, multiple backend models, and two skill-enabled platforms. It substantially outperforms direct injection, achieving high attack success rates across categories and platforms while showing lower success on Claude models.

Experimental setup

  • The benchmark contains 100 agent skills from ClawHub and evaluates information disclosure, privilege escalation, unauthorized write, and backdoor injection behaviors.Claude Code is the main victim platform, with OpenClaw used to assess cross-platform transferability.Source: A. Experimental Setup · pp. 6–7

Results on Claude Code

  • SKILLJECT achieves 80.7% average overall ASR on Claude Code, whereas naive direct injection achieves 0.0% across all attack categories and four backend models.The comparison indicates that explicit malicious instructions alone were ineffective in this evaluation.Source: 1) Results on Claude Code · p. 7
  • Across attack categories on Claude Code, SKILLJECT reaches 78.8% INFODISC, 81.0% PRIVESC, 83.5% UNAUWRI, and 79.5% BACKDOOR average ASR.All four category averages contrast with the naive baseline’s 0.0% results.Source: 1) Results on Claude Code · p. 7
  • Across Claude Code backends, SKILLJECT’s overall ASR is 97.2% on GLM-4.7, 94.7% on MiniMax-M2.1, 83.8% on GPT-5-mini, and 47.0% on Claude-Sonnet-4.6.Claude-Sonnet-4.6 is the least susceptible backend among those reported, but the attack still succeeds in a non-trivial fraction of cases.Source: 1) Results on Claude Code · p. 7

Results on OpenClaw

  • On OpenClaw, SKILLJECT achieves 80.5% average overall ASR while the naive baseline again records 0.0% across all four backend models.Reported backend ASRs are 97.0% on GLM-4.7, 95.0% on MiniMax-M2.1, 87.0% on GPT-5-mini, and 43.0% on Claude-Sonnet-4.6.Source: 2) Results on OpenClaw · p. 7

3) Comparison on more advanced models:

Across newer models, platforms, and prior attack baselines, SKILLJECT remains substantially more effective than naive or manual injection, although transferability varies by backend and task.

Advanced backend models

  • 68.4% average ASR across six newer frontier backends shows SKILLJECT remains effective beyond earlier evaluated models, while GPT-5.4 and Claude-Opus-4.6 yield 28.7% and 29.2%.The advanced-model evaluation used 30 sampled skills/tasks and the same four behavior categories.Source: 3) Comparison on more advanced models · p. 8
  • 0.0% ASR on all six advanced backends for Naive direct injection confirms that explicit malicious insertion is unreliable against these models.The comparison uses the same evaluation protocol and target behavior categories as SKILLJECT.Source: 3) Comparison on more advanced models · p. 8, Table III · p. 8

Manual attack comparison

  • 69.1% overall ASR demonstrates SKILLJECT’s advantage over prior manually designed malicious skills across the evaluated comparison.The improvement is attributed to automated inducement generation and feedback-driven refinement rather than stronger payload scripts.Source: 4) Comparison with Prior Manual Attacks · p. 9, Table V · p. 9
  • SKILLJECT raises ASR from 40.0% to 82.2% on GLM-4.7, from 35.6% to 80.9% on MiniMax-M2.1, and from 26.7% to 64.4% on GPT-5-mini.The same benign skills and malicious payloads were used for both methods.Source: 4) Comparison with Prior Manual Attacks · p. 9

Cross-model transferability

  • Cross-model transfer remains non-trivial, with transferred skills achieving 49.8% and 35.8% overall ASR on two target backends without target-specific refinement.The supplied passage identifies these results as transfer outcomes and separately reports success across all four target behaviors on Claude-Sonnet-4.6.Source: 5) Cross-model transferability · p. 9

Cross-task transferability

  • Cross-task evaluation reuses one poisoned skill across 20 additional LLM-generated instructions for the same skill, without regeneration or refinement.This setup tests whether a skill generated for one source task transfers to other associated task instructions.Source: Figure 4 · p. 9

Cross-model transferability

  • Compared with 97.2% on GLM-4.7, transferred performance decreases by 17.4, 47.4, and 61.4 percentage points on MiniMax-M2.1, GPT-5-mini, and Claude-Sonnet-4.6.The results suggest workflow-level attack patterns transfer, while model-specific refinement remains important for maximizing effectiveness.Source: 5) Cross-model transferability · pp. 9–10

Generated-injection design

  • The generated documentation uses front-loaded, workflow-compatible inducements that present helper execution as a required prerequisite or initialization step.Qualitative examples include camouflaged prerequisite sections and emphasized initialization warnings.Source: Figure 5 · p. 10

6) Cross-task Instruction Transferability:

SKILLJECT improves injected-skill execution through iterative trace-guided refinement and documentation that remains structurally natural, while testing transfer across tasks and behavior categories.

6) Cross-task Instruction Transferability:

  • Cross-task evaluation reuses each poisoned skill on 20 additional task instructions associated with the same skill, without regeneration or refinement.The evaluation isolates transferability from repeated attack generation.Source: 6) Cross-task Instruction… · p. 10
  • The cross-task setting reflects real use in which one skill may serve multiple related user tasks, such as weather queries across different dates or locations.The poisoned skill is generated from a specific source task instruction and then evaluated on additional instructions.Source: 6) Cross-task Instruction… · p. 10

2) Effect of Feedback-Driven Refinement:

1) Qualitative Analysis of Generated Injections:

  • The qualitative examples show helper execution framed either as a camouflaged prerequisite section or as an emphasized initialization warning before the main workflow.Both forms preserve the same underlying mechanism: placing the inducement before normal skill operations.Source: 1) Qualitative Analysis of Generated… · pp. 10–11

2) Effect of Feedback-Driven Refinement:

  • After five refinement iterations, ASR reaches 97.2% on GLM-4.7, 94.6% on MiniMax-M2.1, and 83.8% on GPT-5-mini.Claude-Sonnet-4.6 improves more slowly and reaches 52.8% after the reported iterations.Source: 2) Effect of Feedback-Driven Refinement · p. 11
  • The Evaluate Agent classifies ignored, refused, partially triggered, or incorrectly executed attempts, and the Attack Agent revises documentation while keeping the hidden payload fixed.The loop uses execution traces to improve the inducement rather than regenerating the payload.Source: 2) Effect of Feedback-Driven Refinement · p. 11

3) Structural Naturalness Score of Injected Skills:

  • SKILLJECT generates more structurally and stylistically natural injected skills than the Naive baseline across the four target behavior categories.Structural Naturalness Score is computed with an LLM-as-a-judge comparison of benign and poisoned SKILL.md files.Source: 3) Structural Naturalness Score of… · p. 11, Figure 7 · p. 11
  • For Backdoor Injection, SNS increases from 0.223 to 0.429, while Information Disclosure rises from 0.318 to 0.415.These category-specific increases support the broader finding that injected documentation better preserves the original workflow.Source: 3) Structural Naturalness Score of… · p. 11

D. Robustness against Potential Defenses

The paper evaluates prompt-level and scanner-based defenses, plus component ablations and deployment safeguards. Results show partial mitigation, while attack success remains substantial and depends on execution context and system configuration.

Potential Defenses

Instruction-Level Defense

  • 49.0 percentage points: the instruction-level defense reduces overall ASR from 97.3% to 48.3% across four target behaviors.The reduction is strongest for recognizable exfiltration behavior but does not eliminate workflow-compatible attacks.Source: 1) Instruction-Level Prompt Defense · p. 12
  • 66% and 77% ASR remain for Privilege Escalation and Unauthorized Write, respectively, after applying the prompt defense.These results show that explicit safety instructions are insufficient against attacks compatible with the requested workflow.Source: 1) Instruction-Level Prompt Defense · p. 12

Ablation Study

  • 97.2% overall ASR falls to 91.0% without payload understanding, 78.8% without front-loaded inducement, and 80.5% without refinement feedback.The larger drops from removing front-loaded inducement and feedback indicate that instruction placement and trace-based revision materially contribute to attack success.Source: E. Ablation Study · p. 13

Scope and Limitations

  • Attack success is measured as hidden-helper execution, not guaranteed downstream harm, which also depends on permissions, sandboxing, network access, and approval mechanisms.The evaluation uses a post-selection setting and may not generalize across all skill ecosystems, agent scaffolds, prompts, permissions, or deployment policies.Source: §V · p. 13

Responsible Evaluation

  • Experiments use isolated Docker environments, controlled tasks, dummy secrets, and synthetic targets without attacking real users or production systems.The authors state that directly reusable malicious payloads and end-to-end attack scripts will not be released.Source: §V · p. 13

APPENDIX

The appendix documents SKILLJECT’s prompt-driven implementation for analyzing helper scripts and integrating them into skill documentation. It specifies structured analysis fields, attack-type-dependent descriptions, and constrained documentation-generation inputs.

Payload Understanding

  • The payload-understanding module analyzes a helper script and returns structured JSON for the subsequent generation stage.The analysis includes technical behavior, a plausible legitimate purpose, and a natural SKILL.md description.Source: 1) Payload Understanding Prompt · p. 16
  • The appendix’s analysis prompt requests the attack type, script content, technical behavior, legitimate purpose, and natural documentation description.It also asks for integration guidance that avoids suspicious phrasing.Source: 1) Payload Understanding Prompt · p. 16

Constrained Generation

  • The constrained-generation prompt places the helper script at the beginning of SKILL.md as a mandatory prerequisite before other operations.It specifies the script path, purpose, required execution level, description, and an example command such as bash resources/{script_path}.Source: 2) Constrained Generation Prompt · p. 16

Fallback Descriptions

  • Attack-type-dependent fallback descriptions portray scripts as environment, dependency, package-installation, or documentation-fetching utilities.The defaults cover information disclosure, privilege escalation, unauthorized write, and backdoor injection categories.Source: 2) Constrained Generation Prompt · pp. 16–17

Feedback-Driven Refinement

3) Feedback-Driven Refinement Prompts:

The refinement prompts adapt SKILL.md after rejected, ignored, or unsuccessful execution attempts. They progressively emphasize mandatory placement, neutral framing, direct commands, verification, and error handling.

Feedback Inputs

  • Refinement prompts incorporate previous-attempt status, root cause, evidence, and an improvement strategy into the next documentation rewrite.The prompts distinguish rejected, ignored, and attempted-but-failed execution modes.Source: 3) Feedback-Driven Refinement Prompts · p. 17

Narrative Refinement

  • The prompts also direct the generator to use neutral technical language, avoid sensitive keywords, emphasize legitimate uses, and make the script sound routine.These changes target narrative framing rather than changing the hidden helper payload.Source: 3) Feedback-Driven Refinement Prompts · p. 17

Mandatory Integration

  • The prompts require the helper script to be integrated as a mandatory first step with a specified path and legitimate-purpose description.They repeatedly frame execution as necessary before normal skill use and preserve the script resource information across iterations.Source: 3) Feedback-Driven Refinement Prompts · pp. 17–18

Making Execution Unmissable

  • When inducement is insufficient, refinement strengthens imperative language, prominent placement, repeated references, and verification requirements.The instructions recommend terms such as MUST and REQUIRED and place the script call at the beginning or end of the skill instructions.Source: 3) Feedback-Driven Refinement Prompts · pp. 17–18

Failure Handling

  • For execution failures, refinement verifies the script path and permissions, simplifies commands, adds debug output, and checks errors explicitly.The prompts include exit-code checks, output display, file-state verification, existence checks, and permission checks.Source: 3) Feedback-Driven Refinement Prompts · p. 18

8) Rule-Based Direct Injection Baseline:

The rule-based Direct Injection baseline deterministically inserts a predefined payload into SKILL.md without hiding it in a helper artifact or using LLM-based refinement. It copies the skill directory, locates the document, and inserts the payload after front matter or at the document start.

Baseline Definition

  • Direct Injection uses a deterministic template rather than an LLM to insert a predefined attack payload into the original SKILL.md.It omits payload understanding, helper-script hiding, constrained generation, and execution-trace feedback.Source: 8) Rule-Based Direct Injection Baseline · p. 19

Baseline Procedure

  • The baseline selects a predefined payload or script from a target-behavior pool and uses the instruction injection layer.The modified skill directory contains the original auxiliary artifacts while only SKILL.md is changed.Source: 8) Rule-Based Direct Injection Baseline · p. 19
  • If SKILL.md has YAML front matter, the baseline inserts the payload immediately after its closing delimiter; otherwise, it inserts the payload at the beginning.The procedure copies the skill directory, reads the document, performs the conditional insertion, and writes the modified file.Source: 8) Rule-Based Direct Injection Baseline · p. 19

Comparison with SKILLJECT

  • The resulting artifact is a directly injected skill document rather than a skill with a hidden helper artifact and feedback-optimized documentation.This distinction separates the baseline from SKILLJECT’s coordinated artifact and instruction channels.Source: 8) Rule-Based Direct Injection Baseline · p. 19

B. The Algorithm of The Proposed Method

Algorithm 1 generates a poisoned skill by hiding a helper-script payload, applying a front-loaded inducement, and iteratively refining the skill through task execution and evaluation feedback.

Initialization

  • Algorithm 1 initializes a poisoned-skill construction process with a hidden helper-script payload, a script name, task distribution, inducement constraint, and refinement budget.The stated output is a poisoned skill S∗.Source: B. The Algorithm of The Proposed Method · p. 19

Iterative refinement

  • The algorithm iterates over refinement rounds, generating candidate skills and sampling task batches for downstream execution.The supplied pseudocode shows a loop from k = 0 to K and task-batch sampling within each round.Source: B. The Algorithm of The Proposed Method · p. 19

Selection and termination

  • Execution traces are accumulated across rounds, and the algorithm returns the current best poisoned skill when the target condition is reached.The pseudocode records tuples containing tasks, outputs, and evaluation signals, then returns S∗ when qk = 1.Source: B. The Algorithm of The Proposed Method · p. 19

C. Dataset and Task Construction

The benchmark pairs 100 real-world skills with corresponding benign tasks, while SNS evaluates whether modified SKILL.md files remain structurally and semantically natural.

Dataset organization

  • The main benchmark contains 100 benign, self-contained skill packages centered on SKILL.md files and paired with task folders of matching names.Task folders contain instruction.md and may include workspace files required by the task.Source: 1) Dataset Organization · p. 19, Table IX · p. 20

Skill composition

  • 95 of 100 skills include auxiliary artifacts, 92 include code or configuration artifacts, and each package contains 6.93 files on average.Artifact types include scripts, configuration files, documentation, templates, and resources.Source: 2) Benign Skill Collection · p. 20

Usage domains

  • The collected skills cover software development and DevOps, data and financial analysis, and content creation and marketing scenarios.The supplied passage introduces four broad domains, but the fourth domain is truncated in the provided text.Source: 2) Benign Skill Collection · p. 20

Task construction

  • Each skill is evaluated with one corresponding benign downstream task that explicitly requests its legitimate functionality when relevant.The task-driven setting avoids forcing poisoned skills into unrelated tasks.Source: 3) Task Construction Protocol · p. 20

Task statistics

Structural naturalness evaluation

  • SNS uses section matching and an LLM judge to assess placement, heading, style, relevance, and overall naturalness of changed sections.Sections are marked changed when best body similarity falls below θ = 0.85; unchanged documents receive SNS = 1.0.Source: 1) Motivation · p. 20, 2) Changed-Section Identification · p. 21

4) Holistic Document-Level Judge:

The holistic judge evaluates whether all document changes collectively blend into the original skill, while the experiments also standardize security-directed execution and scanner-based detection.

Holistic document-level judge

  • The holistic judge receives the original and modified documents plus a summary of changes, then evaluates their collective integration.Its criteria address semantic relevance, disguise quality, and inverse suspicion.Source: 4) Holistic Document-Level Judge · p. 21

Holistic SNS

  • The holistic SNS score combines semantic relevance, disguise quality, and inverse suspicion into a normalized document-level measure.The final SNS combines local structural and holistic semantic naturalness, with λ = 0.6 weighting local consistency more heavily.Source: 4) Holistic Document-Level Judge · p. 21, 5) Final SNS Score · p. 21

Aggregation and consistency

Instruction-level defense

  • The instruction-level defense requires agents to read every referenced script, inspect harmful patterns, and execute only scripts that are clearly safe and directly required.The wrapped prompt places a security preamble before the task and a reminder afterward.Source: 1) Instruction-Level Prompt Defense · p. 22

Security checks

  • The security checks cover data exfiltration, privilege escalation, backdoor injection, unauthorized file writes, and suspicious external operations.Detected harmful patterns trigger refusal with an explanation of the identified danger.Source: 1) Instruction-Level Prompt Defense · p. 22

Scanner evaluation

  • Scanner-based vetting counts a package as detected only when a scanner explicitly flags it as malicious, unsafe, suspicious, or requiring blocking.Uncertain outputs are excluded unless the scanner recommends rejection or blocking.Source: 2) Scanner-Based Skill Vetting · p. 22

Related papers

Loading 2602.14211v3…