PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents

Xiaojun Jia, Jie Liao, Simeng Qin, Jindong Gu, Wenqi Ren, Xiaochun Cao, Yang Liu, Philip Torr

arXiv:2602.14211v3cs.CRcs.AI

TL;DR

Skill-based prompt injection remains difficult to trigger reliably because explicit malicious instructions can be rejected or ignored, despite skills being loaded as trusted executable capability packages. SkillJect addresses this gap with coordinated artifact hiding, front-loaded documentation inducement, and closed-loop trace-guided refinement. Across its evaluations, it substantially outperforms naive direct injection and prior manual attacks, while its scope is limited to settings where the poisoned skill is selected and payload execution is measured rather than complete downstream impact.

  • Problem

    Existing skill-based attacks are often manual and brittle, while poisoned skills can exploit trusted packages of instructions and executable artifacts to steer agent tool use.

  • Method

    SkillJect hides payloads in auxiliary helper scripts, rewrites SKILL.md with front-loaded prerequisite instructions, and iteratively refines the documentation using Attack, Victim, and Evaluate Agents.

  • Results

    SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks across platforms, backend LLMs, and attack categories.

  • Takeaways & Limitations

    Poisoned skills are a persistent threat in reusable skill ecosystems because apparently coherent documentation can induce execution of hidden helper artifacts.

  • Takeaways & Limitations

    The evaluation assumes the poisoned skill is selected and measures hidden-helper execution, not whether every execution produces a fully realized downstream consequence.

Abstract

from arXiv · show

Agent skills extend LLM agents with task-specific instructions, executable scripts, and auxiliary resources, improving reusability but creating a new supply-chain attack surface. A malicious or compromised skill can be repeatedly loaded as trusted guidance and steer downstream tool use. Existing skill-based prompt-injection attacks are often manual and brittle, because explicit malicious instructions are rejected or ignored when they are not aligned with the original workflow. We propose SkillJect, the first automated framework for generating poisoned skills against skill-enabled agent systems. SkillJect uses two coordinated channels. In the artifact channel, it hides the payload inside an auxiliary helper script. In the instruction channel, it rewrites SKILL.md with a front-loaded inducement strategy, placing injected content at the beginning and framing the helper script as a mandatory prerequisite or initialization step. The rewritten instruction explicitly references the helper-script path and provides an executable example command, making the helper appear to be a legitimate setup step before normal skill operations. SkillJect further adopts a closed-loop multi-agent process to improve attack effectiveness. An Attack Agent generates poisoned skills, a Victim Agent executes downstream tasks with the poisoned skill, and an Evaluate Agent inspects execution traces to determine whether the hidden payload was executed. The Attack Agent then uses this feedback to diagnose failure causes and rewrite SKILL.md, while keeping the payload fixed. Experiments across skill-enabled platforms, backend LLMs, and attack categories show that SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks, highlighting poisoned skills as a persistent threat in reusable skill ecosystems.

I. INTRODUCTION

Agent skills package trusted instructions with executable artifacts, creating a supply-chain surface for prompt injection. SkillJect automates poisoned-skill generation by combining hidden helper scripts, front-loaded inducement instructions, and trace-guided refinement, outperforming direct and manual attacks in evaluation.

  • Motivation: Agent skills extend tool-augmented agents with modular, task-specific instructions, scripts, and resources that are loaded on demand.This packaging model supports reusable capabilities across diverse tasks and environments.
  • Threat: Poisoned skills can covertly steer tool use toward sensitive-file access, project modification, or other unauthorized actions when imported and executed.Impact depends on runtime permissions, sandboxing, network access, and user approval policies.
  • Threat: Publicly collected malicious skills rarely trigger their intended behavior because explicit instructions are often rejected or ignored by agents.The figure distinguishes explicit refusal from failure to follow the instruction after skill execution.
  • Approach: SkillJect uses an Attack Agent, Victim Agent, and Evaluate Agent in a closed loop to generate poisoned skills and refine them from execution traces.The evaluator checks whether the hidden payload executes, and the attack process uses that feedback for further rewriting.
  • Approach: SkillJect coordinates hidden auxiliary payloads with front-loaded SKILL.md instructions that present a helper script as a required workflow prerequisite.The injected documentation references the helper path and executable command, making the script appear operationally legitimate.
  • Results: SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks across platforms, backend models, and attack categories.The reported experiments support improved effectiveness in the evaluated benchmark.

C. Skill Security

SkillJect treats skills as executable security boundaries and constructs poisoned packages whose documentation and artifacts jointly induce helper-script execution. Its closed-loop agents use execution traces to refine the documentation while preserving the payload.

  • Skill security: Skills resemble software plugins because they bundle trusted instructions, executable scripts, configuration, and auxiliary resources.This packaging creates a distinct supply-chain surface beyond passive prompt context.
  • Security gap: Existing manual attacks are brittle because explicit malicious intent may be rejected and weakly integrated instructions may be ignored during downstream execution.This motivates workflow-compatible poisoning rather than overtly harmful documentation.
  • Closed-loop refinement: The Victim Agent produces execution traces, while the Evaluate Agent checks hidden-helper execution and returns feedback for iterative documentation refinement.The Attack Agent keeps the artifact payload fixed while rewriting the inducement.
  • Attack mechanism: SkillJect hides behavior-specific payloads in auxiliary artifacts and inserts front-loaded inducements into the documentation channel.The resulting package preserves visible documentation coherence while enabling trace-level effects through the artifacts.
  • Scope: The attack is evaluated only when the poisoned skill is selected and counts success at hidden-helper execution rather than full downstream harm.Router compromise is not assumed, and downstream consequences may depend on deployment controls.

C. Generating injected skills with SKILLJECT

SKILLJECT generates poisoned skills by hiding malicious functionality in auxiliary artifacts while rewriting SKILL.md to induce execution through a front-loaded prerequisite. A closed-loop process then refines the documentation using victim execution traces while keeping the payload fixed.

  • Trace-level objective: SKILLJECT evaluates success with a trace-level predicate that checks whether the victim agent explicitly executed the hidden helper script.Execution traces include tool and command executions, file operations, and intermediate or final outputs.
  • Payload understanding: The Attack Agent analyzes the helper script to infer its functionality, inputs, outputs, execution stage, and plausible workflow role before rewriting the documentation.This payload understanding guides construction of an operationally aligned injected document.
  • Constrained generation: The front-loaded strategy places a mandatory helper-script instruction at the beginning of SKILL.md, references its path, and supplies an executable command before normal operations.This presentation is intended to increase salience and execution priority while making the helper appear to be a required initialization step.
  • Feedback-driven refinement: The hidden payload is generated once and held fixed throughout refinement, isolating improvements in inducement quality from changes in payload capability.This design focuses the optimization process on how the payload is presented to the victim agent.
  • Feedback-driven refinement: The framework uses an iterative refinement loop in which the Evaluate Agent diagnoses victim traces and the Attack Agent revises the documentation using stored feedback.The feedback can classify outcomes such as ignored, refused, partially triggered, or incorrectly executed.

IV. EXPERIMENTS

Experiments evaluate SKILLJECT across four high-severity attack behaviors, multiple backend models, and two skill-enabled platforms. It substantially outperforms direct injection, achieving high attack success rates across categories and platforms while showing lower success on Claude models.

  • Experimental setup: The benchmark contains 100 agent skills from ClawHub and evaluates information disclosure, privilege escalation, unauthorized write, and backdoor injection behaviors.Claude Code is the main victim platform, with OpenClaw used to assess cross-platform transferability.
  • Results on Claude Code: SKILLJECT achieves 80.7% average overall ASR on Claude Code, whereas naive direct injection achieves 0.0% across all attack categories and four backend models.The comparison indicates that explicit malicious instructions alone were ineffective in this evaluation.
  • Results on Claude Code: Across attack categories on Claude Code, SKILLJECT reaches 78.8% INFODISC, 81.0% PRIVESC, 83.5% UNAUWRI, and 79.5% BACKDOOR average ASR.All four category averages contrast with the naive baseline’s 0.0% results.
  • Results on Claude Code: Across Claude Code backends, SKILLJECT’s overall ASR is 97.2% on GLM-4.7, 94.7% on MiniMax-M2.1, 83.8% on GPT-5-mini, and 47.0% on Claude-Sonnet-4.6.Claude-Sonnet-4.6 is the least susceptible backend among those reported, but the attack still succeeds in a non-trivial fraction of cases.
  • Results on OpenClaw: On OpenClaw, SKILLJECT achieves 80.5% average overall ASR while the naive baseline again records 0.0% across all four backend models.Reported backend ASRs are 97.0% on GLM-4.7, 95.0% on MiniMax-M2.1, 87.0% on GPT-5-mini, and 43.0% on Claude-Sonnet-4.6.

3) Comparison on more advanced models:

Across newer models, platforms, and prior attack baselines, SKILLJECT remains substantially more effective than naive or manual injection, although transferability varies by backend and task.

  • Advanced backend models: 68.4% average ASR across six newer frontier backends shows SKILLJECT remains effective beyond earlier evaluated models, while GPT-5.4 and Claude-Opus-4.6 yield 28.7% and 29.2%.The advanced-model evaluation used 30 sampled skills/tasks and the same four behavior categories.
  • Advanced backend models: 0.0% ASR on all six advanced backends for Naive direct injection confirms that explicit malicious insertion is unreliable against these models.The comparison uses the same evaluation protocol and target behavior categories as SKILLJECT.
  • Manual attack comparison: 69.1% overall ASR demonstrates SKILLJECT’s advantage over prior manually designed malicious skills across the evaluated comparison.The improvement is attributed to automated inducement generation and feedback-driven refinement rather than stronger payload scripts.
  • Manual attack comparison: SKILLJECT raises ASR from 40.0% to 82.2% on GLM-4.7, from 35.6% to 80.9% on MiniMax-M2.1, and from 26.7% to 64.4% on GPT-5-mini.The same benign skills and malicious payloads were used for both methods.
  • Cross-model transferability: Cross-model transfer remains non-trivial, with transferred skills achieving 49.8% and 35.8% overall ASR on two target backends without target-specific refinement.The supplied passage identifies these results as transfer outcomes and separately reports success across all four target behaviors on Claude-Sonnet-4.6.
  • Cross-task transferability: Cross-task evaluation reuses one poisoned skill across 20 additional LLM-generated instructions for the same skill, without regeneration or refinement.This setup tests whether a skill generated for one source task transfers to other associated task instructions.
  • Cross-model transferability: Compared with 97.2% on GLM-4.7, transferred performance decreases by 17.4, 47.4, and 61.4 percentage points on MiniMax-M2.1, GPT-5-mini, and Claude-Sonnet-4.6.The results suggest workflow-level attack patterns transfer, while model-specific refinement remains important for maximizing effectiveness.
  • Generated-injection design: The generated documentation uses front-loaded, workflow-compatible inducements that present helper execution as a required prerequisite or initialization step.Qualitative examples include camouflaged prerequisite sections and emphasized initialization warnings.

6) Cross-task Instruction Transferability:

SKILLJECT improves injected-skill execution through iterative trace-guided refinement and documentation that remains structurally natural, while testing transfer across tasks and behavior categories.

  • 6) Cross-task Instruction Transferability:: Cross-task evaluation reuses each poisoned skill on 20 additional task instructions associated with the same skill, without regeneration or refinement.The evaluation isolates transferability from repeated attack generation.
  • 6) Cross-task Instruction Transferability:: The cross-task setting reflects real use in which one skill may serve multiple related user tasks, such as weather queries across different dates or locations.The poisoned skill is generated from a specific source task instruction and then evaluated on additional instructions.
  • 2) Effect of Feedback-Driven Refinement:: Figure 6 evaluates whether feedback-driven refinement improves attack success across backend models.The refinement loop is assessed as an iterative process rather than a single generated skill.
  • 1) Qualitative Analysis of Generated Injections:: The qualitative examples show helper execution framed either as a camouflaged prerequisite section or as an emphasized initialization warning before the main workflow.Both forms preserve the same underlying mechanism: placing the inducement before normal skill operations.
  • 2) Effect of Feedback-Driven Refinement:: After five refinement iterations, ASR reaches 97.2% on GLM-4.7, 94.6% on MiniMax-M2.1, and 83.8% on GPT-5-mini.Claude-Sonnet-4.6 improves more slowly and reaches 52.8% after the reported iterations.
  • 2) Effect of Feedback-Driven Refinement:: The Evaluate Agent classifies ignored, refused, partially triggered, or incorrectly executed attempts, and the Attack Agent revises documentation while keeping the hidden payload fixed.The loop uses execution traces to improve the inducement rather than regenerating the payload.
  • 3) Structural Naturalness Score of Injected Skills:: SKILLJECT generates more structurally and stylistically natural injected skills than the Naive baseline across the four target behavior categories.Structural Naturalness Score is computed with an LLM-as-a-judge comparison of benign and poisoned SKILL.md files.
  • 3) Structural Naturalness Score of Injected Skills:: For Backdoor Injection, SNS increases from 0.223 to 0.429, while Information Disclosure rises from 0.318 to 0.415.These category-specific increases support the broader finding that injected documentation better preserves the original workflow.

D. Robustness against Potential Defenses

The paper evaluates prompt-level and scanner-based defenses, plus component ablations and deployment safeguards. Results show partial mitigation, while attack success remains substantial and depends on execution context and system configuration.

  • Potential Defenses: The study evaluates runtime instruction defenses and pre-installation skill scanners as practical mitigations for poisoned skills.The runtime defense requires inspection of referenced scripts, while scanners inspect documentation, metadata, auxiliary scripts, and behavior patterns.
  • Instruction-Level Defense: 49.0 percentage points: the instruction-level defense reduces overall ASR from 97.3% to 48.3% across four target behaviors.The reduction is strongest for recognizable exfiltration behavior but does not eliminate workflow-compatible attacks.
  • Instruction-Level Defense: 66% and 77% ASR remain for Privilege Escalation and Unauthorized Write, respectively, after applying the prompt defense.These results show that explicit safety instructions are insufficient against attacks compatible with the requested workflow.
  • Ablation Study: 97.2% overall ASR falls to 91.0% without payload understanding, 78.8% without front-loaded inducement, and 80.5% without refinement feedback.The larger drops from removing front-loaded inducement and feedback indicate that instruction placement and trace-based revision materially contribute to attack success.
  • Scope and Limitations: Attack success is measured as hidden-helper execution, not guaranteed downstream harm, which also depends on permissions, sandboxing, network access, and approval mechanisms.The evaluation uses a post-selection setting and may not generalize across all skill ecosystems, agent scaffolds, prompts, permissions, or deployment policies.
  • Responsible Evaluation: Experiments use isolated Docker environments, controlled tasks, dummy secrets, and synthetic targets without attacking real users or production systems.The authors state that directly reusable malicious payloads and end-to-end attack scripts will not be released.

APPENDIX

The appendix documents SKILLJECT’s prompt-driven implementation for analyzing helper scripts and integrating them into skill documentation. It specifies structured analysis fields, attack-type-dependent descriptions, and constrained documentation-generation inputs.

  • Payload Understanding: The payload-understanding module analyzes a helper script and returns structured JSON for the subsequent generation stage.The analysis includes technical behavior, a plausible legitimate purpose, and a natural SKILL.md description.
  • Payload Understanding: The appendix’s analysis prompt requests the attack type, script content, technical behavior, legitimate purpose, and natural documentation description.It also asks for integration guidance that avoids suspicious phrasing.
  • Constrained Generation: The constrained-generation prompt places the helper script at the beginning of SKILL.md as a mandatory prerequisite before other operations.It specifies the script path, purpose, required execution level, description, and an example command such as bash resources/{script_path}.
  • Fallback Descriptions: Attack-type-dependent fallback descriptions portray scripts as environment, dependency, package-installation, or documentation-fetching utilities.The defaults cover information disclosure, privilege escalation, unauthorized write, and backdoor injection categories.
  • Feedback-Driven Refinement: The refinement process selects prompts according to diagnostic feedback returned by the evaluation stage.The appendix identifies a common script-analysis block and a documentation-enhancement framing for refinement.

3) Feedback-Driven Refinement Prompts:

The refinement prompts adapt SKILL.md after rejected, ignored, or unsuccessful execution attempts. They progressively emphasize mandatory placement, neutral framing, direct commands, verification, and error handling.

  • Feedback Inputs: Refinement prompts incorporate previous-attempt status, root cause, evidence, and an improvement strategy into the next documentation rewrite.The prompts distinguish rejected, ignored, and attempted-but-failed execution modes.
  • Narrative Refinement: The prompts also direct the generator to use neutral technical language, avoid sensitive keywords, emphasize legitimate uses, and make the script sound routine.These changes target narrative framing rather than changing the hidden helper payload.
  • Mandatory Integration: The prompts require the helper script to be integrated as a mandatory first step with a specified path and legitimate-purpose description.They repeatedly frame execution as necessary before normal skill use and preserve the script resource information across iterations.
  • Making Execution Unmissable: When inducement is insufficient, refinement strengthens imperative language, prominent placement, repeated references, and verification requirements.The instructions recommend terms such as MUST and REQUIRED and place the script call at the beginning or end of the skill instructions.
  • Failure Handling: For execution failures, refinement verifies the script path and permissions, simplifies commands, adds debug output, and checks errors explicitly.The prompts include exit-code checks, output display, file-state verification, existence checks, and permission checks.

8) Rule-Based Direct Injection Baseline:

The rule-based Direct Injection baseline deterministically inserts a predefined payload into SKILL.md without hiding it in a helper artifact or using LLM-based refinement. It copies the skill directory, locates the document, and inserts the payload after front matter or at the document start.

  • Baseline Definition: Direct Injection uses a deterministic template rather than an LLM to insert a predefined attack payload into the original SKILL.md.It omits payload understanding, helper-script hiding, constrained generation, and execution-trace feedback.
  • Baseline Procedure: The baseline selects a predefined payload or script from a target-behavior pool and uses the instruction injection layer.The modified skill directory contains the original auxiliary artifacts while only SKILL.md is changed.
  • Baseline Procedure: If SKILL.md has YAML front matter, the baseline inserts the payload immediately after its closing delimiter; otherwise, it inserts the payload at the beginning.The procedure copies the skill directory, reads the document, performs the conditional insertion, and writes the modified file.
  • Comparison with SKILLJECT: The resulting artifact is a directly injected skill document rather than a skill with a hidden helper artifact and feedback-optimized documentation.This distinction separates the baseline from SKILLJECT’s coordinated artifact and instruction channels.

B. The Algorithm of The Proposed Method

Algorithm 1 generates a poisoned skill by hiding a helper-script payload, applying a front-loaded inducement, and iteratively refining the skill through task execution and evaluation feedback.

  • Initialization: Algorithm 1 initializes a poisoned-skill construction process with a hidden helper-script payload, a script name, task distribution, inducement constraint, and refinement budget.The stated output is a poisoned skill S∗.
  • Iterative refinement: The algorithm iterates over refinement rounds, generating candidate skills and sampling task batches for downstream execution.The supplied pseudocode shows a loop from k = 0 to K and task-batch sampling within each round.
  • Selection and termination: Execution traces are accumulated across rounds, and the algorithm returns the current best poisoned skill when the target condition is reached.The pseudocode records tuples containing tasks, outputs, and evaluation signals, then returns S∗ when qk = 1.

C. Dataset and Task Construction

The benchmark pairs 100 real-world skills with corresponding benign tasks, while SNS evaluates whether modified SKILL.md files remain structurally and semantically natural.

  • Dataset organization: The main benchmark contains 100 benign, self-contained skill packages centered on SKILL.md files and paired with task folders of matching names.Task folders contain instruction.md and may include workspace files required by the task.
  • Skill composition: 95 of 100 skills include auxiliary artifacts, 92 include code or configuration artifacts, and each package contains 6.93 files on average.Artifact types include scripts, configuration files, documentation, templates, and resources.
  • Usage domains: The collected skills cover software development and DevOps, data and financial analysis, and content creation and marketing scenarios.The supplied passage introduces four broad domains, but the fourth domain is truncated in the provided text.
  • Task construction: Each skill is evaluated with one corresponding benign downstream task that explicitly requests its legitimate functionality when relevant.The task-driven setting avoids forcing poisoned skills into unrelated tasks.
  • Task statistics: The task set includes 100 instructions, 71 with auxiliary workspace files, with instruction lengths averaging 26.84 words.Instruction lengths range from 14 to 36 words, and common workspace types include .txt, .py, .csv, .sql, .rs, and .swift.
  • Structural naturalness evaluation: SNS uses section matching and an LLM judge to assess placement, heading, style, relevance, and overall naturalness of changed sections.Sections are marked changed when best body similarity falls below θ = 0.85; unchanged documents receive SNS = 1.0.

4) Holistic Document-Level Judge:

The holistic judge evaluates whether all document changes collectively blend into the original skill, while the experiments also standardize security-directed execution and scanner-based detection.

  • Holistic document-level judge: The holistic judge receives the original and modified documents plus a summary of changes, then evaluates their collective integration.Its criteria address semantic relevance, disguise quality, and inverse suspicion.
  • Holistic SNS: The holistic SNS score combines semantic relevance, disguise quality, and inverse suspicion into a normalized document-level measure.The final SNS combines local structural and holistic semantic naturalness, with λ = 0.6 weighting local consistency more heavily.
  • Aggregation and consistency: SNS scores are averaged across evaluated skill samples so reported values represent generated poisoned skills rather than a single example.The same judge configuration is used across compared methods and target behavior categories.
  • Instruction-level defense: The instruction-level defense requires agents to read every referenced script, inspect harmful patterns, and execute only scripts that are clearly safe and directly required.The wrapped prompt places a security preamble before the task and a reminder afterward.
  • Security checks: The security checks cover data exfiltration, privilege escalation, backdoor injection, unauthorized file writes, and suspicious external operations.Detected harmful patterns trigger refusal with an explanation of the identified danger.
  • Scanner evaluation: Scanner-based vetting counts a package as detected only when a scanner explicitly flags it as malicious, unsafe, suspicious, or requiring blocking.Uncertain outputs are excluded unless the scanner recommends rejection or blocking.
Loading 2602.14211v3…