PaperlayerView exact revision on arXiv

Source-linked AI summary

Surviving the Router: Optimizing Skill Injections for Retrieval and Execution

Haneen Najjar, Luca Scionis, Haritz Puerto, Sahar Abdelnabi

arXiv:2610.08098v1cs.CRcs.LG

TL;DR

Prior evaluations often assume that a poisoned skill is already selected, leaving its retrieval under competitive routing untested. CORSA optimizes injected skills across related-task clusters, first for retrieval and then for end-to-end execution. The experiments report improved retrieval and attack success while preserving utility, with transfer across routers and models.

  • Problem

    Prior skill-injection evaluations often assume the poisoned skill is selected, rather than testing whether it is retrieved in a multi-skill setting.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    CORSA optimizes injected skills across clusters of related tasks, first for retrieval and then for end-to-end payload execution.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    CORSA substantially increases retrieval and attack success, preserves benign-task utility, and transfers across victim models, routers, and agent scaffolds.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Skill routing is a key part of the threat model for evaluating skill-injection attacks in realistic multi-skill settings.

    Source: §7 · p. 10

    The threat model assumes the attacker controls a skill while preserving its benign function, and the victim directory contains a relevant benign alternative.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

AI agents increasingly rely on modular third-party "skills" that are dynamically selected by skill routers to execute complex tasks. While recent studies highlight the threat of prompt injections embedded in these skills, existing evaluations often assume settings where the malicious skill is already selected for execution. We show that this assumption can substantially overestimate attack success. In realistic multi-skill environments, injected skills must first compete for retrieval, reducing the effective attack success rate (ASR) of existing injections by 87-97%. To address this limitation, we introduce CORSA (Cluster Optimization for Router-Aware Skill Attacks), a router-aware attack that optimizes skill injections for both retrieval and execution across clusters of related tasks. We evaluate skill injection attacks under router-managed multi-skill settings by extending the benchmark introduced by SkillRouter with eight malicious payload categories. CORSA uses successive optimization stages to first improve retrieval and then optimize end-to-end attack success, while we evaluate user utility and injection naturalism separately. Our experiments show that CORSA substantially improves both retrieval and end-to-end attack success over existing skill injections while preserving user utility, and that the resulting attacks transfer across different router architectures and LLM backbones.

1 INTRODUCTION

Skill routers add a retrieval step that prior skill-injection evaluations often ignored, potentially preventing a poisoned skill from being selected. CORSA optimizes injections for retrieval and execution, and the reported experiments show improvements in attack effectiveness while preserving benign-task utility.

1 INTRODUCTION

  • Prior evaluations assume poisoned skills are selected and executed, overlooking the router’s role in retrieval.If a poisoned skill is not retrieved, its payload cannot be attempted.Source: §1 · p. 1, Figure 1 · p. 2
  • CORSA optimizes injected skills for both router retrieval and payload execution in realistic multi-skill settings.The evaluation extends the SkillRouter benchmark with eight malicious payload categories and separately assesses benign-task utility and injection naturalism.Source: §1 · pp. 1–2
  • The paper evaluates retrieval, end-to-end attack success, benign-task utility, and injection naturalism as distinct concerns.The benchmark is extended with eight malicious payload categories.Source: §1 · pp. 1–2
  • Conventional injections are rarely selected by skill routers, substantially reducing their effective attack success rate.CORSA maintains high retrieval rankings while preserving benign task utility, transfers across routers and LLM backbones, and benefits from optimization over related-task distributions.Source: §1 · p. 2

2 RELATED WORK

Prior skill-injection work has studied harmful instructions, automated payload execution, and attacks on retrieval, but these lines of work often treat retrieval and execution separately. CORSA brings competitive routing and payload execution together in end-to-end evaluation.

2 RELATED WORK

  • Earlier work established skill files as an attack surface and benchmarked skill-based prompt injections across skills, tasks, and agent systems.Prompt-injection benchmarks also study attacks on tool-using agents.Source: §2 · p. 2
  • Automated skill attacks have largely focused on payload execution once a skill is available, without testing whether a router selects it.SKILLJECT refines injected skills using execution feedback; other work targets unsafe prompts, examples, skill composition, or bundled poisoned models.Source: §2 · pp. 2–3
  • Retrieval attacks manipulate tool or skill selection, while SkillRouter highlights difficulty selecting among large pools of similar skills.CORSA combines retrieval competition with payload execution rather than assuming the injected skill is selected.Source: §2 · p. 3
  • Existing defenses inspect skills before installation or restrict runtime permissions, and injection wording can affect attack success.CORSA evaluates whether an injected skill survives competitive routing and still executes its payload.Source: §2 · p. 3

3 METHOD

The method defines a competitive routing threat model and optimizes one injected skill across clusters of related tasks. Its two stages first improve retrieval and then optimize for retrieval plus payload execution.

3.1 THREAT MODEL

  • A skill injection pairs a malicious instruction with a payload while retaining the benign skill’s advertised function as a usable drop-in replacement.The attacker controls the injected skill’s name, description, body, and shipped artifacts, while the payload is fixed.Source: §3.1 · p. 3
  • The victim’s router ranks skills for a task, and the agent loads a selected candidate before planning and acting.The directory must include a benign skill relevant to the task, so the injected skill must displace a legitimate alternative.Source: §3.1 · p. 3
  • The attacker is assumed to publish skills to a marketplace and have query access to router scores and offline access to the victim model.The attacker cannot access model weights or modify other skills, the system prompt, scaffold, or user task.Source: §3.1 · pp. 3–4

3.2 SKILL INJECTION GENERATION

  • For each task cluster, the method optimizes one injected skill across related tasks rather than tuning it to an individual query or all tasks.Clusters typically contain 3–13 tasks from one category, whose shared domain vocabulary can make one skill description relevant to the group.Source: §3.2 · p. 4
  • Attack success requires both top-ranked retrieval and execution of the payload’s helper script.Retrieval rewards task-relevant content, while execution requires clear instructions to invoke the helper script.Source: §3.2 · p. 4
  • CORSA uses two cluster-level optimization stages: Stage A rewards retrieval, and Stage B optimizes end-to-end attack success from the Stage A winner.The payload remains fixed while the skill’s name, description, and body are editable.Source: §3.2 · p. 4, Figure 2 · p. 5

4 EXPERIMENTAL SETUP

The experimental setup evaluates router-aware skill injections in multi-skill settings, using staged optimization, clustered tasks, multiple payloads, and separate measures of attack success and utility.

4.1 ATTACK GENERATOR

  • GEPA optimizes injected skill text using task-level retrieval feedback and, when evaluated, execution feedback, without gradients through the router or victim agent.Optimization proceeds in two successive stages because end-to-end success requires both top-ranked retrieval and payload execution.Source: §4.1 · p. 5

4.2 BENCHMARK

  • The setup evaluates skill injections across 75 prompts and approximately 80K skills, with a hard setting adding 780 topically related distractors.The prompts include 24 single-skill and 51 multi-skill tasks.Source: §4.2 · p. 5
  • The benchmark groups queries into eight semantic clusters and evaluates one poisoned skill per cluster against 2,000 benign skills for each of eight malicious payload categories.All payloads use an indirect attack in which the poisoned skill induces execution of an associated helper script.Source: §4.2 · pp. 5–6
  • SKILLJECT is the baseline, reproduced with the same eight payloads and multi-skill evaluation setting as CORSA.The benchmark extends SkillRouter for settings where injected skills must compete for retrieval and execution.Source: §4.2 · pp. 5–6

4.3 EVALUATION METRICS

  • Hit@1 measures first-place retrieval, ASR requires both first-place retrieval and payload execution, and user utility measures completion of the original task.The study also scores injection stealthiness with a GPT-5.4 judge; metrics are aggregated across clusters with task-count weighting.Source: §4.3 · p. 6

4.4 MODELS, SCAFFOLD, AND ROUTERS

  • Experiments use GPT-5.4 for attacker, victim, and utility judge, test five routers spanning lexical, dense, and two-stage retrieval, and evaluate across five victim LLMs.A DeepSeek-V4-Pro attacker is also tested while keeping the victim and judge fixed at GPT-5.4; victim evaluation includes models unseen during optimization.Source: §4.4 · p. 6

5 RESULTS

The results evaluate CORSA in routed multi-skill settings, including its main performance, generalization to task variations, and transfer across victims, scaffolds, and routers. Across these settings, CORSA improves on the router-agnostic baseline, though performance varies by task and target.

5 RESULTS

  • The evaluation also examines retrieval with similar distractor skills and design choices involving optimization granularity, staged optimization, and stealthiness.These analyses are reported in Sections B and C.Source: §5 · p. 6

5.1 MAIN RESULTS

  • SkillJect falls from the benign skill’s 100% Hit@1 to roughly 11% on both attacker models; CORSA more than doubles Hit@1 and similarly improves ASR while maintaining comparable utility and higher naturalism.These trends hold across eight payloads and both attacker models.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 GENERALIZATION TO TASK VARIATIONS

  • On paraphrase and synthetic task variations, CORSA retains 11.5% and 5.4% ASR, respectively, while SkillJect falls below 1.0 ASR.The optimized skills are frozen and evaluated without re-optimization; the decline is more pronounced for synthetic tasks.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 TRANSFERABILITY ACROSS MODEL VICTIMS, SCAFFOLDS AND ROUTERS

  • Across victim models, CORSA’s ASR stays above 10% except on Qwen3.8-27B and exceeds the SKILLJECT baseline in every case.Hit@1 is unchanged across victims because the router is fixed, while ASR and utility vary by victim.Source: §5.3 · p. 8, Table 2 · p. 8
  • With skills optimized using Codex and held fixed on OpenHands, CORSA achieves higher Hit@1 and ASR than SKILLJECT while preserving similar utility.The evaluation covers all 75 tasks across eight payloads.Source: §5.3 · p. 8, Table 3 · p. 8
  • Cross-router transfers are asymmetric: source routers yield the highest Hit@1, but not always the highest ASR, and Qwen3-8B has the lowest Hit@1 across all three sources.User utility varies across routers without directly tracking ASR.Source: §5.3 · p. 8, Table 4 · p. 9

6 DEFENSES

The defense evaluation tests pre-retrieval skill scanners on malicious and benign packages. Static scanners miss all tested attack packages; LLM augmentation improves recall, but detection depends on the scanner–model combination and remains incomplete.

Detection Results and Deployment

  • Static scanners miss all attack packages, and the strongest LLM-augmented configuration still misses five of 24 CORSA packages.The evaluation covers a single payload; scanners inspect complete packages, and benign-package false-positive rates are measured on the same 2,000 packages.Source: §6 · p. 9
  • LLM augmentation improves recall, with effectiveness depending strongly on the scanner–model combination; Qwen3.8-27B outperforms GPT-5.4 with both scanners.SkillSpector with Qwen has the highest recall, while Cisco with Qwen offers slightly lower recall at a lower FPR, with some invalid outputs.Source: §6 · p. 9
  • Detection rates are broadly similar across attack methods, and screening reduces but does not eliminate the threat because admitted skills may execute repeatedly.The authors relate the similar detection rates to shared indirect helper-script payloads and CORSA’s focus on retrieval and execution rather than scanner evasion.Source: §6 · p. 9

7 CONCLUSION

The conclusion argues that realistic skill-injection evaluation must account for retrieval before execution. CORSA improves retrieval and attack success over router-unaware attacks, and the paper also finds generalization, transfer, and incomplete scanner detection.

  • In realistic skill marketplaces, poisoned skills must compete with benign skills for retrieval before their payloads can execute.Source: §7 · pp. 9–10
  • By optimizing for retrieval, CORSA substantially increases retrieval and attack success, and its optimized skills generalize and transfer better than the router-unaware baseline.The stated transfer settings include victim models, routers, and agent scaffolds.Source: §7 · p. 10
  • LLM-augmented scanners improve detection but do not identify all malicious packages, so the authors identify skill routing as part of the threat model for realistic evaluations.Source: §7 · p. 10

AI USE STATEMENT

The authors report using AI tools for language polishing, literature retrieval, discovery, and synthetic-data generation, with outputs reviewed and verified before inclusion.

  • Authors reviewed and verified AI-assisted outputs and take responsibility for the paper’s final content, claims, experiments, and results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The authors describe the work’s dual-use implications and state that experiments were conducted in controlled settings. The appendix reports per-cluster and per-payload results for two attacker models.

ETHICS STATEMENT

  • The attacks have potential dual-use implications, and the stated goal is to support more realistic security evaluation of skill-enabled agents.The experiments did not target real users or production systems and were not deployed to public skill marketplaces or used against third-party systems.Source: ETHICS STATEMENT · p. 10

A PER-CLUSTER AND PER-PAYLOAD MAIN RESULTS

A.1 RESULTS WITH GPT-5.4 AS THE ATTACKER MODEL

A.2 RESULTS WITH DEEPSEEK-V4-PRO AS THE ATTACKER MODEL

B ROBUSTNESS AGAINST DISTRACTOR SETTINGS

The analyses test CORSA under distractor competition and examine its optimization design. CORSA outperforms SKILLJECT in the Hard setting, while the design analyses report trade-offs among optimization granularity, staging, and stealthiness.

B ROBUSTNESS AGAINST DISTRACTOR SETTINGS

C.1 OPTIMIZATION GRANULARITY AND STAGING

  • Task-Wise Optimization performs worse than CORSA in Hit@1 and ASR, while Joint Optimization performs worse particularly in ASR; user utility remains similar.The analysis attributes the task-wise gap to poor transfer across related tasks and finds separate retrieval and execution optimization more effective than optimizing both together.Source: §C.1 · p. 14, Figure 4 · p. 16

C.2 STEALTHINESS AS AN OPTIMIZATION OBJECTIVE

  • Adding a stealthiness reward improves Hit@1 but reduces end-to-end attack success and slightly decreases user utility.The results show that blending injected content more naturally with a skill can help it pass the router without making payload execution more likely.Source: §C.2 · pp. 14–15, Figure 5 · p. 16

D PRE-RETRIEVAL ANALYSIS

The defense evaluation tests two skill scanners on malicious packages under specified static and LLM-augmented configurations. It reports recall and false-positive rate using a defined set of attack and benign packages.

Scanners

  • The evaluation tests Cisco Skill Scanner and NVIDIA SkillSpector in static and LLM-augmented modes, with Qwen3.8-27B and GPT-5.4 as backend models.These configurations analyze package contents without executing the skills.Source: D PRE-RETRIEVAL ANALYSIS · p. 16

Evaluation protocol

  • The evaluation uses 24 CORSA packages across three router configurations and eight clusters, compared with eight SKILLJECT packages.It tests a single indirect payload and uses scanner-specific flagging thresholds that are not tuned on the dataset; invalid outputs count as not flagged.Source: D PRE-RETRIEVAL ANALYSIS · p. 16
  • Recall is the fraction of malicious packages flagged, while FPR is the fraction of benign packages flagged.The same 2,000 benign packages are used across attack conditions, so FPR is shared across attack methods for each scanner configuration.Source: D PRE-RETRIEVAL ANALYSIS · p. 16

Related papers

Loading 2610.08098v1…