PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching

Qikai Wang, Yongzhao Zhang, Zhiwei Chen, Yimiao Sun, Jiguo Yu, Xiaosong Zhang

arXiv:2609.02035v1cs.CR

TL;DR

LLM-agent skill selection uses semantic matching, but existing explicit-steering attacks can expose recognizable manipulation signals. This paper introduces ISM, which jointly shapes reusable prompts and target-skill descriptions without explicit selection instructions. Across four domains and eight selector models, ISM raises macro TSR from 15.2% to 63.5% while showing greater implicitness than Explicit Steering. Its claims remain limited to pre-execution selection and do not establish harmful downstream execution.

  • Problem

    Semantic matching between prompts and skill descriptions creates an implicit skill-selection attack surface that explicit-instruction defenses may not recognize.

  • Method

    ISM jointly shapes reusable prompts and target-skill descriptions through task-related cues, technical hooks, and semantic-constrained prompt naturalization.

  • Results

    63.5% macro-averaged TSR under ISM versus 15.2% under Native across four task domains and eight selector models.

  • Takeaways & Limitations

    Pre-execution skill selection is a security boundary that requires assessing relative prompt–metadata compatibility, not only individually malicious-looking content.

  • Takeaways & Limitations

    The paper’s claims are confined to pre-execution selection and do not establish harmful downstream execution by the selected skill.

Abstract

from arXiv · show

Skill selection is a key stage in LLM-agent workflows, determining which installed skill should handle a user request. Existing attacks on this stage primarily rely on explicit prompt injection or instruction-level steering, which can expose recognizable manipulation signals. In this work, we identify a new implicit attack surface for skill selection: even when the user prompt and skill description appear benign in isolation, their semantic relationship can still be strategically shaped to favor an attacker-chosen skill. Based on this observation, we present Implicit Skill-Selection Manipulation via Semantic Matching (ISM), which jointly shapes target-skill metadata and reusable prompts to manipulate skill selection without explicit selection instructions. Specifically, we develop a three-stage strategy to broaden semantic coverage, strengthen target distinctiveness, and preserve natural prompt wording. Across four task domains and eight selector models, ISM increases the average target-selection rate (TSR) from 15.2% to 63.5%. In a matched comparison, ISM achieves a 73.5% TSR, only 9.8 percentage points below Explicit Steering. Human reviewers block ISM in only 2.9% of judgments, versus 91.4% for Explicit Steering, while five LLM-based inspectors pass ISM at an average rate of 82.9%, versus 37.4% for Explicit Steering. Moreover, ISM remains effective against PPL-W, Llama Prompt Guard 2, and PIGuard.

1 Introduction

LLM-agent skill selection relies on semantic matching between user prompts and skill descriptions, creating a pre-execution attack surface. ISM exploits this relationship by coordinating reusable prompts and target-skill metadata through three implicit-manipulation stages, achieving substantial selection gains and inspection resistance.

  • LLM agents select among installed skills by comparing user prompts with natural-language skill metadata before invoking skill-specific backends.
  • Semantic matching can be manipulated even when prompts and skill descriptions appear benign in isolation, biasing selection toward an attacker-controlled skill.
  • 63.5% macro-averaged TSR under ISM versus 15.2% under Native across four task domains and eight selector models.
  • 73.5% TSR in the matched comparison leaves ISM 9.8 percentage points below Explicit Steering’s 83.3%, while remaining substantially more implicit.
  • Human reviewers blocked ISM in 2.9% of judgments, compared with 91.4% for Explicit Steering, while LLM inspectors passed ISM at 82.9% versus 37.4%.
  • ISM coordinates reusable prompts and target-skill descriptions without explicit selection instructions, using task-related cues, technical hooks, and prompt naturalization.

2 Background

Skill selection affects agent security exposure and competition because it determines which installed capability receives a request. Reusable prompts help users specify complex tasks but also introduce third-party wording into selection, while the threat model assumes text-only attacker control and limits scope to pre-execution selection.

  • Skill selection determines which installed skill receives a request before execution, affecting security exposure and invocation opportunities among competing skills.
  • Selected backends enter the normal execution path with limited semantic validation, and deceptive or adversarial outputs may corrupt subsequent agent reasoning or decisions.
  • Skill selection also controls traffic distribution among competing skills, creating potential consequences for usage scale, market share, and monetization.
  • Reusable prompts and mega-prompts reduce effort for complex tasks but allow third-party wording to enter the agent’s skill-selection context.
  • The threat model gives the attacker control over published skill descriptions and reusable prompts in a black-box setting, without model, system-prompt, or framework access.
  • The attack requires an installed target skill, same-domain competitors, and user reuse of a compatible attacker-published prompt, while excluding host compromise and post-selection backend behavior.

3 Methodology and Attack Design

ISM manipulates semantic matching by jointly shaping reusable prompts and target-skill descriptions through task-related cues, distinctive technical hooks, and prompt naturalization. Its design uses semantic similarity to broaden task coverage, favor target-specific evidence, and preserve natural wording.

  • Overall Design: ISM uses three sequential stages to address task coverage, target distinctiveness, and prompt-side conspicuousness while preserving semantic correspondence.The stages are task-related cue construction, customized technical-hook injection, and prompt naturalization.
  • 3.2.1 Semantic Measurement: Semantic matching measures meaning-level correspondence between prompts and skill descriptions rather than relying on surface-word overlap.ISM uses MiniLM embeddings and cosine similarity for repeated phrase-to-description comparisons.
  • 3.2.2 Task-Related Cue Extraction: ISM filters prompt candidates first for recurrence across a scenario, then for semantic relevance to the target task.Frequency captures scenario coverage, while semantic score captures target-task relevance; the retained phrases are prompt-side shared cues.
  • 3.2.2 Task-Related Cue Extraction: Task-related cue construction links diverse prompt formulations to the target by combining shared prompt cues with capability cues extracted from the target description.Description-side capability cues are expressed as corresponding user requirements without introducing a new task.
  • 3.3.1 Technical-Hook Selection: Technical hooks are uncommon, task-consistent expressions supported by the target skill and selected to distinguish it from same-domain competitors.Examples include citation spans or appendix linkage; unrelated details do not improve target selection, while competitor-aligned details perform worst.
  • 3.4.2 Semantic-Constrained Naturalization: Prompt naturalization rewrites only the prompt-side technical hook using a semantic bridge, while retaining its technical form in the target description.The bridge specifies the relation between the hook and its user-level meaning and constrains the rephrasing.

4 Evaluation

The evaluation tests ISM’s effectiveness, generality, implicitness, and resistance to rule-based inspection across multiple domains, selectors, and comparison conditions. ISM broadly increases target selection while remaining less recognizable than Explicit Steering and partially evading evaluated defenses.

  • Experimental Setup: The evaluation spans four domains, eight selector models, and seven-skill candidate pools, using TSR, inspection, and rule-based defense metrics.The setup includes eight LLM selectors and compares Native, ISM, related text controls, Explicit Steering, and customized-prompt conditions.
  • Effectiveness: 63.5% macro TSR under ISM versus 15.2% under Native demonstrates a broad selection shift across selectors and domains.All eight selectors show positive macro-level gains, 62 of 64 model–target pairs improve, and all four domains show positive effects.
  • Component Contribution: Task-Related Cues raise macro TSR to 47.8%, Technical Hooks to 79.2%, and Prompt Naturalization lowers it to 63.5% while retaining a 48.3-point gain over Native.The cumulative results show a trade-off between peak selection effectiveness and a less conspicuous prompt surface.
  • Effectiveness Comparison: 73.5% TSR for ISM is 9.8 percentage points below Explicit Steering’s 83.3% and above Random Keywords’ 24.6% and Domain Wording’s 24.2%.This matched comparison evaluates whether implicit manipulation approaches explicit steering without explicit selection instructions.
  • Judgment-Based Inspection: Human reviewers block ISM in 2.9% of judgments versus 91.4% for Explicit Steering, while LLM-based inspection also blocks ISM less often.The judgment-based evaluation uses condition-hidden reviews and five LLM inspectors to assess perceived implicitness.
  • Rule-Based Inspection: PPL-W blocks 0.0% of ISM inputs, while Llama Prompt Guard 2 and PIGuard block 25.4% and 23.1%; post-inspection TSR remains 46.9%–62.2%.The latter two defenses reduce undefended TSR from 62.2% to 47.8% and 46.9%, respectively.

5 Countermeasures

The paper proposes structural rephrasing as a pre-selection canonicalization layer that normalizes skill descriptions before semantic matching. It is intended to reduce presentation-driven variation while preserving source-supported capabilities, but remains only a partial mitigation.

  • Structural Rephrasing: Structural rephrasing derives normalized skill descriptions from underlying documentation instead of exposing developer-authored descriptions verbatim.The proposed rewriter summarizes source documentation into a fixed schema covering purpose, tasks, inputs, outputs, capabilities, and constraints.
  • Structural Rephrasing: The selector compares normalized representations to reduce presentation-driven variation while retaining genuine functional differences.The method removes promotional, redundant, or unsupported wording while preserving materially distinct capabilities grounded in source documentation.
  • Limitations: Structural rephrasing remains a partial mitigation because fabricated source claims may survive normalization and aggressive rewriting may erase legitimate distinctions.The paper frames it as a pre-selection canonicalization layer rather than a complete defense and calls for joint evaluation of resistance, capability preservation, and benign utility.

6 Discussion

The discussion frames pre-execution skill selection as a security-critical decision shaped by relative prompt–metadata compatibility, not merely by isolated malicious text. It argues that preserving selection integrity requires defenses that balance manipulation resistance with benign selection quality.

  • Selection as a Security Boundary: Skill selection is a security boundary because manipulating candidate ranking changes which executable capability enters the agent’s downstream path.The discussion distinguishes selection manipulation from a retrieval discrepancy and confines the claim to pre-execution security significance.
  • Selection Integrity: Selection integrity asks whether one candidate gains an unjustified advantage over otherwise suitable alternatives, rather than only whether text is malicious in isolation.This abstraction targets comparative compatibility and differs from instruction-oriented defenses that search for recognizable steering signals.
  • Defense Design: Stronger text-level filtering alone is insufficient because reducing manipulated selection can coincide with substantial false positives on benign inputs.The discussion therefore calls for safeguards that protect comparative selection while preserving benign selection utility.
  • Scope: The claims remain confined to pre-execution selection and do not by themselves establish harmful downstream execution.Future evaluations should measure both manipulation resistance and benign selection quality.

7 Related Work

Related work establishes that tool-selection behavior depends on how capability metadata is expressed and that prompt injection can manipulate tool use. This paper extends that line of work by studying coordinated, implicit manipulation of pre-execution skill selection among plausible same-domain candidates.

  • Tool Selection in LLM Agents: Tool-selection research shows that semantic alignment between requests and tool metadata can strongly affect model preferences.Prior work primarily studies selection quality or naturally occurring behavior, whereas this paper examines the sensitivity adversarially.
  • Manipulating Tool Selection: Prompt-injection research covers direct and indirect attacks that redirect objectives, manipulate tool use, or induce unsafe downstream actions.These attacks place adversarial instructions in user inputs or external content entering the model context.
  • Selection-Focused Attacks: ToolHijacker, Attractive Metadata Attack, ToolTweak, and LLM Whisperer show that malicious metadata and inconspicuous prompt modifications can influence model decisions.Together, these studies cover tool documents, names and descriptions, and subtle user-facing wording.
  • Positioning: This work differs by coordinating user-facing prompts and target-skill descriptions without explicit tool-selection instructions.The setting biases pre-execution selection among plausible same-domain candidates rather than directly instructing the selector.
  • Defending LLM Agents: Existing defenses largely target suspicious instructions, trust separation, or task deviations, motivating tests of their transfer to selection manipulation.The paper treats relative candidate preference as a distinct threat at the selection surface.

8 Conclusion

The conclusion shows that pre-execution skill selection can be manipulated implicitly through semantic matching, without explicit selection instructions. ISM achieves substantial selection effectiveness while evading human and LLM-based inspection more successfully than Explicit Steering, identifying skill selection as a security-relevant attack surface.

  • ISM raises macro target-selection rate from 15.2% under Native to 63.5% across four domains, eight target skills, and eight selector models.
  • In matched comparisons, ISM reaches 73.5% TSR, 9.8 percentage points below Explicit Steering.
  • Human and LLM-based inspections find ISM substantially more implicit than Explicit Steering, while rule-based inspection provides only partial coverage.
  • The findings identify semantic skill selection as a security-relevant pre-execution attack surface.
  • Future defenses should protect the selection decision itself rather than treating selection as benign preprocessing or relying solely on text-level inspection signals.

A Supplementary Evaluation

The appendix supplies protocol details and supplementary results supporting the main-text evaluation. It preserves the main evaluation setup and provides artifact materials for reproducibility.

  • The appendix provides protocol details and supplementary results without repeating aggregate results already reported in Section 4.
  • Unless otherwise noted, supplementary analyses use the same frozen prompts, candidate pools, model configurations, and condition definitions as Section 4.1.
  • The artifact includes exact prompts, configurations, structured outputs, and raw model responses.

A.1 Evaluation Protocol

The evaluation protocol isolates pre-execution skill selection and standardizes the candidate pool, metadata, instructions, outputs, and decoding across matched conditions. Separate LLM and human inspections assess whether the materials would be blocked or flagged.

  • Evaluation Protocol: The selector chooses exactly one skill from a seven-skill candidate pool before any selected backend is invoked.
  • Evaluation Protocol: Candidate metadata, candidate order, selector instructions, output schema, and decoding configuration remain fixed across matched conditions, with each material instance evaluated three times.
  • LLM inspection: LLM inspectors judge whether visible prompt–metadata pairs attempt to manipulate, bias, force, or override skill selection without inferring hidden experimental conditions.
  • Human inspection: Human reviewers assess condition-hidden prompt–metadata pairs using Approve, Review, or Block decisions and five-point subjective ratings.
  • Human inspection: Reported human-inspection percentages are pooled reviewer judgments rather than item-level majority votes.

A.2 Semantic-Relation Analysis

The semantic-relation diagnostic compares several ways of realizing technical-hook meaning, including relation-constrained variants and the mixed ISM construction. Relation-constrained designs retain strong TSR with lower would-block rates, while the authors treat the diagnostic as construction evidence rather than a causal explanation.

  • Semantic-relation diagnostic: The diagnostic compares direct-hook exposure, generic naturalization, four semantic-relation families, a hard-to-bridge control, and mixed ISM construction.
  • Semantic-relation diagnostic: Direct-hook exposure yields the highest TSR but also the highest would-block rate among the tested constructions.
  • Semantic-relation diagnostic: Relation-constrained variants retain TSRs of 60.0–73.3% with would-block rates of 6.7–10.0%.
  • Semantic-relation diagnostic: The mixed ISM construction reaches 76.7% TSR while reducing would-block rate by 26.7 percentage points relative to direct-hook exposure.
  • Interpretation: The authors treat this diagnostic as construction evidence rather than a causal account of selector behavior.

A.3 Prompt Naturalization Analysis

Prompt Naturalization reduces the visibility and blocking of ISM prompts while preserving ordinary-task judgments. The broader per-target analysis reports positive uplift for all eight targets, with variation in magnitude, and the human inspection evaluates technically experienced users.

  • A.3 Prompt Naturalization Analysis: Prompt Naturalization compares Native, direct technical hooks, naturalized prompts, and Explicit Steering on prompt-only inspection.The diagnostic isolates the user-facing prompt surface.
  • A.3 Prompt Naturalization Analysis: 46.7% of direct-hook prompts are judged suspicious, versus 10.0% after naturalization.Keyword-stuffed judgments also fall from 60.0% to 6.7%.
  • A.3 Prompt Naturalization Analysis: 3.3% of naturalized prompts are blocked, compared with 20.0% for direct technical hooks and 93.3% for Explicit Steering.The natural-task rate rises from 53.3% to 90.0% after naturalization.
  • A.4 Per-Target Results: All eight targets show positive mean uplift, although the magnitude varies across targets.Table 4 provides the corresponding target-level breakdown, with rates averaged across selectors.
  • A.4 Per-Target Results: 22.6 to 70.8 percentage points: the uplift spans all evaluated targets rather than being driven by a single target or domain.∆TSR is computed from unrounded rates and may differ slightly from displayed-value differences.
  • A.5 Human Inspection Results: Condition-hidden human inspection pools decision rates over 70 reviewer judgments per condition.Naturalness, plausibility, over-optimization, and selection suspicion use five-point ratings.
  • A.5 Human Inspection Results: The reviewers were technically experienced users rather than a sample representing recognition in the general population.Participants had backgrounds in computer science, artificial intelligence, or related fields and substantial LLM or agent-system experience.
  • A.5 Human Inspection Results: 2.9% of ISM judgments are blocked, compared with 91.4% for Explicit Steering.ISM metadata remains highly plausible, while its selection-suspicion score remains far below Explicit Steering.
Loading 2609.02035v1…