Prompt Injection Attack to Tool Selection in LLM Agents

Background

• LLM agents commonly select tools through retrieval followed by LLM-based selection from the retrieved tool documents.The retriever identifies the top-k documents, after which the LLM selects the tool for calling.Source: §I · p. 1

Motivation

• Existing prompt injection attacks are incomplete or suboptimal for end-to-end tool selection because they primarily target selection or retrieval separately.Manual attacks and JudgeDeceiver focus mainly on selection, whereas PoisonedRAG targets retrieval through multiple malicious knowledge-base entries rather than directly manipulating tool selection.Source: §I · p. 1

Approach

• ToolHijacker injects a malicious tool document that manipulates both retrieval and selection, compelling the agent to choose it for an attacker-selected task in a no-box scenario.The attack is designed to maintain control across varying semantic descriptions of the target task.Source: §I · pp. 1–2

Results

• 96.7% attack success rate is achieved on MetaTool when Llama-3.3-70B is the shadow LLM and GPT-4o is the target LLM.The gradient-free method remains effective despite the architectural difference between shadow and target models.Source: §I · p. 2
• 100% attack hit rate is achieved during MetaTool retrieval, and ToolHijacker outperforms existing prompt injection attacks in tool selection.The evaluation covers two benchmark datasets, 8 LLMs, and 4 retrievers.Source: §I · p. 2

Defenses

• 99.6% success rate is achieved under StruQ, while detection defenses miss most malicious documents or fail to identify them.PPL misses 90% of gradient-based malicious documents while falsely detecting less than 1% of benign documents.Source: §I · p. 2

A. Tool Selection

• The tool-selection framework contains a tool library, retriever, and LLM, with retrieval and selection as its two core stages.Tool documents specify tool names, descriptions, and API specifications.Source: A. Tool Selection · p. 3
• The retriever encodes the task and tool documents, ranks them by similarity, and returns the top-k documents.The retrieved set is denoted Dk = Top-k(q; D).Source: A. Tool Selection · p. 3
• The LLM receives the task and retrieved tool information in a structured prompt and outputs the selected tool name.The prompt concatenates header instructions, the task, tool documents, and trailer instructions.Source: A. Tool Selection · p. 3

B. Threat Model

• The attacker seeks a malicious tool that is preferentially selected for any target-task description, bypassing benign tools.Target-task descriptions may be semantically varied while representing the same task.Source: B. Threat Model · p. 3
• The attack focuses on the tool name and description because prior research indicates that tool selection primarily relies on them.The tool document also contains API specifications, but the study crafts the name and description.Source: B. Threat Model · p. 3
• In the no-box scenario, the attacker cannot access tool documents, top-k results, target retriever or LLM parameters, or directly query the target system.The attacker can use standardized platform development guidelines and construct a shadow pipeline.Source: B. Threat Model · pp. 3–4
• Once selected, a malicious tool may execute without further verification, enabling arbitrary manipulation of execution outcomes.The paper gives unauthorized data access and privacy breaches as examples of potential security concerns.Source: B. Threat Model · p. 3
• The attacker uses shadow task descriptions, shadow documents, a shadow retriever, and a shadow LLM to design and validate the attack.Shadow task descriptions are disjoint from the target descriptions, and malicious tools can be published on third-party tool hubs.Source: B. Threat Model · p. 4

Overview

• ToolHijacker formulates malicious tool-document crafting as a discrete optimization problem spanning retrieval and selection.The optimized document contains a malicious tool name and description.Source: A. Overview · p. 4, B. Formulating an Optimization Problem · p. 4
• The method constructs shadow task descriptions and shadow tool documents to simulate the target tool library.The shadow library includes task-relevant and task-irrelevant documents.Source: B. Formulating an Optimization Problem · p. 4
• The shadow optimization objective seeks selection of the malicious tool across the shadow task descriptions.The formulation uses a shadow retriever, shadow LLM, and an indicator for whether the malicious tool is selected.Source: B. Formulating an Optimization Problem · p. 4
• The problem is discrete, discontinuous, and non-differentiable, with many local optima that make direct gradient-based optimization difficult.ToolHijacker therefore decomposes the task into retrieval and selection sub-objectives.Source: B. Formulating an Optimization Problem · p. 4
• The retrieval phase generates a subsequence R that places the malicious document among the top-k′ retrieved documents.It does so by increasing similarity between R and diverse shadow task descriptions.Source: C. Optimizing R for Retrieval · p. 4
• The gradient-free approach generates R from semantic alignment between tool functionality descriptions and shadow tasks.Its prompt requests a general functionality description applicable across scenarios and limits the length of R.Source: C. Optimizing R for Retrieval · pp. 4–5
• The gradient-based approach maximizes average similarity between R and shadow tasks using retriever gradients and token-level HotFlip optimization.It initializes R with the gradient-free output before applying gradient descent.Source: C. Optimizing R for Retrieval · p. 5

• The optimization objective is to make the shadow LLM generate responses containing the malicious tool name, thereby consistently selecting the malicious document across task–retrieval pairs.Source: D. Optimizing S for Selection · p. 5

Gradient-Free

• The gradient-free approach uses an attacker LLM to generate variants of S, evaluates their matches across shadow tasks, and prunes the search tree using FLAG scores.Source: D. Optimizing S for Selection · pp. 5–6
• A node is accepted when it matches the malicious tool for all m′ shadow task descriptions; otherwise, the highest-scoring nodes are retained for further iterations.Source: D. Optimizing S for Selection · p. 6

• The overall loss combines alignment, consistency, and perplexity terms, with α and β balancing their contributions.The perplexity term is intended to improve the readability of S.Source: D. Optimizing S for Selection · p. 6

Optimization strategy

• The method optimizes S at different insertion positions within retrieved shadow documents and progressively adds task–retrieval pairs to stabilize optimization.Source: D. Optimizing S for Selection · p. 6

Datasets

• MetaTool contains 21,127 instances and 199 benign tool documents sourced from OpenAI Plugins.Source: 1) Datasets · p. 6
• ToolBench contains 126,486 instruction-tuning samples and 16,464 RapidAPI tool documents, reduced to 9,650 benign tools after deduplication and description filtering.Source: 1) Datasets · p. 6
• For each dataset, the authors design 10 diverse target tasks representing real-world needs and generate 100 target task descriptions per task.Source: 1) Datasets · p. 6

Compared baselines

• The baseline suite contains five manual attacks—naive, escape characters, context ignore, fake completion, and combined attack—and two automated attacks, JudgeDeceiver and PoisonedRAG.Source: 2) Compared Baselines · p. 6

Manual attacks

• Manual attacks use explicit instructions, text segmentation, context override, fabricated completions, or combinations of these strategies.Source: 2) Compared Baselines · pp. 6–7

Automated attacks

• JudgeDeceiver uses a gradient-optimized adversarial sequence to influence LLM-as-a-Judge, whereas PoisonedRAG injects optimized adversarial texts into a retrieval database.Source: 2) Compared Baselines · p. 7

Attack settings

• The evaluation spans open- and closed-source target LLMs, including Llama, Claude, GPT-3.5, and GPT-4o, alongside four retrieval models.Source: 3) Tool Selection Setup · p. 7

Metrics

• The experiments use accuracy, attack success rate, hit rate, and attack hit rate as evaluation metrics.Source: 5) Evaluation Metrics · p. 7

Evaluation Metrics

• ACC measures correct benign-tool selection, while ASR measures malicious-tool selection after injection.Both metrics evaluate 100 task descriptions per target task.Source: 5) Evaluation Metrics · p. 7
• HR measures whether any correct tool enters the retriever’s top-k results, whereas AHR measures whether the malicious document enters them.HR and AHR therefore assess the retrieval stage rather than the complete selection process.Source: 5) Evaluation Metrics · pp. 7–8

Main Results

• 96.1% and 97.8% AHRs are achieved on ToolBench’s 9,650-document library by gradient-free and gradient-based attacks using one malicious document.The attacks remain effective across target tasks, datasets, and target LLMs.Source: B. Main Results · p. 8
• 39.3% and 58.3% are the highest baseline ASRs on MetaTool and ToolBench, while ToolHijacker outperforms those baselines.The generated malicious documents are short and indistinguishable from benign documents based solely on token length.Source: B. Main Results · p. 8

Ablation Studies

• 100% AHR and 99% ASR are achieved across all evaluated retrievers by the gradient-free attack.The gradient-based attack reaches 100% AHR across retrievers and 100% ASR on open-source retrievers, versus 95% ASR for text-embedding-ada-002.Source: C. Ablation Studies · p. 9

Prevention-based Defense

• Prevention-based defenses either preprocess prompts or fine-tune LLMs to reduce susceptibility to prompt injection.Because tool selection already uses sandwich prevention, the evaluation focuses primarily on fine-tuning-based defenses.Source: §V · p. 11
• StruQ separates secure prompts from user data, while SecAlign fine-tunes LLMs using prompt-injected inputs and secure/insecure response pairs.The experiments use the fine-tuned models provided by both defenses.Source: A. Prevention-based Defense · p. 11
• SecAlign achieves a larger ASR reduction than StruQ, while higher ΔASR on ToolBench suggests stronger defense effectiveness with larger tool libraries.ΔASR is defined as ASRnone − ASRd.Source: A. Prevention-based Defense · p. 12, Figure 8 · p. 12

Detection-based Defense

• Known-answer detection and DataSentinel both exceed 90% FNR, indicating difficulty detecting malicious tool documents.The paper attributes this to malicious descriptions preserving overall semantics by avoiding task-irrelevant injected instructions.Source: B. Detection-based Defense · p. 12, Table X · p. 12
• PPL-based detection performs differently for gradient-based and gradient-free attacks, while PPL-W flags text when any contiguous window exceeds its threshold.Detection is evaluated using FNR and FPR on MetaTool and ToolBench.Source: B. Detection-based Defense · p. 12

Tool-Enhanced LLM Agents

• Prior systems connect LLMs with APIs, knowledge bases, and specialized modules to support flexible tool use and tool selection.Examples include TaskMatrix.AI, Gorilla, REST-GPT, and tool-selection benchmarks.Source: A. Tool Selection in LLM Agents · pp. 12–13
• Recent work improves tool use through progressive retrieval, multi-stage training, virtual tool tokens, adaptive reranking, and graph-based methods.These approaches focus on improving tool-use capabilities rather than attacking tool selection.Source: A. Tool Selection in LLM Agents · p. 13

Prompt Injection Attacks

• Prompt injection attacks use manual heuristics or optimization to manipulate LLM behavior through malicious instructions in external data.Related agent attacks target tool calling, web interaction, multimodal systems, and multi-agent settings.Source: B. Prompt Injection Attacks · p. 13

Defenses

• Prevention defenses use prompt structuring or adversarial training, whereas detection defenses identify injected instructions using perplexity or known-answer tests.Prior work reports trade-offs in general instruction following and vulnerability to adaptive attacks.Source: C. Defenses · p. 13

Paper Positioning

• ToolHijacker is an automated framework that manipulates tool selection and outperforms other attacks, while evaluated prevention and detection defenses remain insufficient.The paper focuses on tool selection as a fundamental agent decision-making component.Source: §VII · pp. 13–14, C. Defenses · p. 13

Responsible Research

• The experiments were conducted in controlled environments without deploying real malicious tools or directly exposing real users to harm.Code and data are planned for restricted-access release, and relevant companies were notified.Source: ETHICS CONSIDERATIONS · p. 14

A. List of Symbols

• Table XI lists symbols for the target and attacker LLMs, tool documents, task descriptions, and loss functions.The table serves as a concise reference for notation used throughout the paper.Source: A. List of Symbols · pp. 15–16, Table XI · p. 16
• The symbol list supports the mathematical formulation and model design discussed in the main paper.It functions as a notation reference rather than introducing a separate method or result.Source: A. List of Symbols · p. 15

LLM Variation

• Attacker models can generalize across target LLMs, but ASR on Claude-3-Haiku falls to 43% and 44% for the reported settings.The paper attributes this discrepancy to Claude-3-Haiku’s higher security.Source: B. Supplementary Experimental Results · p. 16, Table XIII · p. 17

Optimization Settings

• Changing B from 1 to 5 preserves effective gradient-free attacks but directly changes total query cost.The total query number is calculated as (B + B × m′) × iter; smaller B may require more iterations, while larger B verifies more variants.Source: B. Supplementary Experimental Results · p. 16, Table XIV · p. 17
• Removing any single loss term reduces ASR by at least 39%, with perplexity loss L3 having the largest reported impact.Without L3, the optimized tool document becomes unnatural or nonsensical and is more likely to be identified as anomalous.Source: B. Supplementary Experimental Results · p. 17, Table XV · p. 17

Dynamic Tool Libraries

• Across dynamic library sizes, gradient-free ASR remains ≥96.7% on MetaTool and ≥93.3% on ToolBench, while gradient-based ASR remains ≥92.8% and ≥84.8%.These results are reported as robustness to tool library updates.Source: B. Supplementary Experimental Results · p. 17, Table XVI · p. 17

Human Detection

• Six participants failed to detect at least 71% of malicious tools in the reported human-detection study.The study used ToolBench versions containing 200, 400, and 600 tools and seven malicious tools.Source: B. Supplementary Experimental Results · p. 17, Table XVII · p. 17

Computational Cost

• The gradient-free method requires approximately 18 LLM queries for S, while the gradient-based method requires about 8 GPU-hours on one NVIDIA A800 GPU.The reported costs for R are one LLM query for gradient-free optimization and about one GPU-minute for gradient-based optimization.Source: B. Supplementary Experimental Results · p. 17

Prompts and optimization setup

• Shadow task descriptions and shadow tool documents are generated with GPT-3.5-turbo using templates shown in Figures 10 and 11.The section also specifies initial R and S conditions for the attacks in Figure 12.Source: C. Details of Prompts and Datasets · p. 17, Figure 10 · p. 18, Figure 11 · p. 18

Datasets and target tasks

• The evaluation defines 10 target tasks for each of MetaTool and ToolBench across diverse real-world domains, including space exploration, finance, resumes, fitness, and email validation.The section presents task-specific tool lists and examples, and reports malicious tool descriptions for seven baseline attacks.Source: C. Details of Prompts and Datasets · pp. 18, 20, 21

Prompts and optimization setup

• The attacker LLM is instructed to optimize an adversarial prompt so the target model selects the target tool in JSON format.The target model receives a task question and tool list, while the optimization prompt is embedded between designated delimiters in the target tool description.Source: C. Details of Prompts and Datasets · p. 19
• Optimization guidance emphasizes firm target-tool instructions, misleading descriptions, transferability across related tasks and tool lists, keyword highlighting, and 20–25-word prompts.These instructions are intended to improve adaptability and influence while keeping the adversarial prompt concise.Source: C. Details of Prompts and Datasets · p. 19
• The gradient-free optimization uses evaluation results, an objective, and a flag counting successful target-tool selections across questions.The attacker LLM is told to maximize the flag to the number of evaluation questions.Source: C. Details of Prompts and Datasets · pp. 18–20, Figure 13 · p. 20

Datasets and target tasks

• ToolBench examples include financial-insight and risk-assessment tasks alongside personalized fitness planning and health-metric management.These examples extend the evaluation beyond the MetaTool tasks and cover multiple tool-selection contexts.Source: C. Details of Prompts and Datasets · p. 22