PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Injection Attack to Tool Selection in LLM Agents

Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun

arXiv:2504.19793v3cs.CR

TL;DR

Tool selection in LLM agents remains vulnerable to prompt injection, while existing attacks are suboptimal for end-to-end manipulation. ToolHijacker optimizes malicious tool documents against a shadow pipeline to influence both retrieval and selection. It achieves high attack success and outperforms existing attacks, whereas evaluated prevention and detection defenses remain insufficient.

  • Problem

    Existing prompt injection methods remain suboptimal for tool selection because they incompletely address retrieval and selection.

  • Method

    ToolHijacker formulates malicious tool-document crafting as an optimization problem and solves it with a two-phase strategy targeting retrieval and selection.

  • Results

    96.7% attack success rate is achieved on MetaTool across architecturally different shadow and target LLMs, while ToolHijacker outperforms other attacks.

  • Takeaways & Limitations

    Both prevention-based and detection-based defenses evaluated in the paper are insufficient to counter ToolHijacker.

Abstract

from arXiv · show

Tool selection is a key component of LLM agents. A popular approach follows a two-step process - \emph{retrieval} and \emph{selection} - to pick the most appropriate tool from a tool library for a given task. In this work, we introduce \textit{ToolHijacker}, a novel prompt injection attack targeting tool selection in no-box scenarios. ToolHijacker injects a malicious tool document into the tool library to manipulate the LLM agent's tool selection process, compelling it to consistently choose the attacker's malicious tool for an attacker-chosen target task. Specifically, we formulate the crafting of such tool documents as an optimization problem and propose a two-phase optimization strategy to solve it. Our extensive experimental evaluation shows that ToolHijacker is highly effective, significantly outperforming existing manual-based and automated prompt injection attacks when applied to tool selection. Moreover, we explore various defenses, including prevention-based defenses (StruQ and SecAlign) and detection-based defenses (known-answer detection, DataSentinel, perplexity detection, and perplexity windowed detection). Our experimental results indicate that these defenses are insufficient, highlighting the urgent need for developing new defense strategies.

I. INTRODUCTION

Tool selection is central to LLM-agent performance but remains vulnerable to prompt injection through external tool documents. ToolHijacker targets this vulnerability in no-box settings by optimizing malicious documents that influence both retrieval and selection.

  • Background: LLM agents commonly select tools through retrieval followed by LLM-based selection from the retrieved tool documents.The retriever identifies the top-k documents, after which the LLM selects the tool for calling.
  • Motivation: Existing prompt injection attacks are incomplete or suboptimal for end-to-end tool selection because they primarily target selection or retrieval separately.Manual attacks and JudgeDeceiver focus mainly on selection, whereas PoisonedRAG targets retrieval through multiple malicious knowledge-base entries rather than directly manipulating tool selection.
  • Approach: ToolHijacker injects a malicious tool document that manipulates both retrieval and selection, compelling the agent to choose it for an attacker-selected task in a no-box scenario.The attack is designed to maintain control across varying semantic descriptions of the target task.
  • Results: 96.7% attack success rate is achieved on MetaTool when Llama-3.3-70B is the shadow LLM and GPT-4o is the target LLM.The gradient-free method remains effective despite the architectural difference between shadow and target models.
  • Results: 100% attack hit rate is achieved during MetaTool retrieval, and ToolHijacker outperforms existing prompt injection attacks in tool selection.The evaluation covers two benchmark datasets, 8 LLMs, and 4 retrievers.
  • Defenses: 99.6% success rate is achieved under StruQ, while detection defenses miss most malicious documents or fail to identify them.PPL misses 90% of gradient-based malicious documents while falsely detecting less than 1% of benign documents.

II. PROBLEM FORMULATION

The paper formalizes tool selection as retrieval followed by LLM selection and defines an attacker who crafts a malicious tool document under no-box constraints. The threat includes influencing tool choice and potentially the subsequent execution of the selected tool.

  • A. Tool Selection: The tool-selection framework contains a tool library, retriever, and LLM, with retrieval and selection as its two core stages.Tool documents specify tool names, descriptions, and API specifications.
  • A. Tool Selection: The retriever encodes the task and tool documents, ranks them by similarity, and returns the top-k documents.The retrieved set is denoted Dk = Top-k(q; D).
  • A. Tool Selection: The LLM receives the task and retrieved tool information in a structured prompt and outputs the selected tool name.The prompt concatenates header instructions, the task, tool documents, and trailer instructions.
  • B. Threat Model: The attacker seeks a malicious tool that is preferentially selected for any target-task description, bypassing benign tools.Target-task descriptions may be semantically varied while representing the same task.
  • B. Threat Model: The attack focuses on the tool name and description because prior research indicates that tool selection primarily relies on them.The tool document also contains API specifications, but the study crafts the name and description.
  • B. Threat Model: In the no-box scenario, the attacker cannot access tool documents, top-k results, target retriever or LLM parameters, or directly query the target system.The attacker can use standardized platform development guidelines and construct a shadow pipeline.
  • B. Threat Model: Once selected, a malicious tool may execute without further verification, enabling arbitrary manipulation of execution outcomes.The paper gives unauthorized data access and privacy breaches as examples of potential security concerns.
  • B. Threat Model: The attacker uses shadow task descriptions, shadow documents, a shadow retriever, and a shadow LLM to design and validate the attack.Shadow task descriptions are disjoint from the target descriptions, and malicious tools can be published on third-party tool hubs.

A. Overview

ToolHijacker addresses no-box tool-selection attacks by optimizing a malicious tool document against a shadow pipeline. Its two-phase strategy first promotes retrieval and then optimizes selection-oriented behavior.

  • Overview: ToolHijacker formulates malicious tool-document crafting as a discrete optimization problem spanning retrieval and selection.The optimized document contains a malicious tool name and description.
  • Overview: The method constructs shadow task descriptions and shadow tool documents to simulate the target tool library.The shadow library includes task-relevant and task-irrelevant documents.
  • Overview: The shadow optimization objective seeks selection of the malicious tool across the shadow task descriptions.The formulation uses a shadow retriever, shadow LLM, and an indicator for whether the malicious tool is selected.
  • Overview: The problem is discrete, discontinuous, and non-differentiable, with many local optima that make direct gradient-based optimization difficult.ToolHijacker therefore decomposes the task into retrieval and selection sub-objectives.
  • Overview: The retrieval phase generates a subsequence R that places the malicious document among the top-k′ retrieved documents.It does so by increasing similarity between R and diverse shadow task descriptions.
  • Overview: The gradient-free approach generates R from semantic alignment between tool functionality descriptions and shadow tasks.Its prompt requests a general functionality description applicable across scenarios and limits the length of R.
  • Overview: The gradient-based approach maximizes average similarity between R and shadow tasks using retriever gradients and token-level HotFlip optimization.It initializes R with the gradient-free output before applying gradient descent.

D. Optimizing S for Selection

The paper optimizes the malicious description S so a shadow LLM consistently selects the attacker’s tool across shadow task–retrieval pairs. It combines multi-component losses with gradient-free tree search and position-adaptive, step-wise optimization.

  • The optimization objective is to make the shadow LLM generate responses containing the malicious tool name, thereby consistently selecting the malicious document across task–retrieval pairs.
  • Gradient-Free: The gradient-free approach uses an attacker LLM to generate variants of S, evaluates their matches across shadow tasks, and prunes the search tree using FLAG scores.
  • Gradient-Free: A node is accepted when it matches the malicious tool for all m′ shadow task descriptions; otherwise, the highest-scoring nodes are retained for further iterations.
  • The overall loss combines alignment, consistency, and perplexity terms, with α and β balancing their contributions.The perplexity term is intended to improve the readability of S.
  • Optimization strategy: The method optimizes S at different insertion positions within retrieved shadow documents and progressively adds task–retrieval pairs to stabilize optimization.

A. Experimental Setup

The evaluation uses MetaTool and ToolBench to test attack effectiveness on tool libraries representing both general tool use and open-source LLM tool-use training.

  • Datasets: MetaTool contains 21,127 instances and 199 benign tool documents sourced from OpenAI Plugins.
  • Datasets: ToolBench contains 126,486 instruction-tuning samples and 16,464 RapidAPI tool documents, reduced to 9,650 benign tools after deduplication and description filtering.
  • Datasets: For each dataset, the authors design 10 diverse target tasks representing real-world needs and generate 100 target task descriptions per task.

2) Compared Baselines:

The experiments compare ToolHijacker with five manual and two automated prompt injection baselines across multiple LLMs and retrievers, while reporting attack and retrieval metrics.

  • Compared baselines: The baseline suite contains five manual attacks—naive, escape characters, context ignore, fake completion, and combined attack—and two automated attacks, JudgeDeceiver and PoisonedRAG.
  • Manual attacks: Manual attacks use explicit instructions, text segmentation, context override, fabricated completions, or combinations of these strategies.
  • Automated attacks: JudgeDeceiver uses a gradient-optimized adversarial sequence to influence LLM-as-a-Judge, whereas PoisonedRAG injects optimized adversarial texts into a retrieval database.
  • Attack settings: The evaluation spans open- and closed-source target LLMs, including Llama, Claude, GPT-3.5, and GPT-4o, alongside four retrieval models.
  • Metrics: The experiments use accuracy, attack success rate, hit rate, and attack hit rate as evaluation metrics.

5) Evaluation Metrics:

The evaluation measures end-to-end tool-selection utility and attack effectiveness with ACC and ASR, while HR and AHR isolate retrieval. ToolHijacker achieves high attack performance across models, datasets, retrievers, tasks, and ablations.

  • Evaluation Metrics: ACC measures correct benign-tool selection, while ASR measures malicious-tool selection after injection.Both metrics evaluate 100 task descriptions per target task.
  • Evaluation Metrics: HR measures whether any correct tool enters the retriever’s top-k results, whereas AHR measures whether the malicious document enters them.HR and AHR therefore assess the retrieval stage rather than the complete selection process.
  • Main Results: 96.1% and 97.8% AHRs are achieved on ToolBench’s 9,650-document library by gradient-free and gradient-based attacks using one malicious document.The attacks remain effective across target tasks, datasets, and target LLMs.
  • Main Results: 39.3% and 58.3% are the highest baseline ASRs on MetaTool and ToolBench, while ToolHijacker outperforms those baselines.The generated malicious documents are short and indistinguishable from benign documents based solely on token length.
  • Ablation Studies: 100% AHR and 99% ASR are achieved across all evaluated retrievers by the gradient-free attack.The gradient-based attack reaches 100% AHR across retrievers and 100% ASR on open-source retrievers, versus 95% ASR for text-embedding-ada-002.

V. DEFENSES

The paper evaluates prevention- and detection-based defenses against ToolHijacker. Both defense categories show substantial weaknesses, although SecAlign reduces ASR more than StruQ and detection performance varies by method.

  • Prevention-based Defense: Prevention-based defenses either preprocess prompts or fine-tune LLMs to reduce susceptibility to prompt injection.Because tool selection already uses sandwich prevention, the evaluation focuses primarily on fine-tuning-based defenses.
  • Prevention-based Defense: StruQ separates secure prompts from user data, while SecAlign fine-tunes LLMs using prompt-injected inputs and secure/insecure response pairs.The experiments use the fine-tuned models provided by both defenses.
  • Prevention-based Defense: SecAlign achieves a larger ASR reduction than StruQ, while higher ΔASR on ToolBench suggests stronger defense effectiveness with larger tool libraries.ΔASR is defined as ASRnone − ASRd.
  • Detection-based Defense: Known-answer detection and DataSentinel both exceed 90% FNR, indicating difficulty detecting malicious tool documents.The paper attributes this to malicious descriptions preserving overall semantics by avoiding task-irrelevant injected instructions.
  • Detection-based Defense: PPL-based detection performs differently for gradient-based and gradient-free attacks, while PPL-W flags text when any contiguous window exceeds its threshold.Detection is evaluated using FNR and FPR on MetaTool and ToolBench.

VI. RELATED WORK

Related work covers tool-enhanced agents, prompt injection attacks, and defenses. The paper distinguishes ToolHijacker by targeting end-to-end tool selection, including both retrieval and selection, in a no-box setting.

  • Tool-Enhanced LLM Agents: Prior systems connect LLMs with APIs, knowledge bases, and specialized modules to support flexible tool use and tool selection.Examples include TaskMatrix.AI, Gorilla, REST-GPT, and tool-selection benchmarks.
  • Tool-Enhanced LLM Agents: Recent work improves tool use through progressive retrieval, multi-stage training, virtual tool tokens, adaptive reranking, and graph-based methods.These approaches focus on improving tool-use capabilities rather than attacking tool selection.
  • Prompt Injection Attacks: Prompt injection attacks use manual heuristics or optimization to manipulate LLM behavior through malicious instructions in external data.Related agent attacks target tool calling, web interaction, multimodal systems, and multi-agent settings.
  • Defenses: Prevention defenses use prompt structuring or adversarial training, whereas detection defenses identify injected instructions using perplexity or known-answer tests.Prior work reports trade-offs in general instruction following and vulnerability to adaptive attacks.
  • Paper Positioning: ToolHijacker is an automated framework that manipulates tool selection and outperforms other attacks, while evaluated prevention and detection defenses remain insufficient.The paper focuses on tool selection as a fundamental agent decision-making component.
  • Responsible Research: The experiments were conducted in controlled environments without deploying real malicious tools or directly exposing real users to harm.Code and data are planned for restricted-access release, and relevant companies were notified.

APPENDIX

The appendix provides a symbol reference for the paper’s mathematical formulation and model design. It lists notation for LLMs, tools, tasks, losses, and optimization components.

  • A. List of Symbols: Table XI lists symbols for the target and attacker LLMs, tool documents, task descriptions, and loss functions.The table serves as a concise reference for notation used throughout the paper.
  • A. List of Symbols: The symbol list supports the mathematical formulation and model design discussed in the main paper.It functions as a notation reference rather than introducing a separate method or result.

B. Supplementary Experimental Results

Supplementary experiments examine general utility, attacker and target LLM variation, optimization settings, dynamic tool libraries, human detection, and computational cost. Results indicate strong attack performance across settings, with optimization choices affecting query cost and attack success.

  • LLM Variation: Attacker models can generalize across target LLMs, but ASR on Claude-3-Haiku falls to 43% and 44% for the reported settings.The paper attributes this discrepancy to Claude-3-Haiku’s higher security.
  • Optimization Settings: Changing B from 1 to 5 preserves effective gradient-free attacks but directly changes total query cost.The total query number is calculated as (B + B × m′) × iter; smaller B may require more iterations, while larger B verifies more variants.
  • Optimization Settings: Removing any single loss term reduces ASR by at least 39%, with perplexity loss L3 having the largest reported impact.Without L3, the optimized tool document becomes unnatural or nonsensical and is more likely to be identified as anomalous.
  • Dynamic Tool Libraries: Across dynamic library sizes, gradient-free ASR remains ≥96.7% on MetaTool and ≥93.3% on ToolBench, while gradient-based ASR remains ≥92.8% and ≥84.8%.These results are reported as robustness to tool library updates.
  • Human Detection: Six participants failed to detect at least 71% of malicious tools in the reported human-detection study.The study used ToolBench versions containing 200, 400, and 600 tools and seven malicious tools.
  • Computational Cost: The gradient-free method requires approximately 18 LLM queries for S, while the gradient-based method requires about 8 GPU-hours on one NVIDIA A800 GPU.The reported costs for R are one LLM query for gradient-free optimization and about one GPU-minute for gradient-based optimization.

C. Details of Prompts and Datasets

This section describes the prompts, optimization setup, and datasets used to evaluate ToolHijacker and baseline attacks. It covers shadow data generation, attacker-LLM instructions, target-task construction, and evaluation examples across MetaTool and ToolBench.

  • Prompts and optimization setup: Shadow task descriptions and shadow tool documents are generated with GPT-3.5-turbo using templates shown in Figures 10 and 11.The section also specifies initial R and S conditions for the attacks in Figure 12.
  • Datasets and target tasks: The evaluation defines 10 target tasks for each of MetaTool and ToolBench across diverse real-world domains, including space exploration, finance, resumes, fitness, and email validation.The section presents task-specific tool lists and examples, and reports malicious tool descriptions for seven baseline attacks.
  • Prompts and optimization setup: The attacker LLM is instructed to optimize an adversarial prompt so the target model selects the target tool in JSON format.The target model receives a task question and tool list, while the optimization prompt is embedded between designated delimiters in the target tool description.
  • Prompts and optimization setup: Optimization guidance emphasizes firm target-tool instructions, misleading descriptions, transferability across related tasks and tool lists, keyword highlighting, and 20–25-word prompts.These instructions are intended to improve adaptability and influence while keeping the adversarial prompt concise.
  • Prompts and optimization setup: The gradient-free optimization uses evaluation results, an objective, and a flag counting successful target-tool selections across questions.The attacker LLM is told to maximize the flag to the number of evaluation questions.
  • Datasets and target tasks: ToolBench examples include financial-insight and risk-assessment tasks alongside personalized fitness planning and health-metric management.These examples extend the evaluation beyond the MetaTool tasks and cover multiple tool-selection contexts.
Loading 2504.19793v3…