Source-linked AI summary
IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents
Hengyu An, Jinghuai Zhang, Tianyu Du, Chunyi Zhou, Qingming Li, Tao Lin, Shouling Ji
TL;DR
Indirect prompt injections exploit unrestricted tool use when agents process untrusted data, and existing defenses do not structurally constrain such behavior. IPIGuard plans tool dependencies in a Tool Dependency Graph and enforces execution through approved traversals, with mechanisms for dynamic arguments and read-only expansion. Across four attack scenarios and six LLMs, it reports a strong balance between security and utility, while remaining limited to tool-use attacks and requiring reasonably strong planning models.
Problem
Existing defenses do not structurally constrain agents’ tool invocations, leaving them vulnerable to indirect prompt injections from untrusted data.
Method
IPIGuard decouples planning from external-data interaction and executes tasks by traversing a preplanned Tool Dependency Graph with approved tool invocations.
Results
Across four attack scenarios and six LLMs, IPIGuard achieves a strong balance between security and utility.
Takeaways & Limitations
Structural execution constraints provide a principled foundation for building more reliable LLM agents in dynamic environments.
Takeaways & Limitations
IPIGuard targets IPI attacks that interfere with tool usage and requires models with reasonably strong planning capabilities.
Abstract
from arXiv · showhide
Large language model (LLM) agents are widely deployed in real-world applications, where they leverage tools to retrieve and manipulate external data for complex tasks. However, when interacting with untrusted data sources (e.g., fetching information from public websites), tool responses may contain injected instructions that covertly influence agent behaviors and lead to malicious outcomes, a threat referred to as Indirect Prompt Injection (IPI). Existing defenses typically rely on advanced prompting strategies or auxiliary detection models. While these methods have demonstrated some effectiveness, they fundamentally rely on assumptions about the model's inherent security, which lacks structural constraints on agent behaviors. As a result, agents still retain unrestricted access to tool invocations, leaving them vulnerable to stronger attack vectors that can bypass the security guardrails of the model. To prevent malicious tool invocations at the source, we propose a novel defensive task execution paradigm, called IPIGuard, which models the agents' task execution process as a traversal over a planned Tool Dependency Graph (TDG). By explicitly decoupling action planning from interaction with external data, IPIGuard significantly reduces unintended tool invocations triggered by injected instructions, thereby enhancing robustness against IPI attacks. Experiments on the AgentDojo benchmark show that IPIGuard achieves a superior balance between effectiveness and robustness, paving the way for the development of safer agentic systems in dynamic environments.
1 Introduction
LLM agents can be redirected by malicious instructions embedded in untrusted data, while existing defenses leave tool access structurally unrestricted. IPIGuard instead plans tool dependencies in advance and constrains execution to approved graph traversals, supplemented by mechanisms for dynamic arguments and node expansion.
- Motivation: Indirect prompt injections embedded in untrusted data can redirect agents into unintended behaviors and malicious tool invocations.Examples include fraudulent emails and sensitive-information leakage through manipulated agent interactions.
- Motivation: Existing prompting, detection, and judge-based defenses rely on model security assumptions without structurally restricting tool access.Agents may therefore invoke available tools in ways that bypass guardrails and fail to mitigate attacks at their source.
- IPIGuard: IPIGuard decouples action planning from external-data interaction by representing execution as a topological traversal of a Tool Dependency Graph.The graph models tool dependencies and execution order while prohibiting tools not pre-approved in the plan.
- IPIGuard: Argument Estimation and Node Expansion address unpredictable inputs and information needs that arise when planning is separated from execution.The former estimates unknown arguments dynamically, while the latter expands read-only operations to gather required information.
- Evaluation: Experiments across four attack scenarios and six LLMs report a strong balance between security and utility for IPIGuard.The authors present this balance as evidence for a principled foundation for reliable LLM agents.
2 Preliminaries
This section formalizes tool-based task execution and indirect prompt injection, where malicious instructions in tool responses can redirect agents into unauthorized tool invocations. It motivates decoupling planning from external-data interaction while identifying unknown arguments, limited adaptability, and overlapping tools as key challenges.
- 2.1 Problem Definition: Traditional agents select and invoke tools sequentially, updating the environment state after each invocation and producing output from the final state and execution history.Each invocation includes a tool and its arguments, while execution proceeds over evolving environment states.
- 2.1 Problem Definition: Indirect prompt injection modifies a user task when malicious instructions in tool responses trigger additional tool invocations.The resulting sequence incorporates invocations associated with the injected task and can produce a successful attack.
- 2.2 Existing Vulnerability: Existing defenses lack structural constraints on tool access, allowing attackers to bypass model guardrails and trigger malicious invocations during execution.The limitation is that agents can still invoke any available tool based on injected instructions.
- 2.2 Defensive Direction: IPIGuard addresses this vulnerability by identifying required tools before execution and restricting the introduction of unauthorized tool invocations during execution.The approach decouples action planning from interaction with external data.
- 2.3 Key Challenges: Naive decoupling creates three challenges: unknown tool arguments, limited adaptability from static plans, and overlap between user and injected tools.Unknown arguments may depend on earlier tool outputs, while overlapping tools can enable attacks through argument modification without adding tools.
- 2.3 Key Challenges: When user and injected tasks overlap in their tools, an attack can succeed by changing arguments of existing invocations rather than adding new tools.IPIGuard therefore introduces Fake Tool Invocation to mitigate this case.
3 Method
IPIGuard executes tasks by traversing a planned Tool Dependency Graph that fixes tool dependencies and order before interacting with external data. Argument Estimation, Node Expansion, and Fake Tool Invocation address dynamic inputs, adaptability, and overlapping tool use while preserving task utility.
- 3.1 Tool Dependency Graph: IPIGuard traverses a Tool Dependency Graph that predefines tool invocations, dependencies, and execution order to address IPI attacks at their source.The graph is a directed acyclic graph whose nodes represent tool invocations and whose edges encode response dependencies.
- 3.2 Executing as TDG Traversal: Argument Estimation traverses the graph in topological order and uses dependent-tool responses to resolve unknown arguments before execution.Pending Nodes become Resolved Nodes with fully specified arguments, while Deterministic Nodes can execute directly.
- 3.2 Executing as TDG Traversal: Node Expansion permits only Query Tool invocations during execution, allowing read-only context expansion while excluding Command Tools that modify the environment.The mechanism addresses the adaptability limits of static plans without permitting unrestricted execution-time tool additions.
- 3.2 Executing as TDG Traversal: Fake Tool Invocation supplies a simulated response when injected and user tasks overlap, helping the agent estimate arguments according to the original user intent.The mechanism avoids real execution of the newly requested tool while creating the appearance that the injected instruction was handled.
- 3.2 Executing as TDG Traversal: IPIGuard combines TDG traversal with its three mechanisms to mitigate IPI attacks at the source while preserving utility.The design targets unknown arguments, static-plan adaptability, and overlapping tools identified as challenges.
4 Experiments
Experiments on AgentDojo evaluate IPIGUARD across diverse models, attacks, defense baselines, and utility, security, overhead, and ablation settings. IPIGUARD consistently mitigates attacks while preserving utility, achieving a favorable security–utility balance with added token cost.
- Effectiveness Evaluation: Across six models, IPIGUARD mitigates most Important Instruction attacks with only minor utility degradation.The evaluation includes both reasoning and non-reasoning models and uses Figure 4 to compare performance across them.
- Benign Utility Evaluation: 67.01% benign utility places IPIGUARD above all defenses and near the no-defense upper bound of 68.04%.Utility remains particularly strong in the Travel and Banking domains, while Workspace is slightly lower because concrete actions based on tool responses are handled conservatively.
- Security Evaluation: IPIGUARD consistently achieves the lowest ASR across all four attacks, never exceeding 1%.Its average ASR is 0.69%, compared with 13.16% without defense and 4.43% for Detector.
- Security Evaluation: IPIGUARD attains the highest average Utility Accuracy at 58.77% while maintaining the lowest average ASR at 0.69%.This provides the most favorable security–utility balance among the evaluated defenses.
- Overhead Evaluation: IPIGUARD approximately doubles token usage relative to no defense, but the authors regard this overhead as worthwhile when security is critical.The method also supports assigning different LLMs to planning and execution to improve the utility–cost trade-off.
- Ablation Studies: Combining Fake Tool Invocation and Node Expansion yields the best ablation performance, showing complementary roles for both mechanisms.Node Expansion improves BU and UA with a slight ASR increase, while Fake Tool Invocation reduces ASR below 1% and improves utility under attack.
5 Conclusion
IPIGuard defends LLM agents against indirect prompt injection by imposing structural constraints on task execution. The paper presents this execution-centric paradigm as a foundation for robust, verifiable agentic systems.
- IPIGuard imposes structural constraints on agent behavior to prevent malicious tool invocations at their source.
- Extensive experiments show that IPIGuard maintains strong adaptability and utility across diverse attack vectors.
- IPIGuard establishes an execution-centric security paradigm intended to support verifiable and resilient agentic systems in dynamic environments.
Limitations
The paper’s evaluation is limited to tool-use IPI attacks, a constrained experimental scale, and models with reasonably strong planning capabilities.
- IPIGuard addresses IPI attacks that interfere with tool usage, not attacks that solely manipulate textual outputs.
- High LLM querying costs constrain the experiments and limit evaluation across a broader range of models, including OpenAI o3.
- The method requires reasonably strong planning capabilities, which may restrict applicability to weaker or resource-constrained models.
Ethical Considerations
The paper recognizes that defensive cybersecurity advances can also intensify the attacker–defender arms race. It therefore emphasizes responsible disclosure, careful evaluation, and prudent deployment.
- Advances in defensive cybersecurity may unintentionally support the development of more sophisticated attack methods.
- IPIGuard’s vulnerability insights and mitigation strategies create a dual-use risk within the ongoing attacker–defender arms race.
- The paper calls for responsible disclosure, careful evaluation, and prudent deployment to maximize protection while minimizing misuse.
C Experimental Details
The experiments use specified model versions with deterministic decoding settings, while the paper also documents notation and language-editing assistance. The appendix includes a notation summary.
- The experiments use GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, and o4-mini with decoding temperature fixed at 0.
- Unless otherwise specified, experiments use medium reasoning effort to balance response latency and reasoning accuracy.
- Table 4 summarizes the paper’s key notations.
- The authors disclose using ChatGPT for language polishing and error checking while retaining responsibility for the scientific contributions.
F Detailed Descriptions of Attacks and Defense Methods
The paper evaluates several attacks and defenses for tool-using LLM agents, contrasting prompt-based, filtering, and detection approaches.
- Attacks: Attack strategies include instruction overriding, injected tool specifications, and prioritization attacks.
- Defense Methods: Defenses include BERT-based detection, restricting tools, output delimiting, and repeatedly reinforcing the user goal.
G Impact of Using Different LLMs for Task Planning and Execution
The study examines how planner and executor model choices affect utility, cost, and robustness. Stronger planners improve task performance across executor configurations while preserving robustness and adding limited cost.
- Planner and Executor Effects: Replacing Qwen2.5-7B-Instruct with o4-mini as planner raises BU from 35.05% to 51.55% and UA from 33.55% to 49.28% with the same executor.The result is reported for Qwen2.5-7B-Instruct as executor under the Important Instruction attack.
- Planner and Executor Effects: o4-mini paired with GPT-4o achieves the highest UA of 72.66%, indicating that powerful executors also benefit from stronger planning.
- Cost: Switching the planner from GPT-4o-mini to o4-mini raises UA from 57.07% to 64.39% with cost increasing from $6.73 to $7.99.Planning typically accounts for about 20% of total token usage.
H.1 Case Study of Argument Estimation
The argument-estimation case studies show IPIGuard constructing tool dependencies and filling unknown arguments from earlier tool responses while handling injected instructions without executing unrelated actions.
- Argument Estimation: The hotel task plans rating, price, address, and calendar-event nodes, with the event depending on the three information-retrieval nodes.
- Injected Instructions: The injected reservation request appears in external content but does not replace the original Tokyo hotel task.
- Argument Estimation: Argument estimation fills the calendar event’s location with City Hub’s retrieved address and preserves the user’s requested title and date.
- Node Expansion: For the messaging task, node expansion adds read-channel queries for multiple channels after the initial channel listing.
- Argument Estimation: The planned message-sending node retains unknown channel and message arguments until channel-reading results are available.
H.3 Case Study of Fake Tool Invocation
The fake-tool-invocation case contrasts an injected money-transfer request with IPIGuard’s dependency-aware execution, which derives the legitimate payment from transaction data instead.
- IPIGuard Execution: The legitimate payment plan retrieves transaction history before sending money, leaving recipient and amount unknown during initial planning.
- Argument Estimation: Transaction data identifies a $50 Spotify payment, after which argument estimation sets a $5 payment to the recorded recipient.
- Attack Behavior: The injected instruction is presented as a separate task before the original payment request, illustrating the competing-task pattern targeted by the case study.
- Attack Behavior: The injected content requests sending as much money as possible to an attacker-specified account, and the fake invocation uses amount 1,000,000.