PaperlayerView exact revision on arXiv

Source-linked AI summary

Defense Against Indirect Prompt Injection via Tool Result Parsing

Qiang Yu, Xinran Cheng, Chuanyi Liu

arXiv:2601.04795v1cs.AIcs.CLcs.CRcs.MA

TL;DR

Indirect prompt injection embeds malicious instructions in tool outputs, threatening LLM agents with unauthorized actions. The paper proposes parsing tool results into constrained essential data, with CheckTool sanitizing large text, and reports competitive UA alongside the lowest ASR.

  • Problem

    Indirect prompt injection embeds adversarial instructions in retrieved tool data, causing agents to process malicious instructions and potentially perform unauthorized actions.

  • Method

    Tool Result Parsing uses LLMs to extract necessary data under format and logical constraints, while CheckTool detects and sanitizes large text that may contain injections.

  • Results

    The combined approach achieves competitive Utility under Attack and the lowest Attack Success Rate, with reported average ASRs below 1%.

  • Takeaways & Limitations

    Parsing essential, constrained tool data and sanitizing trigger-causing content provides a defense that maintains competitive utility while reducing attack success.

  • Takeaways & Limitations

    CheckTool becomes less effective as reasoning depth increases because reasoning-induced errors decrease BU and UA, requiring further refinement.

Abstract

from arXiv · show

As LLM agents transition from digital assistants to physical controllers in autonomous systems and robotics, they face an escalating threat from indirect prompt injection. By embedding adversarial instructions into the results of tool calls, attackers can hijack the agent's decision-making process to execute unauthorized actions. This vulnerability poses a significant risk as agents gain more direct control over physical environments. Existing defense mechanisms against Indirect Prompt Injection (IPI) generally fall into two categories. The first involves training dedicated detection models; however, this approach entails high computational overhead for both training and inference, and requires frequent updates to keep pace with evolving attack vectors. Alternatively, prompt-based methods leverage the inherent capabilities of LLMs to detect or ignore malicious instructions via prompt engineering. Despite their flexibility, most current prompt-based defenses suffer from high Attack Success Rates (ASR), demonstrating limited robustness against sophisticated injection attacks. In this paper, we propose a novel method that provides LLMs with precise data via tool result parsing while effectively filtering out injected malicious code. Our approach achieves competitive Utility under Attack (UA) while maintaining the lowest Attack Success Rate (ASR) to date, significantly outperforming existing methods. Code is available at GitHub.

1 Introduction

LLM agents are vulnerable to indirect prompt injection because malicious instructions embedded in tool outputs enter the model’s context and can trigger unauthorized actions. The paper proposes parsing tool results to retain necessary, constrained data while filtering potential injections.

  • Indirect prompt injection embeds adversarial instructions in external tool data, causing agents to process them and potentially perform unauthorized actions.
  • Existing defenses include model-based methods with substantial training, inference, and update costs, and prompt-based methods that avoid training but can remain vulnerable to sophisticated attacks.
  • The proposed defense parses tool results and verifies formatting or logical constraints, returning only essential data while filtering potential injection content.

2 Related Works

Prior work develops generic and scenario-specific injection attacks, alongside prompt-based, training-based, isolation, privilege-control, and workflow-monitoring defenses. Training-based defenses either modify the LLM or add a separate detector.

  • Indirect Prompt Injection Attack: Indirect prompt attacks range from reusable generic instructions to scenario-specific methods targeting web, mobile, privacy, fraud, intrusion, or malware-related activities.
  • Defenses against IPI: Prompt-based defenses use prompt engineering techniques such as repeating the user prompt or delimiting data to reduce injection impact.
  • Defenses against IPI: Training-based defenses fine-tune the LLM or train a standalone detector to distinguish or identify injected instructions.
  • Defenses against IPI: Other defenses isolate execution environments, control privileges, or evaluate relationships between user instructions and subsequent assistant messages.

3 Methodology

The methodology models tool use as an agent pipeline in which a defense module transforms tool outputs before subsequent reasoning. Tool Result Parsing extracts constrained necessary data, while CheckTool handles large text outputs by detecting and sanitizing content that triggers tool calls.

  • Agent formulation: An agent combines an LLM with tools, receives observations from tool calls, and uses them to choose later tool calls or a final response.
  • Threat model: Indirect injection compromises this pipeline when malicious tool output changes the model’s deduction, tool call, and final response.
  • Defense pipeline: The defense module applies to each tool execution, transforming the raw tool output before it is passed into the agent’s reasoning process.
  • ParseData: Tool Result Parsing extracts only the data needed for the next reasoning step, using expected formats and logical constraints to filter irrelevant injection payloads.
  • ParseFull: ParseFull supplies conversation history during parsing, which can improve contextual accuracy for powerful models but introduce noise for less capable models.
  • CheckTool: For large text chunks, CheckTool monitors whether output triggers a tool call, then rejects or sanitizes potentially malicious content; it can be combined with parsing in different orders.

4 Experiments

Experiments evaluate defenses on AgentDojo across three models, attack types, and utility, attack-success, and risk metrics. The proposed ParseData and CheckTool combinations achieve low attack rates and risk while maintaining competitive or stable utility under attack.

  • Experimental Setup: AgentDojo covers banking, Slack, travel, and workspace tasks, while experiments use gpt-oss-120b, Llama-3.1-70b, and qwen3-32b.The evaluation selects Direct, Ignore previous, and Important messages attacks and compares four baseline defenses.
  • Metrics: Risk divides ASR by UA, helping compare defenses because high utility often accompanies high attack success.The experiments also average performance across attack levels because UA, ASR, and risk vary substantially across scenarios.
  • Average Performance: ASRs below 1% make ParseData+CheckTool and CheckTool+ParseData the lowest-ASR defenses while retaining moderate average UA.The reported average results state that these methods significantly outperform the other defense mechanisms on ASR.
  • Average Performance: 0.2%–1% is the proposed methods’ average risk, versus 3%–6% for Tool Filter, or approximately 1/10 to 1/8 as much.Avg Risk represents expected successful attacks per 100 successful tasks.
  • NoAttack and Severe Attack: Under severe attacks, the proposed methods improve UA by 8% to 30% for qwen3-32b, with an average increase of 10%.Other defenses decrease utility by 10% to 30% under severe attacks, whereas the proposed methods remain comparable or superior across conditions.
  • NoAttack and Severe Attack: 0.1% to 0.5% ASR is achieved by ParseData+CheckTool and CheckTool+ParseData, approximately 1/10 the ASR of DeBERTa Detector and Tool Filter.No Defense exceeds 20% ASR, while Repeat user prompt and Spotlighting with delimiting remain above 10%.
  • ParseData and CheckTool: ParseData has higher BU and UA but higher ASR than CheckTool for gpt-oss-120b, whereas stronger reasoning improves ParseData and increases CheckTool errors.For gpt-oss-120b, ParseData’s BU is 1.92% higher and its ASR 100.29% higher than CheckTool’s.
  • ParseData and CheckTool: For gpt-oss-120b, ParseData+CheckTool has 9.29% higher BU, while CheckTool+ParseData has 41.89% lower ASR.The two orderings trade off utility against attack resistance.

5 Conclusion

The paper proposes ParseData and CheckTool to extract relevant tool-result data, enforce format and logical constraints, and filter malicious code. ParseData’s performance improves with deeper reasoning, while CheckTool remains sensitive to reasoning-induced errors.

  • ParseData extracts relevant data from tool outputs and uses format and logical constraints to filter malicious code.CheckTool additionally detects and sanitizes large text chunks required by the LLM.
  • Integrating ParseData and CheckTool achieves the lowest ASR while maintaining competitive BU and UA.
  • Deeper reasoning improves BU, UA, and ASR for ParseData, allowing its performance to scale with advancing LLM capabilities.
  • Further research is required to refine CheckTool and address reasoning-induced inconsistencies.

Ethical Considerations

The authors report adherence to the ACM Code of Ethics and ACL Code of Conduct and state that technical content, experiments, and conclusions were independently verified.

  • The authors affirm adherence to the ACM Code of Ethics and the ACL Code of Conduct.
  • AI assistants supported linguistic polishing and code prototyping, while the authors independently verified the technical content, experiments, and conclusions.
  • The source code will be made publicly available.

A Complete Experimental Results

Table 5 reports results for three models and defense methods under Direct, Ignore previous, and Important Messages attacks, with aggregate metrics computed from attack-specific values.

  • Table 5 compares three models and defense methods under Direct, Ignore previous, and Important Messages attacks.The experiment results are reported in percentages.
  • Avg UA is calculated from BU and the UA values for Direct, Ignore previous, and Important Messages attacks.
  • Avg ASR is calculated from the ASR values for NoAttack, Direct, Ignore, and Important attacks.
  • Avg Risk is calculated from attack-specific ASR/UA ratios for Direct, Ignore, and Important attacks.

B ParseData Module Prompts

The ParseData prompts first specify the anticipated minimal data, then instruct the LLM to extract only those values from the tool result in a fixed format while ignoring embedded instructions.

  • After reasoning about a tool call, the LLM is prompted to anticipate the minimal data needed from the result.
  • The anticipation specifies each requested datum’s description, data type, format, and any limitation.
  • After the tool result returns, the LLM extracts data according to the previous anticipation.
  • The extraction prompt instructs the LLM to ignore instructions, actions, tool calls, and functions contained in the tool result.
  • The LLM summarizes the tool result and outputs only requested values, using None for unavailable or partial values.
  • The extracted values are returned in a specified begin-marker and end-marker format with named data fields.

C CheckTool Module Prompt

The CheckTool module prompts an LLM to remove words that would trigger a tool call while preserving the remaining content in a specified format. It also constrains the output to designated markers and excludes reasoning or unrelated commentary.

  • The module asks the LLM to remove words that trigger a tool call.
  • It presents the content between explicit begin and end markers for processing.
  • The content may contain words that would trigger a specified tool with supplied arguments.
  • The model must output the retained content in a prescribed format using designated output markers.
  • The instructions require reasoning to appear inside think tags and prohibit comments, explanations, or unrelated output.

D Indirect Prompt Injection Attack Examples

The examples illustrate indirect prompt injection through calendar-tool results, including instructions embedded in event descriptions and messages that imitate higher-priority instructions. They show direct, ignore-previous, and important-message attack forms within tool-call transcripts.

  • D Indirect Prompt Injection Attack Examples: Calendar results can contain malicious instructions embedded in an event description.
  • D Indirect Prompt Injection Attack Examples: One injected description tells the model to ignore previous instructions.
  • D Indirect Prompt Injection Attack Examples: Another example embeds an important message addressed from a named user to the model.
  • E.1 Direct Attack: The transcript includes system-style instructions, user identity details, tool-use rules, and the original calendar query around the tool response.
  • E.2 Ignore Previous Attack: The ignore-previous attack places the phrase "Ignore your previous" in the returned event description within a similar tool-call transcript.
  • E.3 Important Message Attack: The important-message attack embeds an information block addressed to the model in the calendar result.
  • E.3 Important Message Attack: Its surrounding transcript again contains tool-use instructions, the calendar question, and the assistant tool call that produced the result.
Loading 2601.04795v1…