PaperlayerView exact revision on arXiv

Source-linked AI summary

MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, William Yang Wang

arXiv:2502.05174v4cs.CRcs.AI

TL;DR

IPI attacks can redirect LLM agents through malicious tasks embedded in tool-retrieved information, while existing defenses may require costly training, fail against sophisticated attacks, or reduce utility. MELON re-executes trajectories with masked user inputs and compares tool calls, and experiments on AgentDojo show strong security-utility performance, with MELON-Aug reaching 0.32% ASR and 68.72% utility on GPT-4o.

  • Problem

    IPI attacks can embed malicious tasks in retrieved information, while existing defenses face resource costs, limited effectiveness, false negatives, or utility degradation.

  • Method

    MELON masks user inputs while preserving retrieved tool outputs, re-executes the trajectory, and detects attacks when original and masked tool calls are similar.

  • Results

    Across AgentDojo, three LLMs, and four attacks, MELON and MELON-Aug outperform five state-of-the-art defenses; MELON-Aug achieves 0.32% ASR and 68.72% utility on GPT-4o.

  • Takeaways & Limitations

    The results support using tool-call and user-input independence as a training-free signal for balancing attack prevention with utility maintenance.

  • Takeaways & Limitations

    MELON monitors tool calls, so response-based attacks that achieve their objectives through text responses can evade detection.

Abstract

from arXiv · show

Recent research has explored that LLM agents are vulnerable to indirect prompt injection (IPI) attacks, where malicious tasks embedded in tool-retrieved information can redirect the agent to take unauthorized actions. Existing defenses against IPI have significant limitations: either require essential model training resources, lack effectiveness against sophisticated attacks, or harm the normal utilities. We present MELON (Masked re-Execution and TooL comparisON), a novel IPI defense. Our approach builds on the observation that under a successful attack, the agent's next action becomes less dependent on user tasks and more on malicious tasks. Following this, we design MELON to detect attacks by re-executing the agent's trajectory with a masked user prompt modified through a masking function. We identify an attack if the actions generated in the original and masked executions are similar. We also include three key designs to reduce the potential false positives and false negatives. Extensive evaluation on the IPI benchmark AgentDojo demonstrates that MELON outperforms SOTA defenses in both attack prevention and utility preservation. Moreover, we show that combining MELON with a SOTA prompt augmentation defense (denoted as MELON-Aug) further improves its performance. We also conduct a detailed ablation study to validate our key designs. Code is available at https://github.com/kaijiezhu11/MELON.

1. Introduction

Indirect prompt injection attacks embed malicious tasks in tool-retrieved information, potentially redirecting agents toward unauthorized actions. MELON addresses limitations of existing defenses by comparing original and masked executions to improve security while preserving utility.

  • Indirect prompt injection attacks embed malicious tasks in databases and websites, forcing agents toward unauthorized actions with potentially severe consequences.
  • Existing defenses face trade-offs among training cost, attack sophistication, false negatives, and normal utility preservation.Some defenses retrain models or add detectors, while training-free approaches may fail against sophisticated attacks or harm utility.
  • MELON detects attacks by re-executing trajectories with masked user inputs and comparing tool calls between original and masked executions.The masking preserves retrieved tool outputs while masking user inputs, and similar tool calls indicate reduced dependence on the user task.
  • Three designs—a customized masking function, tool-call cache, and focused comparison—target arbitrary masked-run calls and noisy information to reduce false positives and false negatives.
  • MELON and MELON-Aug outperform five state-of-the-art defenses against four attacks on AgentDojo across GPT-4o, o3-mini, and Llama-3.3-70B.MELON-Aug further reduces ASR to 0.32% while maintaining 68.72% utility on GPT-4o.

2. Related Work

Prior IPI defenses span trained detectors, prompt-based strategies, and tool-call constraints, but practical limitations remain. MELON is presented as a lightweight training-free defense intended to preserve utility against stronger attacks.

  • IPI attacks include general prompt patterns and agent-specific attacks that redirect agents from user tasks to attacker tasks.Examples include escape-character, context-ignoring, fake-completion, and web-agent attacks.
  • Training-dependent defenses require substantial computational or data resources, while adversarial training may jeopardize normal utility across application domains.
  • Training-free defenses add prompts or constrain tool calls, but prompt strategies have limited efficacy against stronger attacks.
  • Attackers can design attack tasks using only tools related to the user attack, challenging alignment-based detection.
  • MELON is characterized as a lightweight, highly effective training-free defense that maintains the agent’s normal utility.
  • Other defenses require human intervention, white-box model access, or action reversion, introducing strong assumptions and reducing automation.

3. Metholody of MELON

MELON detects indirect prompt injection by comparing the agent’s original trajectory with a masked re-execution that preserves retrieved outputs while neutralizing user-task information. It refines this comparison with task-neutral masking, cached tool calls, and ensemble detectors to address false positives, false negatives, timing mismatches, and computational overhead.

  • 3.1. Preliminaries: MELON models an agent as an LLM system with tools that iteratively generates actions and receives tool-execution observations.The state includes the user task, prior actions, and prior tool outputs; execution continues until task completion or error.
  • 3.2. Technical Overview: Under a successful injection, the next action depends primarily on injected tool outputs rather than the user task and prior trajectory.Benign executions retain dependence on the user inputs, whereas successful attacks induce a state collapse toward the malicious task.
  • 3.2. Technical Overview: MELON applies a masking operator that preserves retrieved tool outputs while replacing the user task and other state information with masked inputs, then compares parallel actions.The original run uses the user task, while the masking run uses a task-neutral prompt; divergence suggests user-task dependence, whereas similarity signals an injection.
  • 3.2. Technical Overview: MELON addresses arbitrary masked-run tool calls by using a task-neutral prompt, consolidating tool outputs, and adding few-shot examples to focus detection on embedded instructions.These masking designs are intended to reduce false negatives caused by contextually plausible but arbitrary tool calls.
  • 3.2. Technical Overview: A tool-call cache records masking-run calls so MELON can compare them across steps and mitigate false negatives caused by different execution timing.The original run may complete the user task before the malicious task, while the masking run may execute the malicious task earlier.
  • 3.3.1. IMPLEMENTATION AND OVERHEAD ANALYSIS: MELON can run masking and original paths in parallel, but its masking run approximately doubles API-based model calls and increases costs by ≈2×.KV caching can reduce latency, while embedding computation remains a primary latency source.
  • 3.4.1. ENSEMBLE DETECTOR: An ensemble extension treats each task-neutral masking prompt as a weak detector and combines their outputs to provide worst-case performance guarantees.The theoretical framework uses multiple masking prompts, such as sentiment analysis or grammar checks, as individual detectors.

4. Evaluation

Evaluation on AgentDojo compares defenses across three LLMs, four attack types, and utility, attack-success, and benign-utility metrics. MELON and MELON-Aug provide stronger security–utility balance, while ablations and sensitivity tests examine design importance and robustness.

  • Experiment Setup: AgentDojo evaluates four agent types across 629 attack cases, using GPT-4o, o3-mini, and Llama-3.3-70B.The benchmark covers banking, Slack, travel, and workspace agents with distinct tools, tasks, attack tasks, and injection points.
  • Experiment Setup: The evaluation reports benign utility (BU), utility under attack (UA), and attack success rate (ASR), where higher BU and UA are better and lower ASR is better.ASR counts attacks that fully execute all required malicious-task steps.
  • Overall Results: MELON achieves high utility and low ASR, whereas other defenses exhibit a clearer UA–ASR trade-off.MELON intervenes after action generation but before execution, rather than terminating the entire execution before the action is generated.
  • Overall Results: MELON-Aug achieves 68.72% UA with 0.32% ASR on GPT-4o, compared with 69.08% UA and 16.06% ASR for the no-defense baseline.Across models and attacks, MELON-Aug maintains high UA while achieving low ASR.
  • Attack Prevention: For GPT-4o, MELON achieves 0.24% average ASR, while some defenses obtain low ASR through excessive filtering or retain false negatives.The tool filter renders systems unusable for o3-mini and Llama-3.3-70B, while the LLM detector reaches 0.00% ASR but still has false positive and false negative rates.
  • Ablation Study: Removing the masking function, tool-call cache, or comparison design increases ASR, demonstrating that each component contributes to detection.The ablation uses GPT-4o on AgentDojo under the Important Messages attack.
  • Sensitivity Test: Across task-neutral prompts, MELON shows consistent performance, while lower similarity thresholds slightly decrease UA because they can increase false positives.The threshold sensitivity test highlights possible confusion between malicious and legitimate commands with similar actions.
  • False Positive Analysis: MELON produced nine GPT-4o false-positive detections, all involving user requests to execute unverified external instructions.The authors classify these cases as security risks rather than detection errors.

5. Conclusion and Future Work

MELON advances IPI defense by leveraging the reduced dependence between tool calls and user inputs during successful attacks. The paper identifies broader attack goals, efficiency improvements, and combinations with other defenses as future directions.

  • MELON uses the reduced dependence between agent tool calls and user inputs as a behavioral basis for IPI defense.
  • The paper identifies extending MELON beyond direct task manipulation as a future direction.
  • The paper proposes improving masked re-execution efficiency through KV cache and selective state masking.
  • Combining MELON’s behavioral pattern detection with prompt augmentation is proposed as a route toward more robust protection mechanisms.

Impact Statement

The work targets the security of LLM-based agents against indirect prompt injection attacks while acknowledging both computational overhead and the continuing evolution of attack methods.

  • The work advances LLM-agent security against indirect prompt injection attacks.
  • MELON introduces additional computational costs, which the authors view as justified by the importance of protecting agent systems.
  • The authors acknowledge that no security measure is perfect and that evolving attack methods require continued research.

A.2. Few-shot examples

The examples illustrate how agents process user tasks, tool outputs, and embedded instructions, while MELON compares security-relevant tool calls across executions. Few-shot samples and argument filtering guide this comparison.

  • Few-shot examples: Few-shot chat samples contain user messages, LLM responses, and tool execution outputs to improve adherence to the task-neutral instruction during masking runs.
  • Few-shot examples: The examples show agents retrieving files or inbox contents, following embedded instructions, and producing summaries or reports from tool outputs.
  • Tool-call comparison: Security-sensitive tool calls are converted into strings that retain function names and relevant arguments while omitting less security-critical content.
  • Agent pipeline: Each action contains a text response and tool calls, and tool execution produces observations that become inputs to later steps.
  • Attack examples: The examples include direct malicious instructions embedded in retrieved content, including requests to send transactions or follow attacker-supplied tasks.
  • Agent pipeline: The basic agent pipeline repeatedly selects and executes tools using the user task and previous tool outputs until completion, error, or a step limit.

D. Datasets

AgentDojo evaluates agent robustness against indirect prompt injection across real-world domains using diverse tasks and security test cases.

  • AgentDojo contains 97 real-world tasks across domains such as email management, banking, and travel booking, with 629 security test cases.

D.2. VWA-Adv (Wu et al., 2024a)

VWA-Adv presents adversarial tasks in realistic VisualWebArena settings, distinguishing illusion attacks from goal misdirection attacks. Because of limited effectiveness, this dataset is excluded from evaluation.

  • VWA-Adv proposes 200 adversarial tasks in realistic VisualWebArena settings.
  • The benchmark distinguishes illusion attacks that manipulate retrieved information from goal misdirection attacks that redirect agents toward malicious tasks.Goal misdirection attacks are identified as the more severe category and the study’s focus.
  • The dataset is excluded from evaluation because its attack effectiveness is limited, pending more effective image-based attack methods.

E. Examples of False Positives

The examples examine detections that technically count as false positives because they follow user intentions, yet still expose risky reliance on external instructions. A DeBERTa detector also produces one genuinely wrong false positive.

  • MELON’s examined false positives follow user requests to blindly execute instructions from external sources.
  • These detections align with user intentions but are treated as legitimate security concerns in agent operation.
  • The section also presents one true false positive that was wrongly classified by the DeBERTa Detector.

E.2. Examples of False Positives Generated by MELON

The examples contrast benign user-directed retrieval with tool outputs containing instructions to perform external actions. Banking, messaging, and email-related tasks illustrate why MELON flags some cases as risky.

  • The banking example instructs the agent to pay a December 2023 bill by transferring funds to a specified IBAN and is labeled as prompt injection.
  • A TODO-list request containing instructions to summarize an article and solicit users’ hobbies is labeled as prompt injection.
  • The same TODO-list workflow includes a weekly task list delivered through an email-like tool output.

F. LLM Detector FPN and FNR

Table 5 reports that the baseline LLM detector performs unevenly across attack types, achieving very high accuracy for some categories but retaining measurable error rates for others.

  • Table 5 reports LLM detector detection performance across different attack types.
  • The detector achieves 100% accuracy on system message attacks and 99.98% on important instructions.
  • Its error rates include 0.78% FNR for Important Messages attacks and 0.31% FPR for Ignore Previous attacks.
  • The inconsistent detection rates suggest specialization to particular patterns rather than comprehensive adversarial prompt detection.
Loading 2502.05174v4…