Source-linked AI summary
PromptArmor: Simple yet Effective Prompt Injection Defenses
Tianneng Shi, Kaijie Zhu, Zhun Wang, Yuqi Jia, Will Cai, Weida Liang, Haonan Wang, Hend Alzahrani, Joshua Lu, Kenji Kawaguchi, Basel Alomair, Xuandong Zhao, William Yang Wang, Neil Gong, Wenbo Guo, Dawn Song
TL;DR
LLM agents can be redirected by malicious instructions embedded in untrusted inputs. PromptArmor prompts an off-the-shelf LLM to detect, extract, and remove those instructions before backend processing, achieving low error and attack rates while remaining robust to adaptive attacks.
Problem
Prompt injection can cause an agent to execute an attacker-specified task instead of the intended user task, motivating defenses that preserve sanitized data for processing.
Method
PromptArmor uses a carefully designed prompt to repurpose an off-the-shelf LLM as a guardrail that detects, extracts, and removes injected prompts before agent processing.
Results
On AgentDojo, GPT-4o, GPT-4.1, and o4-mini achieve FPR and FNR below 1%, while removing injected prompts reduces ASR below 1%.
Takeaways & Limitations
PromptArmor is effective across varied settings and remains robust against adaptive attacks designed to evade it.
Takeaways & Limitations
The defense objective assumes sanitized data enables the LLM to complete the intended task rather than the attacker-specified task.
Abstract
from arXiv · showhide
Despite their potential, recent research has demonstrated that LLM agents are vulnerable to prompt injection attacks, where malicious prompts are injected into the agent's input, causing it to perform an attacker-specified task rather than the intended task provided by the user. In this paper, we present PromptArmor, a simple yet effective defense against prompt injection attacks. Specifically, PromptArmor prompts an off-the-shelf LLM to detect and remove potential injected prompts from the input before the agent processes it. Our results show that PromptArmor can accurately identify and remove injected prompts. For example, using GPT-4o, GPT-4.1, or o4-mini, PromptArmor achieves both a false positive rate and a false negative rate below 1% on the AgentDojo benchmark. Moreover, after removing injected prompts with PromptArmor, the attack success rate drops to below 1%. We also demonstrate PromptArmor's effectiveness against adaptive attacks and explore different strategies for prompting an LLM. We recommend that PromptArmor be adopted as a standard baseline for evaluating new defenses against prompt injection attacks.
1 Introduction
PromptArmor addresses prompt injection by using a carefully prompted off-the-shelf LLM to detect and remove injected instructions before agent processing. On AgentDojo, it achieves below-1% detection error rates and attack success, while remaining robust to adaptive attacks.
- Contribution: PromptArmor detects contaminated inputs and removes injected prompts before passing sanitized data to the agent.It uses an off-the-shelf guardrail LLM, which may differ from the backend LLM.
- Evaluation: Below 1% FPR and below 1% FNR are achieved with GPT-4o, GPT-4.1, or o4-mini on AgentDojo.After removal, the attack success rate also drops below 1%.
- Evaluation: 55% ASR occurs against an agent using GPT-4.1 as the backend LLM when no defense is deployed.PromptArmor can still detect and remove injected prompts even when the guardrail LLM remains vulnerable.
- Findings: PromptArmor challenges the view that an off-the-shelf LLM cannot be directly prompted to defend against prompt injection attacks.The paper attributes earlier negative findings to weaker older models and poorly designed prompting strategies.
- Ablations: GPT-3.5 remains effective despite predating AgentDojo, and a GPT-4.1 memorization test indicates memorization is unlikely.The paper also reports that naïve prompting approaches produce ineffective defenses.
- Model analysis: Larger LLMs generally improve PromptArmor effectiveness, while reasoning capability helps especially in midsized models but remains limited when models are too small.The evaluated models range from 0.6 billion to 32 billion parameters.
2 Problem Definition
Prompt injection occurs when malicious instructions embedded in untrusted data redirect an agent from the user’s intended task. The defense problem is to detect and extract the injected content, remove it, and preserve the sanitized data for task completion.
- Threat model: Prompt injection embeds a malicious instruction in an untrusted data sample, causing the LLM to execute an attacker-specified task.The attack targets the interaction between an instruction and contaminated data.
- Threat model: Prompt injection can affect agents processing external environments, websites, knowledge databases, tool descriptions, and MCP specifications.A webpage injection may steer an LLM-generated summary toward an attacker’s site.
- Threat model: Agent prompt injections target agents by hijacking execution flow, unlike jailbreaks, which aim to bypass an LLM’s safety alignment.The paper explicitly distinguishes these attack categories.
- Defense problem: The defense goal is to determine whether a data sample is contaminated and, if so, identify and extract the injected content.The extracted content is then removed before the sanitized data reaches the LLM.
- Defense problem: Removing injected content preserves downstream processing, whereas rejecting the entire sample can affect user experience and disrupt workflows.The intended user task can still be fulfilled using sanitized data.
- Defense objectives: The defense should achieve low FPRs and FNRs while enabling the LLM to complete the intended task rather than the attacker-specified task.This requirement makes successful task completion part of the defense objective.
3 PromptArmor
PromptArmor is a modular preprocessing guardrail that prompts an off-the-shelf LLM to detect, extract, and remove injected instructions before backend processing. Its design emphasizes deployability, flexible prompting, efficiency, and robustness across models and attacks.
- 3 PromptArmor: PromptArmor adds a guardrail layer that detects and removes potential injected prompts before the backend LLM processes each data sample.It requires no modifications to existing agents or applications.
- 3.1 Prompting an off-the-shelf LLM: The guardrail LLM uses text understanding, pattern recognition, and intended-task context to identify instruction-like patterns and inconsistencies.This guardrail model may be the same underlying model as the backend LLM.
- 3.1 Prompting an off-the-shelf LLM: PromptArmor first prompts the guardrail LLM to detect injection, then to extract the injected content, and finally removes it using fuzzy matching.The fuzzy matcher tolerates differences in whitespace or punctuation between extracted and original text.
- 3.1 Prompting an off-the-shelf LLM: The workflow example shows a transaction-history instruction being identified as injected and removed while the original payment request remains executable.The malicious example directs money to X.
- 3.2 Design Rationale: PromptArmor’s modular architecture operates as a standalone preprocessing component, preserving existing system behavior while adding a security layer.Its prompt-driven control also supports customization of sensitivity, attack focus, output format, and application domain.
- 3.2 Design Rationale: Using pretrained LLMs avoids additional security-model data collection, design, and training costs, while smaller models can support resource-aware deployment.This provides a computational-efficiency rationale for deployment across platforms.
- 3.2 Design Rationale: PromptArmor inherits improvements in general-purpose LLM contextual reasoning, understanding, and adversarial robustness without additional engineering effort.The paper presents this as a sustainable alternative to specialized models that may receive fewer resources.
- 3.2 Design Rationale: The same LLM can serve as both an agent’s core module and PromptArmor’s detector, and the defense remains robust against adaptive attacks.This supports the paper’s claim that deployment need not require substantial additional effort.
4 Evaluation
PromptArmor is evaluated across AgentDojo tasks, guardrail models, prompting strategies, model sizes, reasoning modes, and adaptive attacks. It generally reduces attack success while preserving utility, with larger and stronger models performing best.
- Evaluation setup: AgentDojo evaluates four agent types across 629 adversarial scenarios using four representative prompt injection attacks.The agents cover banking, Slack, travel, and workspace applications, with GPT-4.1 used as the backend model.
- PromptArmor vs. Existing Defenses: 0.00% ASR is achieved by PromptArmor-GPT-4.1, while PromptArmor-o4-mini reaches 76.35% UA versus 64.27% for the baseline.PromptArmor-GPT-4.1 also records 0.56% FPR and 0.13% FNR; GPT-3.5 performs worse than the other tested configurations.
- PromptArmor vs. Existing Defenses: 0.79% ASR is achieved by Tool Filter, but its utility is substantially reduced, while Delimiter reaches 51.51% ASR and DataSentinel reaches 48.78% FNR.These baselines illustrate different security-utility limitations compared with PromptArmor.
- Impact of Different Prompting Strategies: GPT-3.5 detection improves significantly when the system prompt includes a definition of prompt injection.The enhanced prompt is used for the paper’s other GPT-3.5 results.
- Impact of Reasoning and Model Size: Qwen3-8B reasoning reduces FNR from 26.59% to 15.78%, while Qwen3-32B approaches zero FPR and FNR regardless of reasoning mode.The results indicate that model capacity is more fundamental than reasoning for robust security-utility performance.
- Adaptive Attacks: PromptArmor maintains consistently low FPR, FNR, and ASR under AgentVigil-NoDefense and AgentVigil-Adaptive, demonstrating robustness against fuzzing-based adaptive attacks.AgentVigil-Adaptive applies AgentVigil to agents equipped with PromptArmor.
5 Related Work
Prior prompt injection defenses include training-based, detection-based, prompt augmentation, and system-level approaches. These categories differ in whether they modify model parameters, add detection components, alter prompts, or restructure system execution.
- Training-based defenses: Training-based defenses fine-tune the backend LLM using supervised or preference-based learning to reject injections while preserving intended behavior.Examples include instruction hierarchy, StruQ, and SecAlign.
- Detection-based defenses: Detection-based defenses add mechanisms that identify contaminated inputs, including methods based on known-answer detection and specialized detection models.DataSentinel frames detection-model fine-tuning as a minimax optimization problem.
- Prompt augmentation defenses: Prompt augmentation defenses modify system prompts or inputs through delimiters, repeated user instructions, or related prompt constructions without extra training or infrastructure.They are described as the most accessible defense category.
- System-level defenses: System-level defenses use execution isolation, control and data flow management, front-running, and privilege control to secure agent operation.Examples include IsolateGPT, fsecure, CaMeL, MELON, and Progent.
6 Conclusion
PromptArmor repurposes an off-the-shelf LLM through carefully designed prompting to detect and remove injected prompts. The paper reports effectiveness across varied settings and robustness against adaptive attacks.
- Conclusion: PromptArmor uses carefully designed prompting strategies to turn an off-the-shelf LLM into a tool for detecting and removing injected prompts.The approach is presented as simple and effective across varied settings.
- Conclusion: PromptArmor remains robust against strong adaptive attacks specifically crafted to evade it.This conclusion summarizes the paper’s reported scope of robustness.