Source-linked AI summary
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
Qiusi Zhan, Richard Fang, Henil Shalin Panchal, Daniel Kang
TL;DR
IPI attacks exploit external content processed by LLM agents, while existing defenses had not been evaluated against adaptive attacks. The paper tests eight defenses with adaptive attack methods and finds success rates above 50% across targeted defenses and agents, motivating adaptive evaluation of defense robustness.
Problem
Existing IPI defenses had not been evaluated against adaptive attacks, leaving their robustness in question despite the risks of malicious external instructions.
Method
The paper implements eight IPI defenses on two LLM-agent types and designs adaptive attacks using optimized adversarial strings tailored to each defense.
Results
Adaptive attacks consistently achieve attack success rates above 50% across the targeted defenses and LLM agents, exceeding pre-defense ASR and outperforming non-adaptive attacks.
Takeaways & Limitations
The findings underscore that evaluating defenses against adaptive attacks is necessary for assessing robustness and reliability.
Takeaways & Limitations
The adaptive attacks target only the agent’s first-step action and do not account for combinations of defenses.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.
1 Introduction
LLM agents use external tools in high-stakes applications, but external content enables indirect prompt injection attacks that can cause serious harm. The paper evaluates defenses against adaptive attacks and finds that all tested defenses remain vulnerable.
- Motivation: External tools expand LLM agents’ capabilities but expose them to malicious instructions embedded in accessed data.IPI attacks can manipulate agent behavior through sources such as reviews or emails.
- Motivation: Adaptive attacks are designed after a defense is disclosed and are necessary to test whether defenses withstand future threats.Prior work shows non-adaptive testing can create a false sense of security.
- Approach: The paper crafts adversarial external content using attack strategies adapted from jailbreak research.Because attackers manipulate only external content, the adaptive attacks optimize inserted strings to influence agent behavior.
- Results: Adaptive attacks consistently achieve success rates above 50% across eight defenses and two types of LLM agents.They exceed the ASR before defense deployment and significantly outperform non-adaptive attacks.
2 Preliminaries
The paper models an LLM agent as a language model equipped with tools that process benign instructions and external tool responses. An IPI attack embeds a malicious instruction in that external content to trigger an attacker tool.
- Agent model: An agent consists of an LLM M and tool set T, receiving benign instruction Iu that invokes user tool Tu and response RTu.The notation represents the agent’s model, tools, user instruction, and corresponding tool response.
- IPI attack: Attackers manipulate external content ETu within RTu by embedding malicious instruction Ia that directs execution of attacker tool Ta.The external content may come from an email or review.
- IPI attack: The agent combines benign and malicious instructions, and the attack succeeds when its output commands execution of Ta.Success is identified by an attacker-tool command such as “Action: <Ta>”.
3 Defense Techniques
The paper evaluates representative IPI defenses spanning detection, prompt modification, and model finetuning. These defenses flag suspicious responses, separate external content from instructions, alter prompts, paraphrase inputs, or train model resistance.
- Detection-based Defense: Detection-based defenses classify tool responses using a finetuned detector, an LLM detector, or perplexity thresholds.Perplexity filtering flags responses exceeding a threshold calibrated from the original attack responses.
- Prompt-based Defense: Instructional Prevention tells the model to ignore commands from external content.It explicitly warns the model about IPI attacks.
- Prompt-based Defense: Data Prompt Isolation wraps tool responses in delimiters to distinguish them from the surrounding context.The implementation uses triple apostrophes around the tool response.
- Prompt-based Defense: Sandwich Prevention appends a user instruction after the tool response to reinforce the legitimate command.This changes the prompt structure rather than filtering or rewriting the external content.
- Model-based Defense: Paraphrasing rewrites external content to disrupt token-level optimized adversarial strings, while adversarial finetuning trains the model on unsuccessful attack examples.Finetuning aims to teach the model to ignore malicious instructions in tool responses.
4 Adaptive Attack Techniques
Adaptive attacks insert optimized adversarial strings into attacker-controlled external content to induce malicious tool execution while bypassing specific defenses. The paper adapts GCG-based optimization, adds stealth objectives for detectors, and uses specialized strategies for paraphrasing.
- Attack formulation: Adaptive attacks insert adversarial suffixes or prefixes around the attacker instruction in external content.The objective is to cause the model to execute the malicious command and invoke attacker tool Ta.
- GCG-based Attacks: GCG optimizes a discrete adversarial string so the agent generates a target output that leads to attacker-tool execution.The target is chosen according to the agent’s behavior.
- Detection bypass: Multi-objective GCG jointly optimizes attack success and detector evasion by making the tool response appear benign.For LLM detectors, the stealth objective maximizes the likelihood of a “No” response; different tokenizers require alternating optimization.
- Paraphrasing bypass: Two-stage GCG targets paraphrasing by first training a string that elicits the desired output and then training a second adversarial string.The strategy is adapted from prior two-step adversarial-string generation.
- Perplexity bypass: AutoDAN generates semantically meaningful adversarial strings to address the high perplexity of token-level GCG outputs.The motivation is to make adversarial strings less obvious to perplexity filtering.
5 Experiments
The experiments evaluate eight IPI defenses and adaptive attacks on prompted and finetuned LLM agents using ASR, target rate, and detection rate. Adaptive attacks exceed 50% ASR across evaluated defenses and agents, while target rate and ASR can diverge.
- Experimental Setup: ASR-all is the primary metric, with ASR-defense measuring defended attacks and ASR-adaptive attack measuring attacks against each defense.Invalid outputs are not credited as successful attacks.
- Results: Adaptive attacks exceed 50% ASR across different defenses and agents, showing that the evaluated defenses can be circumvented.Figure 2 compares defended ASRs, adaptive-attack ASRs, and original attack ASRs.
- Results: For Vicuna-7B, most defenses reduce the original 56% ASR, including adversarial finetuning to 12%, while Llama3-based agents start at 9% original ASR.Perplexity filtering, instructional prevention, paraphrasing, and data prompt isolation retain higher ASRs for Vicuna-7B; adaptively trained strings still bypass defenses.
- Limitations: ASR-defense alone cannot determine which defense is better because the experiments do not assess effects on normal cases.A good defense should also minimize impact on normal cases, but this study focuses on adaptive-attack robustness.
- Results: Target rate and ASR do not always coincide: GCG over adversarial finetuning has a relatively high target rate but lower ASR, whereas AutoDAN has the lowest target rate but high ASR.Counting semantically similar AutoDAN outputs as targets increases its target rate to 62%.
6 Analysis
The analysis shows that adaptive adversarial strings bypass defenses more effectively than non-adaptive attacks, while their impact varies across defense types, output validity, and attack stages.
- Detection-based Defense: Adaptive attacks bypass detectors, reducing detection rates to nearly zero in most cases.GCG-generated strings raise detection to 65% before adaptation, but adaptive strings evade detection substantially.
- Input-level Defense: Most prompt-modification defenses provide weak protection for the Vicuna-7B agent, while sandwich prevention performs best among prompt designs on valid outputs.Its apparent strength is partly associated with producing more invalid cases than other defenses.
- Model-level Defense: The GCG attack can achieve a high target rate but relatively low ASR against adversarial finetuning because many data-stealing attempts fail at the second step.The attack strings are trained to control the agent’s response only immediately after external content is received.
- Defenses of Adversarial Attacks: Adaptive adversarial strings achieve high ASRs against perplexity filtering and paraphrasing and outperform non-adaptive strings.These two defenses specifically target adversarial strings, making their adaptive bypass especially relevant.
- Impact on Valid Rate: Adversarial strings can both increase invalid outputs through contextual chaos and increase valid outputs by forcing ReAct-format responses beginning with “Thought:”.The valid rate is the ratio of valid outputs for the Vicuna-7B agent.
- Attack stages: Adaptive attacks produce minimal ASR improvement in the second stage of data stealing because their trained target output directly influences only the first step.This limits their effect on later agent actions.
7 Related Work
Related work covers multiple threats to LLM agents and motivates adaptive evaluation as a way to test whether defenses remain robust after disclosure. This paper applies jailbreak attack strategies to indirect prompt injection against tool-using agents.
- Agent Attacks: LLM agents face intentional threats including indirect prompt injection, retrieval-augmented generation poisoning, and backdoor attacks.IPI attacks place malicious instructions in tool responses, while RAG poisoning targets knowledge bases and backdoors use finetuned triggers.
- Indirect Prompt Injection: IPI attacks are especially consequential because they can manipulate tool-using agents toward harmful actions rather than merely eliciting target text.Agent attacks also involve more complex inputs and outputs, and success can be measured through attacker-tool execution.
- Adaptive Attacks: Automated jailbreak methods such as GCG and AutoDAN provide attack strategies that can be adapted to LLM-agent IPI settings.The paper evaluates these strategies under more challenging defense scenarios.
- Adaptive Evaluation: Adaptive attacks are necessary because defenses that appear robust against non-adaptive attacks may be compromised once attackers design against the disclosed defense.The paper presents the first study focused specifically on adaptive attacks targeting LLM-agent safety.
8 Conclusion
The paper analyzes eight IPI defenses and designs adaptive attacks for each, successfully breaking all of them. It argues that adaptive-attack evaluation is necessary for developing effective defenses against evolving threats.
- Adaptive attacks successfully break all eight analyzed defenses against IPI attacks on LLM agents.
- The evaluation demonstrates that defenses must address future threats in addition to existing attacks.
- Thorough evaluation, including adaptive attacks, is presented as a key factor in designing effective defenses.
9 Ethical Considerations
The paper identifies vulnerabilities in current IPI defenses and presents straightforward, literature-based techniques for exploiting them. It discloses the work while recognizing the dual-use ethical concern.
- The introduced techniques exploit weaknesses in current IPI defenses and are straightforward to implement.
- The techniques are adapted from existing attack methods and could eventually be discovered by teams seeking to compromise LLM agents.
- The paper’s primary ethical concern is the dual-use nature of the work.
10 Limitations
The study’s limitations concern attack horizon, attacker knowledge, defense combinations, defense coverage, and selected experimental implementation details. These boundaries leave longer-term, less-informed, combined, and broader defense evaluations for future work.
- The adaptive attacks target only the agent’s first-step action, limiting their modeled long-term impact.The authors report the least improvement in the second step of the data-stealing attack.
- The attacks assume white-box access to the agent model, defense models, and detailed prompts.Black-box and grey-box attacks are identified as important for more comprehensive evaluation.
- The study does not evaluate combinations of defenses, although combining detection strategies could provide stronger protection.Adaptive attacks targeting defense combinations are proposed as future work.
- The research is not an exhaustive exploration of all defenses that could be adapted for IPI attacks.Examples not covered include LLM self-evaluation and alternative model finetuning methods.
- Adversarial strings for the Llama3-8B agent are more challenging to optimize than those for the Vicuna-7B agent.For Llama3-8B, prefix adversarial strings perform better than suffix strings.
- The adversarial finetuning setup uses unsuccessful attacks, LoRA with 4-bit quantization, and 15 training epochs.The reported data comprise 215 Vicuna-7B cases and 816 Llama3-8B cases.
B Experiments on the AgentDojo Benchmark
On the AgentDojo benchmark, the study evaluates adaptive IPI attacks in a dynamic simulated environment with long contexts and multiple agent-tool interactions. Adaptive attacks still substantially improve ASR, but are less effective than on InjecAgent, motivating further work on long-context attacks.
- AgentDojo setup: AgentDojo contains workspace, Slack, travel, and banking scenarios, with two attacker and two user cases selected from each scenario.The resulting test set contains 16 cases evaluated on the Llama3-8B agent.
- Results: Adaptive attacks still yield substantial ASR improvements on AgentDojo, but are less effective than on InjecAgent.On InjecAgent, all adaptive attack ASRs exceed 50%.
- Context effects: AgentDojo contexts average 3,823 tokens and reach 6,152 tokens, compared with 1,033 and 1,711 on InjecAgent.The longer contexts often include multiple rounds of agent-tool interaction and make adversarial-string optimization more challenging.