Source-linked AI summary
Formalizing and Benchmarking Prompt Injection Attacks and Defenses
Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, Neil Zhenqiang Gong
TL;DR
Prompt injection attacks exploit LLM-Integrated Applications by manipulating task data, while prior work has lacked formalization and comprehensive evaluation. This paper formalizes the attacks, designs a combined attack, and benchmarks attacks and defenses across models and tasks, finding broad attack effectiveness and insufficient existing defenses.
Problem
Prior prompt-injection research is mostly case-based and lacks formal frameworks and comprehensive evaluations, leaving attacks and defenses without systematic understanding.
Method
The paper formalizes prompt injection, treats existing attacks as framework special cases, designs a combined attack, and systematically evaluates attacks and defenses.
Results
Prompt injection attacks are effective across a wide range of LLMs and tasks, while existing defenses are insufficient.
Takeaways & Limitations
The framework provides a common benchmark for quantitatively evaluating future prompt-injection attacks and defenses.
Takeaways & Limitations
Existing defenses focus on prevention and detection, while mechanisms to recover clean data after successful detection are lacking.
Abstract
from arXiv · showhide
A prompt injection attack aims to inject malicious instruction/data into the input of an LLM-Integrated Application such that it produces results as an attacker desires. Existing works are limited to case studies. As a result, the literature lacks a systematic understanding of prompt injection attacks and their defenses. We aim to bridge the gap in this work. In particular, we propose a framework to formalize prompt injection attacks. Existing attacks are special cases in our framework. Moreover, based on our framework, we design a new attack by combining existing ones. Using our framework, we conduct a systematic evaluation on 5 prompt injection attacks and 10 defenses with 10 LLMs and 7 tasks. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses. To facilitate research on this topic, we make our platform public at https://github.com/liu00222/Open-Prompt-Injection.
1 Introduction
LLM-Integrated Applications can be manipulated through externally sourced data, but existing prompt-injection research has lacked formalization and comprehensive evaluation. This work introduces an attack framework, a combined attack, and systematic benchmarks for attacks and defenses.
- Motivation: Externally sourced data can be manipulated to make an LLM-Integrated Application produce an attacker-desired result.The paper illustrates this with a resume containing an instruction that causes automated screening to output “yes.”
- Motivation: Existing studies are mostly case studies and lack frameworks plus comprehensive evaluations of prompt-injection attacks and defenses.These limitations leave threats, severity, and defense effectiveness unclear.
- Contributions: The paper formalizes prompt injection as compromising target-task data so an application performs an arbitrary attacker-chosen injected task.The framework distinguishes the target task from the injected task and clarifies that injected content is instruction or instruction plus data, not bare data.
- Contributions: The framework treats existing attacks as special cases and supports designing new attacks by combining strategies for crafting compromised data.The construction uses target data, injected instructions, and injected data.
- Evaluation: 5 prompt-injection attacks are quantitatively evaluated across 10 LLMs and 7 tasks, with the combined attack consistently effective and outperforming existing attacks.The benchmark is intended as a baseline for evaluating future defenses.
- Evaluation: 10 candidate defenses are systematically evaluated, and the research platform is released publicly to support future attack and defense research.The paper presents the platform as a resource for evaluating future defenses and attacks.
2 LLM-Integrated Applications
An LLM-Integrated Application combines an instruction prompt with external data, queries a backend LLM, and returns its response to the user. These applications support tasks including screening, spam detection, question answering, summarization, and translation.
- Components: An LLM-Integrated Application includes a user, application, backend LLM, and external resource.The application queries the LLM and returns its post-processed response to the user.
- Inputs: The application prompt is the concatenation of an instruction prompt and data.The instruction directs the LLM to perform the task, while data is analyzed by the LLM.
- Inputs: Instruction prompts can include task directions and demonstration examples, such as spam classification or French-to-English translation.Demonstrations may be added to improve performance.
- Inputs: Data commonly comes from external resources, including social-media posts and Internet webpages.The application processes this data for tasks such as spam detection and translation.
3 Threat Model
The threat model defines attackers by their goals, knowledge, and capabilities. Attackers manipulate application data with injected instructions or data, while internal application details may remain unknown.
- Attacker’s goal: Attackers aim to make an LLM-Integrated Application produce an attacker-desired response.The desired response may be a limited modification, such as misclassifying spam, or an arbitrary result.
- Background knowledge: Attackers know the application is LLM-integrated but may lack internal details such as its instruction prompt, in-context learning, and backend LLM.The benchmark assumes attackers do not know these internal details.
- Capabilities: Attackers can manipulate application data and inject arbitrary instructions or data into it.Examples include modifying resumes, social-media posts, and hosted webpages.
- Impact criteria: Attacks are more impactful when they assume less attacker knowledge and capability, while defenses are more impactful against stronger attackers and weaker goals.The paper uses these criteria to characterize attack and defense impact.
4 Our Attack Framework
The framework formalizes prompt injection as modifying target-task data so an application performs an attacker-chosen injected task, then represents attacks as strategies for constructing compromised data. It also enables systematic quantification and motivates a combined attack that unifies existing strategies.
- 4.1 Defining Prompt Injection Attacks: A prompt injection attack modifies target data so the application performs an attacker-chosen injected task instead of the target task.
- 4.2 Formalizing an Attack Framework: The framework represents an attack as crafting compromised data from target data, injected instruction, and injected data.
- 4.2 Formalizing an Attack Framework: The application normally queries the backend LLM with target instruction concatenated with target data, but under attack it uses compromised data and produces an injected-task response.
- 4.2 Formalizing an Attack Framework: Existing attacks become special cases of the framework, which also supports designing new attacks.
- Existing Attacks: Naive Attack concatenates target data, injected instruction, and injected data, while Escape Characters appends a special character before that concatenation.
- Existing Attacks: Context Ignoring appends task-ignoring text, Fake Completion appends a fake target-task response, and the Combined Attack combines these three strategies.
5 Defenses
The paper organizes defenses into prevention and detection, including data transformations, instruction redesign, perplexity thresholds, LLM-based checks, and response validation. Detection methods inspect data or model responses, with response-based detection limited when target and injected tasks share the same type.
- Defense Categories: Prevention defenses redesign the instruction prompt or preprocess data so the application still performs the target task with compromised data.
- Prevention-based Defenses: Paraphrasing and retokenization disrupt the ordering or token structure of injected content, while delimiters isolate data from instructions.
- Prevention-based Defenses: Sandwich prevention appends a reminder of the target instruction to the data, whereas instructional prevention tells the LLM to follow the target instruction regardless of data instructions.
- Detection Categories: Detection defenses either analyze data directly or use an LLM response to determine whether data is compromised.
- Detection-based Defenses: PPL detection flags data whose perplexity exceeds a threshold, and Windowed PPL detection flags data when any contiguous window exceeds that threshold.
- Detection-based Defenses: The proposed perplexity threshold is calibrated on clean records so the false positive rate stays no larger than a specified value, such as 1%.
- Detection-based Defenses: Response-based detection fails when injected and target tasks are the same type, while known-answer detection checks whether a secret-key instruction is followed.
6 Evaluation
The evaluation benchmarks attacks and defenses across multiple LLMs and NLP tasks, finding that Combined Attack is broadly effective while existing defenses remain insufficient or costly. Results also show that attack effectiveness varies with model size and task configuration.
- Benchmarking attacks: Combined Attack outperforms other attacks across average ASV and nearly every target/injected task combination.Fake Completion is generally second most successful, while Naive Attack is least successful; Escape Characters and Context Ignoring have no consistent winner across LLMs.
- Benchmarking attacks: ASV and MR for Combined Attack average 0.62 and 0.78 across ten LLMs and 7 × 7 target/injected task combinations.The attack remains effective across target and injected tasks; sentiment analysis has the highest average values and summarization the lowest, which the authors associate with task difficulty.
- Benchmarking attacks: Combined Attack effectiveness generally increases with model size, with Pearson correlations of 0.63 for ASV and 0.64 for MR.GPT-4 has higher average ASV and MR than the other tested LLMs, and Vicuna-33b-v1.3 exceeds Vicuna-13b-v1.3.
- Benchmarking defenses: Prevention-based defenses reduce attack metrics in some cases but either remain limited or sacrifice clean-task utility.Paraphrasing lowers average PNA-T by 0.14, while retokenization fails to reliably remove injected instructions and also harms clean data utility.
- Benchmarking defenses: Existing detection-based defenses are insufficient: perplexity methods fail, response-based detection is task-dependent, and naive LLM detection has very large FPRs.Known-answer detection performs best among existing detection methods for Combined Attack but misses many samples in several settings, especially grammar correction.
7 Related Work
The benchmarked attacks target scenarios where the victim is an LLM-Integrated Application user and require no black-box access during compromised-data construction.
- The benchmarked attacks assume the victim is an LLM-Integrated Application user, rather than the application itself.
8 Discussion and Limitations
The discussion identifies limitations and open directions involving heuristic attacks, standard-model experiments, missing recovery mechanisms, and narrow known-answer detection evaluation.
- Optimization-based attacks: Existing attacks rely on heuristics such as special characters, task-ignoring text, and fake responses.The authors suggest optimization-based strategies as future work.
- Fine-tuning an LLM as a defense: The experiments use standard LLMs, leaving fine-tuning-based defenses for future investigation.Fine-tuned models may remain vulnerable to attacks absent from the fine-tuning data.
- Recovering from attacks: Existing defenses emphasize prevention and detection but lack mechanisms to recover clean data after successful detection.Without recovery, detecting an attack can still prevent the application from completing its target task.
- Known-answer detection: Known-answer detection is evaluated with one detection prompt, leaving alternative prompts and adaptive attacks for future work.
9 Conclusion and Future Work
The paper concludes that prompt injection attacks create serious deployment concerns and presents a framework for quantitative benchmarking across attacks, defenses, LLMs, and tasks.
- Prompt injection attacks are effective across a wide range of LLMs and tasks, while existing defenses are insufficient.
- Future work includes stronger optimization-based attacks, new prevention and detection defenses, and recovery mechanisms after detection.
- Figure 5 presents examples of delimiters, instructional prevention, and sandwich prevention.
A Details on Selecting Target/Injected Data
The appendix documents dataset and prompt selection for target and injected tasks, along with tables reporting Combined Attack results and detector false-negative rates across models and settings.
- Details on Selecting Target/Injected Data: Target and injected data are sampled from SST2, SMS Spam, HSOL, Gigaword, Jfleg, MRPC, and RTE datasets.
- Details on Selecting Target/Injected Data: When target and injected tasks share a classification dataset, their ground-truth labels are intentionally made different to clarify attack success.
- Details on Selecting Target/Injected Data: In-context examples are sampled from task-specific datasets, including training or validation splits when testing or validation labels are unavailable.
- Details on Selecting Target/Injected Data: Clean records used to select PPL-detector thresholds and window sizes do not overlap with target or injected data.
- Experimental Results: Figure 6 reports attack success values for injected tasks across the seven target tasks, with each panel corresponding to an injected task.
- Experimental Results: Tables 28–32 report false-negative rates for PPL, windowed PPL, naive LLM-based, response-based, and known-answer detectors.
B Impact of the Length of Injected Task
The study examines how injected-data and injected-instruction length affect Combined Attack across target and injected tasks. Combined Attack becomes effective once injected-task content is reasonably long, while the instruction-length study uses controlled variants documented in Table 33.
- Injected data length: Injected data is truncated to at most l tokens, retaining only the first l tokens when longer.This procedure compares Combined Attack performance across different injected-data length thresholds.
- Injected data length: ASV first increases with injected-data length and then remains stable as l increases further.Figure 7 reports ASV under different injected-data length thresholds for GPT-4 across target and injected tasks.
- Injected data length: Combined Attack is less effective with short injected data because the LLM lacks enough information for the injected task.The experiments find effectiveness once injected-task token length is reasonably large, such as above 30 tokens.
- Injected instruction length: Injected-instruction length is also varied to study its impact on Combined Attack.Figure 8 organizes the comparison by injected task and target task, while Table 33 lists the instruction variants used.