PaperlayerView exact revision on arXiv

Source-linked AI summary

Formalizing and Benchmarking Prompt Injection Attacks and Defenses

Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, Neil Zhenqiang Gong

arXiv:2310.12815v5cs.CRcs.AIcs.CLcs.LG

TL;DR

Prompt injection research has lacked a systematic understanding of attacks and defenses beyond case studies. This paper formalizes attacks and benchmarks them and their defenses, finding that attacks work across a wide range of LLMs and tasks while existing defenses are insufficient.

  • Problem

    Prior work was limited to case studies and lacked comprehensive evaluation, leaving prompt injection attacks and defenses without a systematic understanding.

    Source: §1 · p. 2

  • Method

    The paper formalizes prompt injection attacks in a framework that encompasses existing strategies and supports a new attack combining them.

    Source: §1 · p. 2

  • Results

    Prompt injection attacks were effective across a wide range of LLMs and tasks, and existing defenses were insufficient.

    Source: §9 · p. 13

  • Takeaways & Limitations

    The benchmark provides a basic baseline for evaluating future defenses against prompt injection attacks.

    Source: §1 · p. 2

    The literature lacks mechanisms to recover clean data after successful detection, so detection alone can still lead to denial of service.

    Source: §8 · p. 13

Abstract

from arXiv · show

A prompt injection attack aims to inject malicious instruction/data into the input of an LLM-Integrated Application such that it produces results as an attacker desires. Existing works are limited to case studies. As a result, the literature lacks a systematic understanding of prompt injection attacks and their defenses. We aim to bridge the gap in this work. In particular, we propose a framework to formalize prompt injection attacks. Existing attacks are special cases in our framework. Moreover, based on our framework, we design a new attack by combining existing ones. Using our framework, we conduct a systematic evaluation on 5 prompt injection attacks and 10 defenses with 10 LLMs and 7 tasks. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses. To facilitate research on this topic, we make our platform public at https://github.com/liu00222/Open-Prompt-Injection.

1 Introduction

Prompt injection exploits attacker-manipulated data to make LLM-integrated applications perform attacker-chosen tasks; this work formalizes the attacks and benchmarks attacks and defenses to address gaps in prior work.

Introduction

  • Prior research, largely case studies, lacked formal frameworks and comprehensive evaluations, leaving the threats and defense effectiveness unclear and hindering systematic understanding.The paper identifies these gaps as obstacles to designing new attacks and defenses.Source: §1 · pp. 1–2

An attack framework

  • The formal definition treats an attack as compromising target-task data so the application is misled from its intended task to an arbitrary attacker-chosen injected task.An attack injects the injected task’s instruction, or its instruction and data, into the target task’s data.Source: §1 · p. 2
  • The implementation framework crafts compromised data from target data, injected instructions, and injected data; it encompasses existing attacks [14] [35] and enabled a new attack combining existing strategies.The framework is intended to support systematic attack design and evaluation.Source: §1 · p. 2

Benchmarking prompt injection attacks

  • Across 5 attacks, 10 LLMs, and 7 tasks, the framework-inspired combined attack consistently worked across target and injected tasks and outperformed existing attacks.The evaluation provides a basic benchmark for assessing future defenses.Source: §1 · p. 2

Benchmarking defenses

  • Across 10 defenses, none was sufficient: prevention had limited effectiveness and/or large utility losses, while detection often missed compromised data or flagged clean data.One detection defense worked in some cases but missed a large fraction of compromised data in many others.Source: §1 · p. 2

Introduction

  • The authors open-source their platform to facilitate research on new prompt injection attacks and defenses.The introduction also notes that external data can be manipulated to redirect application outputs, making prompt injection a security concern listed as OWASP’s top threat.Source: §1 · pp. 1–2

2 LLM-Integrated Applications

An LLM maps a text prompt to a text response, and an LLM-integrated application uses it to perform a user’s task. Its prompt combines task instructions with data for analysis, and the application returns a post-processed response.

LLM-Integrated Applications

  • An LLM maps an input prompt to a text response; an LLM-integrated application connects the user, application, LLM, and external resource to accomplish a task.The paper denotes the LLM as f, the prompt as p, and the resulting response as f(p).Source: §2 · pp. 2–3
  • The application queries the LLM with a prompt formed by concatenating an instruction prompt and data, then returns the post-processed response to the user.Tasks include automated screening, spam detection, question answering, summarization, and translation.Source: §2 · p. 3
  • The instruction prompt tells the LLM how to perform the task, may include in-context examples, and can be provided by the user, the application, or both.The paper gives spam classification and French-to-English translation as example instructions.Source: §2 · p. 3
  • Data is the material analyzed for the task and often comes from an external resource, such as a social media post for spam detection or an Internet webpage for translation.The examples involve classifying a post as spam or non-spam and translating a webpage.Source: §2 · p. 3

3 Threat Model

The threat model assumes an attacker seeks an attacker-desired response by manipulating data used by an LLM-integrated application. The attacker knows the application is LLM-integrated but, in this work, is assumed not to know its internal details.

Attacker’s goal

  • The attacker may seek either a limited change to the correct response or an arbitrary attacker-desired response.Examples include changing a spam detector’s output from “spam” to “non-spam,” or making it summarize a document instead of classifying a post.Source: §3 · p. 3

Attacker’s background knowledge

  • The attacker knows the application is LLM-integrated but is assumed not to know its internal details, such as its prompt, use of in-context learning, or backend LLM.This assumption matches the threat model considered by all benchmarked prompt injection attacks.Source: §3 · p. 3

Attacker’s capabilities

  • The attacker can inject arbitrary instruction or data into application-used data, but cannot manipulate the instruction prompt, and the backend LLM is assumed to maintain integrity.The prompt is determined by the user and/or application.Source: §3 · p. 3

4 Our Attack Framework

The section defines prompt injection as modifying target data so an application performs an attacker-chosen injected task, then formalizes strategies for crafting compromised data. The framework encompasses existing attacks and inspires a combined attack.

4.1 Defining Prompt Injection Attacks

  • A prompt injection attack modifies the target task’s data so the application performs an injected task instead; the attacker can choose arbitrary injected instructions and data.A task consists of an instruction and data, and the application ordinarily combines them to query its backend LLM.Source: §4.1 · pp. 3–4

4.2 Formalizing an Attack Framework

  • The framework crafts compromised data from target data, injected instruction, and injected data, treating existing prompt injection attacks [14] [35] as special cases and enabling new attacks.The application queries the backend LLM with the target instruction concatenated with the compromised data.Source: §4.2 · p. 4
  • Existing strategies concatenate target data with injected instructions and data; escape-character and task-ignoring [35] attacks alter context, while fake-completion appends a response for the target task.When the attacker knows or can infer the target task, the fake response can be constructed specifically for it.Source: §4.2 · pp. 4–5
  • The Combined Attack combines escape characters, task-ignoring text, and fake completion, using special characters to separate the fake response from the task-ignoring text.The experiments use “Answer: task complete” as a generic fake response.Source: §4.2 · p. 5

5 Defenses

The paper organizes defenses into prevention, which aims to preserve task performance through prompt redesign or data preprocessing, and detection, which identifies compromised data.

5.1 Prevention-based Defenses

  • Paraphrasing [25] rewrites data to disrupt injected structures, while retokenization [25] uses BPE-dropout to break rare words into smaller tokens.Both methods extend defenses originally designed for jailbreaking prompts to prompt injection.Source: §5.1 · p. 6
  • Delimiters enclose data; sandwich prevention appends a task reminder, while instructional prevention tells the model to ignore instructions in data.The experiments use three single quotes as the default delimiters.Source: §5.1 · p. 6

5.2 Detection-based Defenses

  • Detection methods either analyze data directly or use LLM responses; naive LLM-based detection asks the backend model whether data is compromised.It interprets “no” as compromised and “yes” as clean.Source: §5.2 · p. 6
  • Perplexity-based detection [11] [25] flags data above a threshold, while windowed detection flags data if any contiguous window exceeds its threshold.Both methods use perplexity as a proxy for text quality; the windowed variant checks each segment separately.Source: §5.2 · p. 6
  • A clean-data calibration sets each perplexity threshold to keep the false-positive rate at or below a chosen value, such as 1%.The authors collect clean records, calculate their model-dependent perplexities, and choose a threshold using those values.Source: §5.2 · p. 6
  • Response-based detection flags responses invalid for the target task, but fails when injected and target tasks are the same type.For example, a response outside “spam” or “non-spam” is invalid for a spam-detection task.Source: §5.2 · pp. 6–7
  • Known-answer detection prepends an instruction with a known answer and flags data as compromised if the LLM fails to produce that answer.The experiments use a seven-character random secret key as the known answer.Source: §5.2 · p. 7

6 Evaluation

Combined Attack is broadly effective and outperforms the other attacks on average, while the evaluated prevention- and detection-based defenses remain insufficient. The evaluation spans seven tasks, ten LLMs, and both attack success and defense utility or detection errors.

6.1 Experimental Setup

  • The evaluation covers seven NLP tasks and 49 target/injected task pairings, using 100 non-overlapping examples per task role and measuring task performance, attack success, matching, and detection errors.It uses accuracy for classification, Rouge-1 for summarization, and GLEU for grammar correction; detection metrics are FPR and FNR.Source: §6.1 · pp. 7–8

6.2 Benchmarking Attacks

  • Fake Completion is the second most successful attack, Naive Attack is the least successful, and Escape Characters and Context Ignoring have no clear winner across GPT-4 and PaLM 2.Combined Attack’s strategy of explicitly indicating that the target task is complete is more successful than escaping characters or ignoring context.Source: §6.2 · pp. 8–9
  • 0.62 average ASV and 0.78 average MR across 10 LLMs and 49 task pairs show Combined Attack’s broad effectiveness; it also has higher average ASV than other attacks, with few task-pair exceptions.Fake Completion slightly exceeds Combined Attack for GPT-4 on grammar-correction targets with duplicate-sentence-detection injections.Source: §6.2 · pp. 8–9, Table 4 · p. 9, Figure 3 · p. 9
  • Combined Attack effectiveness is positively correlated with model size (Pearson correlations 0.63 for ASV and 0.64 for MR), varies by injected task, and changes little with the number of target-task demonstrations.It has similar ASV and MR across target tasks; sentiment analysis has the highest average ASV and MR as the injected task, while summarization has the lowest.Source: Figure 3 · p. 9, §6.2 · pp. 9–10, Table 6 · p. 10

6.3 Benchmarking Defenses

  • Existing prevention-based defenses are insufficient: they leave attack ASV and MR high and can reduce clean-task utility, with paraphrasing lowering average PNA-T by 0.14.Retokenization misses injected content by dropping tokens randomly; delimiters alter clean-data structure, while added instructions in sandwich and instructional prevention can harm summarization, with sandwich prevention lowering PNA-T from 0.38 to 0.24.Source: §6.3 · p. 10, Table 7 · p. 11
  • Most detection defenses have high FNRs, while naive LLM-based detection has very small FNRs but very large FPRs; response-based detection works mainly for classification targets with different injected tasks.Response validation is difficult for non-classification tasks or same-task classification attacks, making that defense ineffective in those settings.Source: Table 8 · p. 12, §6.3 · p. 12
  • Known-answer detection is the strongest evaluated detector for Combined Attack at small FPRs, but it misses many attacks in other settings, especially grammar-correction targets.It detects attacks using escape characters and those targeting duplicate-sentence detection more effectively; missed samples can still achieve high ASV and MR.Source: Table 8 · p. 12, Table 9 · p. 12, §6.3 · p. 12

7 Related Work

Related work distinguishes this benchmark’s prompt-injection scenario from other attack settings, concurrent defenses, jailbreaking, and other attacks on LLMs.

Prompt injection attacks from malicious users

  • The benchmark targets attacks on application users without requiring black-box access, unlike attacks [27] [35] that target the application and may require repeated queries [27].For example, an applicant may lack access to the automated screening application when crafting a compromised resume.Source: §7 · p. 13

Other defenses against prompt injection attacks

  • Concurrent defenses [17] [37] [54] were not benchmarked; Jatmo [37] fine-tunes a non-instruction-tuned LLM for a specific task to resist injected instructions.Its premise is that a model never trained to follow instructions will not follow an injected instruction.Source: §7 · p. 13

Jailbreaking attacks

  • Prompt injection redirects an LLM from its target task to an attacker-injected task, whereas jailbreaking seeks completion of a refused unsafe target task.Prompt-injection tasks may be safe or unsafe; jailbreaking focuses on unsafe target tasks.Source: §7 · p. 13

Other attacks to LLM

  • Privacy attacks [16] [29] infer memorized information, poisoning attacks [13] [41] [46] [53] alter training data or model parameters, and adversarial prompts preserve the task but induce incorrect responses.Source: §7 · p. 13

8 Discussion and Limitations

The discussion identifies limitations in current prompt-injection attacks and defenses and outlines future work on optimization-based attacks, LLM fine-tuning, recovery after detection, and known-answer detection.

Optimization-based attacks

  • Existing prompt-injection attacks rely on heuristics; future work could optimize special characters, task-ignoring text, or fake responses to craft compromised data.Source: §8 · p. 13

Fine-tuning an LLM as a defense

  • Future work could fine-tune LLMs to complete target tasks despite compromised data or to perform tasks without following injected instructions.Fine-tuned models may remain vulnerable to attacks not considered during fine-tuning; a related task-specific strategy was explored in [37].Source: §8 · p. 13

Recovering from attacks

  • Detection without recovering clean data leaves the application unable to complete its target task and can still cause denial-of-service.Existing defenses focus on prevention and detection, but the literature lacks mechanisms to recover clean data after successful detection.Source: §8 · p. 13

Known-answer detection

  • Known-answer detection was evaluated with one detection prompt; future work could test other prompts and adaptive attacks against effective prompts.The proposed prompt should have a known answer that injected instructions can easily overwrite across different attacks.Source: §8 · p. 13

9 Conclusion and Future Work

The paper introduces a framework to formalize and quantitatively benchmark prompt injection attacks and defenses, finding attacks effective across a wide range of LLMs and tasks while existing defenses remain insufficient.

9 Conclusion and Future Work

  • The proposed framework formalizes prompt injection attacks and enables comprehensive, quantitative benchmarking of attacks and defenses.The authors describe it as the first framework of its kind.Source: §9 · p. 13
  • Prompt injection attacks are effective across a wide range of LLMs and tasks, while existing defenses are insufficient.Source: §9 · p. 13
  • Future work includes stronger optimization-based attacks, prevention- and detection-based defenses, and mechanisms to recover after detected attacks.Source: §9 · p. 13

A Details on Selecting Target/Injected Data

The appendix specifies how target/injected records, in-context examples, and clean records for PPL detectors were sampled across seven tasks. It also describes label handling and non-overlap constraints.

A Details on Selecting Target/Injected Data

  • Target/injected records came from designated validation, test, or training splits across seven tasks, with ground-truth labels mapped to task-specific meanings.SST2 and SMS Spam labels 0/1 mean negative/positive and non-spam/spam; HSOL label 2 means not hateful and other labels hateful; MRPC 0/1 mean not equivalent/equivalent; RTE 0/1 mean entailment/not entailment. Gigaword and Jfleg retain their original ground-truth labels.Source: A Details on Selecting Target/Injected… · p. 16
  • For matching classification tasks, target and injected records had different ground-truth labels; SMS Spam and HSOL pairs specifically used spam/hateful targets and not-spam/not-hateful injections.This choice was made because matching labels make attack success difficult to determine.Source: A Details on Selecting Target/Injected… · p. 16
  • In-context examples were sampled from task-specific splits and kept separate from target/injected records; SMS Spam and HSOL examples and records came from training splits because holdout data were absent or unlabeled.The examples used SST2, SMS Spam, HSOL, and MRPC training sets; Gigaword training, Jfleg testing, and RTE validation sets.Source: A Details on Selecting Target/Injected… · p. 16
  • Clean records for selecting PPL-detector thresholds and window sizes were sampled from corresponding task splits and had no overlap with target/injected records.The sampled splits were SST2, Gigaword, and Jfleg validation; SMS Spam and HSOL training; MRPC testing; and RTE training.Source: A Details on Selecting Target/Injected… · p. 16

B Impact of the Length of Injected Task

For Combined Attack, performance improves as injected data gets longer until it stabilizes, while the effect of injected-instruction length varies by task. The attack performs well when injected data exceeds 30 tokens or injected instructions exceed 20 tokens.

Impact of the number of tokens in injected data

  • Combined Attack’s ASV increases with injected-data length and then stabilizes; it is less effective below 30 tokens and effective once the length is reasonably large.The study truncates injected data to the first l tokens when it exceeds the threshold l.Source: B Impact of the Length of Injected Task · p. 25, Figure 7 · p. 26

Impact of the number of tokens in injected instruction

  • Injected-instruction length has negligible impact on sentiment analysis but can affect grammar correction; Combined Attack performs well when instructions exceed 20 tokens.The paper attributes the task-specific difference to grammar correction being more challenging and requiring a longer instruction.Source: B Impact of the Length of Injected Task · p. 25, Figure 8 · p. 26

Related papers

Loading 2310.12815v5…