Source-linked AI summary
Ignore Previous Prompt: Attack Techniques For Language Models
Fábio Perez, Ian Ribeiro
TL;DR
The paper addresses the limited study of malicious user interactions that can misalign LLM applications. It introduces PROMPTINJECT to evaluate goal hijacking and prompt leaking, finding substantial attack success under tested conditions and persistent prevention challenges.
Problem
Studies of prompt injection vulnerabilities arising from malicious user interaction with LLM applications remain scarce.
Method
PROMPTINJECT modularly composes base and attack prompts while evaluating attack-specific success criteria across varied prompt and model settings.
Results
58.6% ± 1.6 goal-hijacking success and 23.6% ± 2.7 prompt-leaking success were achieved, with prompt leaking harder than goal hijacking.
Takeaways & Limitations
The findings highlight security risks in LLM applications and the difficulty of completely preventing prompt injection in open-ended language models.
Takeaways & Limitations
Future work is needed to test more models, additional factors and attacks, and further prevention methods.
Abstract
from arXiv · showhide
Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.
1 Introduction
GPT-3 enables simple user-facing applications through prompt substitution, but malicious users can inject instructions that misalign application goals. The paper studies goal hijacking and prompt leaking as two such attacks.
- 1 Introduction: GPT-3 and similar LLMs support complex applications by substituting user input into designed prompts.This approach can simplify tasks such as grammar correction compared with rule-based strategies.
- 1 Introduction: Prompt injection is malicious text inserted through an application interface to misalign an LLM.The open-ended, unstructured nature of GPT-3 prompts makes protection challenging.
- 1 Introduction: Studies of prompt injection remained scarce despite public examples of users misaligning GPT-3-based applications.The paper analyzes the feasibility and effectiveness of two attacks: goal hijacking and prompt leaking.
- 1 Introduction: Goal hijacking redirects a prompt toward printing a target phrase, while prompt leaking redirects it toward printing part or all of the original prompt.The latter can expose the application prompt, which may contain important application-specific information.
- 1 Introduction: The paper studies prompt injection attacks, investigates goal hijacking and prompt leaking, and provides an AI x-risk analysis.These are stated as the paper’s main contributions.
2 Related work
Related work documents harmful LLM behavior and mitigation efforts, but prior prompt attacks mainly targeted classification outcomes. This paper extends that line toward malicious target text and prompt theft across original tasks.
- 2 Related work: Prior studies report that LLMs can reproduce social biases, leak private training data, and generate vitriol at scale.These findings motivate broader examination of harmful model behavior.
- 2 Related work: Mitigation research has examined toxic-behavior controls and human-feedback fine-tuning, but no guaranteed prevention method was identified.Human feedback improved instruction following, truthfulness, and harmful-behavior reduction in GPT-3.
- 2 Related work: Earlier adversarial-prompt work showed that simple prompt injection can change GPT-3 and other LLMs’ classification results.The present work instead targets malicious output or recovery of the original prompt regardless of the original task.
3 The PROMPTINJECT framework
PROMPTINJECT is a modular framework for quantitatively testing LLM robustness to adversarial prompts. It varies base prompts, attack strategies, prompt factors, and model-sampling settings.
- 3 The PROMPTINJECT framework: PROMPTINJECT assembles prompts modularly to provide quantitative analysis of LLM robustness to adversarial prompt attacks.Its inner components and behavior relationships are presented in Figure 2.
- 3 The PROMPTINJECT framework: Base prompts combine an initial application instruction with variations in examples, user-or-model labels, and private values.Private values can contain sensitive instructions, prohibited subjects, themes, or contextual enhancers.
- 3 The PROMPTINJECT framework: Attack prompts implement goal hijacking with rogue strings and prompt leaking with embedded private values that should not be externally revealed.The framework also considers language-model sensitivity to escape and delimiter choices.
- 3 The PROMPTINJECT framework: Experiments vary temperature, top-p sampling, frequency and presence penalties, and the prompted model because outputs vary across settings.These factors are incorporated into the framework’s scenarios.
4 Experiments
The experiments construct varied attack prompts and test them across 35 public application prompts. Success is measured using attack-specific exact or containment criteria and repeated runs.
- 4 Experiments: The study evaluates multiple attack prompts on 35 base prompts collected from the OpenAI Examples page.The examples were chosen as likely starting points for developers’ applications.
- 4 Experiments: All experiments use text-davinci-002 except for results reported in Table B10.The paper describes it as the most powerful publicly available OpenAI model.
- 4 Experiments: Success rates are means and standard deviations over 35 prompts, with each experiment repeated 4 times to address GPT-3 stochasticity.A reported rate such as 50 ± 1.0% summarizes those repeated evaluations.
- 4 Experiments: Goal hijacking succeeds when output exactly matches the target string, while prompt leaking succeeds when output contains the original instruction.Fuzzy matching is implemented but not used in the reported results.
5 Results and discussion
The experiments show that prompt-injection success depends on attack wording, delimiters, model settings, and prompt structure, while prompt leaking is harder than goal hijacking. Available defenses reduce attacks but do not guarantee prevention, underscoring the difficulty of securing open-ended LLM applications.
- Temperature influences attacks, whereas top-p and frequency or presence penalties do not; more harmful rogue strings, stop sequences, and trailing prompt text inhibit attacks.
- 58.6% ± 1.6 success was achieved for goal hijacking versus 23.6% ± 2.7 for prompt leaking.
- Small wording changes and delimiters improve attack effectiveness, although delimiter type, length, and repetitions have unclear effects.
- Effective mitigations include stop sequences, text after user input, output-token limits, and post-processing, but completely preventing attacks may be virtually impossible.
- text-davinci-002 is the most vulnerable publicly available OpenAI API model despite being the most capable.
- Prompt leaking is harder than goal hijacking, but proxy tasks and minor prompt changes can substantially improve leaking efficacy.
- A solution remains open, and the findings highlight the difficulty of defending against prompt injection and the need for further research.
6 Future works
The paper identifies several directions for extending prompt-injection research and releases PROMPTINJECT to facilitate community work on safer language-model applications.
- Future work includes automatically searching for effective malicious instructions and testing injection techniques with additional models.
- Further directions include exploring new factors and attacks, improving prevention methods, and examining GPT-3 edit and insert models.
- The released PROMPTINJECT code is intended to facilitate future research and support safer, more robust language-model applications.
A X-Risk Analysis
The appendix applies an existing x-risk analysis template, while cautioning that individual responses do not decisively establish relevance to existential-risk reduction.
- The paper uses the x-risk analysis template introduced by Hendrycks and Mazeika.
- Individual question responses do not decisively imply relevance or irrelevance to existential-risk reduction.
A.1 Long-Term Impact on Advanced AI Systems
The work frames prompt-injection robustness as increasingly important as LLMs spread into user-facing applications. It also identifies both safety benefits and risks from studying these attacks.
- A.1 Long-Term Impact on Advanced AI Systems: Wider deployment of LLM applications increases exposure to misalignment from adversarial user inputs.The paper connects this exposure to stochastic and unpredictable model behavior and insufficient deployment guidelines.
- A.1 Long-Term Impact on Advanced AI Systems: Robustness evaluation heuristics are needed because successful misalignment attacks could range from unacceptable language to private-prompt disclosure or life-endangering actions.The authors call for expanded robustness evaluation and testing methods as LLM capabilities enter novel settings.
- A.2 Safety-Capabilities Balance: The work may improve safety by exposing idiosyncratic behaviors in state-of-the-art LLM applications and providing tools to measure and improve robustness metrics.This safety benefit is presented alongside the possibility that the framework could facilitate malicious attacks.
- A.2 Safety-Capabilities Balance: The framework may be used by adversaries to develop novel misalignment strategies without increasing AI capabilities.This creates a security risk through malicious use rather than through capability advancement.
- A.3 Elaborations and Other Considerations: Lower-capability models are less susceptible to the presented techniques because they are less reliable at following instructions generally.The authors relate susceptibility to the attention that more powerful models give to prompts and adversarial user inputs.
B Experimental Results
The experiments section states that unspecified attack prompts use a default prompt. Table B1 identifies that default attack prompt.
- B Experimental Results: Experiments without a specified attack prompt use the default attack prompt from Table B1.This establishes the fallback prompt used in those experiments.
- B Experimental Results: Experiments with explicitly specified attack prompts should be distinguished from experiments using the Table B1 default.The supplied passages support this comparison through the stated fallback rule.
- B Experimental Results: Table B1 is the reference location for the default attack prompt.The table is labeled explicitly as the default attack prompt.
B.1 Results for goal hijacking
This section evaluates goal hijacking across attack-prompt components, model settings, stop sequences, and related prompt configurations. Success rates are reported as means and standard deviations across experiments.
- Attack instruction: The experiments examine how attack instructions affect goal hijacking success rates.The corresponding table reports the mean and standard deviation of successful attacks.
- Delimiter configuration: Delimiter type, length, and repetition are evaluated as separate factors in goal hijacking attacks.Each factor has a dedicated table reporting mean and standard deviation success rates.
- Model and prompt settings: The analysis also considers the rogue string, GPT-3 parameters, and stop sequences as attack-related factors.Stop-sequence experiments use the subset of ten OpenAI Examples prompts that include stop sequences by default.