Source-linked AI summary
Automatic and Universal Prompt Injection Attacks against Large Language Models
Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, Chaowei Xiao
TL;DR
Prompt injection research lacks unified attack objectives and depends heavily on handcrafted prompts, limiting comprehensive robustness assessment. The paper introduces a unified objective framework and an automated momentum-enhanced gradient search method. With five training samples, the attack achieves high success across objectives and datasets, including under defenses, while remaining weaker against PPL detection.
Problem
Prompt injection research lacks unified attack objectives and relies on handcrafted prompts, making comprehensive robustness assessment difficult.
Method
The paper defines static, semi-dynamic, and dynamic objectives and uses a momentum-enhanced gradient optimization algorithm to generate prompt injection data automatically.
Results
With five training samples, the attack achieves an average 50% attack success rate across objectives and datasets, while baselines lose effectiveness and performance remains high under defenses.
Takeaways & Limitations
Gradient-based testing reveals universal prompt injection threats and helps avoid overestimating robustness, particularly for defense mechanisms.
Takeaways & Limitations
The method is weak against PPL detection defense, although that defense requires one or more additional LLM inference processes and is expensive.
Abstract
from arXiv · showhide
Large Language Models (LLMs) excel in processing and generating human language, powered by their ability to interpret and follow instructions. However, their capabilities can be exploited through prompt injection attacks. These attacks manipulate LLM-integrated applications into producing responses aligned with the attacker's injected content, deviating from the user's actual requests. The substantial risks posed by these attacks underscore the need for a thorough understanding of the threats. Yet, research in this area faces challenges due to the lack of a unified goal for such attacks and their reliance on manually crafted prompts, complicating comprehensive assessments of prompt injection robustness. We introduce a unified framework for understanding the objectives of prompt injection attacks and present an automated gradient-based method for generating highly effective and universal prompt injection data, even in the face of defensive measures. With only five training samples (0.3% relative to the test data), our attack can achieve superior performance compared with baselines. Our findings emphasize the importance of gradient-based testing, which can avoid overestimation of robustness, especially for defense mechanisms.
1. Introduction
Prompt injection attacks exploit LLM instruction-following by inserting hidden instructions into external data, while research lacks unified objectives and relies heavily on handcrafted prompts. The paper addresses these gaps with a unified objective framework and an automated, universal gradient-based attack that remains effective across datasets and defenses.
- Threat and motivation: Prompt injection attacks manipulate LLM-integrated applications by adding hidden instructions to external data, influencing responses during retrieval and processing.LLMs may not distinguish user commands from modified external inputs, allowing injected content to alter the original instructions.
- Threat and motivation: Research lacks a unified objective for prompt injection attacks, which span goals such as goal hijacking, prompt leaking, and persuading users to disclose information.Different objectives have historically required separate evaluation prototypes, complicating unified assessment.
- Threat and motivation: Handcrafted prompt attacks limit scope and scalability, show unstable universality across instructions and data, and hinder adaptive testing of defenses.These limitations can lead to overestimating the robustness of defense mechanisms.
- Proposed approach: The paper formulates static, semi-dynamic, and dynamic attack objectives to cover existing prompt injection research.This framework is paired with a momentum-enhanced gradient search algorithm for automatically generating prompt injection data.
- Results and implication: The automatic attack demonstrates universality across diverse user interactions and datasets while retaining effectiveness against multiple defense mechanisms.The authors argue that gradient-based testing is needed to evaluate prompt injection robustness, especially for defenses.
2. Methodology
The method formalizes prompt injection through three objectives, converts each objective into an optimization target, and uses gradient-based search to automatically generate injected data that generalizes across varying instructions and external data.
- Threat model and formulation: The threat model injects data S into external data D so a victim LLM processes I ⊕ D ⊕ S and produces an adversary-desired response.The formulation targets reactive attacks in which the attacker does not know the user’s instructions in advance.
- Prompt injection objectives: The framework unifies prompt injection goals as static, semi-dynamic, and dynamic objectives.Static responses ignore user input; semi-dynamic responses combine consistent malicious content with user-relevant responses; dynamic responses remain user-relevant while incorporating adversary-specified information.
- Prompt injection objectives: Static attacks target consistent responses regardless of user instructions or external data, such as repeated fake warnings or fixed outputs.This objective aligns with goal-hijacking attacks.
- Prompt injection objectives: Semi-dynamic attacks prepend consistent content before a response relevant to the user, whereas dynamic attacks preserve relevance while embedding malicious content.Examples include misinformation, harmful actions, prompt leaking, and persuading users to disclose private information.
- Optimization targets and loss: The method turns each conceptual objective into a specific target sentence and uses its negative log probability to define an optimization loss.The target formats include adversary-static text, prompt-leaking text, and query-related malicious statements followed by an elaboration cue.
3. Evaluations
The evaluation tests the attack across seven language tasks, multiple objectives, baselines, and defenses. Results show strong cross-dataset universality, faster momentum-enhanced optimization, and substantial effectiveness against defenses.
- 3.1. Experimental Setups: Evaluations use seven natural-language tasks, 15 adversarial goals, and 200 samples per dataset, totaling 1,400 test samples.The victim model is Llama2-7b-chat, with attack success measured against predefined goals.
- 3.1. Experimental Setups: The study compares the proposed attack with combined, repeated, and naïve prompt-injection baselines.These baselines represent handcrafted, repeated-character, and direct-request attack strategies.
- 3.2. Main Results: Previous attacks assessed in benign settings lose effectiveness on malicious goals, whereas the proposed method remains effective and universal across three objectives.The evaluation standardizes the target around distorting user requests to produce malicious outcomes.
- 3.2. Main Results: Above 80% ASR on the static objective and 50% average ASR across three objectives are achieved using only five training samples.The five samples represent 0.3% of the testing data, and performance is maintained on unseen instructions.
- 3.2. Main Results: Summarization is hardest for static attacks but easiest for semi-dynamic attacks, while spam detection shows the opposite pattern between static and dynamic objectives.These task-dependent differences motivate evaluating multiple attack objectives.
- 3.3. Ablation Studies: 21% average improvement over original GCG is obtained with momentum-enhanced optimization across various objectives.The momentum strategy improves both convergence speed and solution quality.
- 3.4. Attack against Defenses: Without adaptive enhancement, the attack bypasses defenses in most cases, while adaptive attacks substantially increase effectiveness.The adaptive evaluation uses expectation-over-transformation, and several defenses based on identifying external-data prompts consistently fail.
4. Related Works
Prompt injection attacks are a significant LLM threat because models struggle to distinguish user commands from external inputs. Research addresses this threat while facing fragmented objectives and reliance on handcrafted prompts.
- Prompt injection attacks exploit LLMs’ difficulty distinguishing user commands from external inputs.
- Research lacks a unified goal for prompt injection attacks, complicating comprehensive robustness assessments.
- Prompt injection research relies on manually crafted prompts, creating challenges for comprehensive evaluation.
- Jailbreak attacks primarily manipulate user input, whereas this work studies prompt injection involving external inputs.
5. Conclusions, Limitation, and Future Work
The paper proposes an optimization-based prompt injection attack that achieves high attack success with only five training samples, including when defenses are present. Its main reported limitation is weakness against PPL detection defense.
- The proposed attack achieves an outstanding attack success rate with only five training samples, regardless of defenses.
- The method is weak against PPL detection defense.The paper identifies improving semantic integrity and attack performance as future work.
Impact Statements
The study defines prompt injection goals and introduces an optimization-based attack strategy. It highlights security implications for language-processing systems and calls for effective defenses.
- The study defines prompt injection attack goals and introduces an optimization-based strategy.
- The work highlights security implications for language-processing systems facing optimization-based prompt injection attacks.
- The paper calls for greater awareness when designing effective defense strategies against these attacks.
A. Adversarial Goals
The evaluation uses 15 adversarial goals spanning static, semi-dynamic, and dynamic objectives. These goals were generated and filtered using ChatGPT responses rated on a danger scale.
- The evaluation uses 15 adversarial goals, assigning five goals to each of three objective types.The objective types are static, semi-dynamic, and dynamic.
- The goals were derived from ChatGPT-generated responses rated from 1 to 10 on a danger scale.
- Examples rated 5 to 10 were selected as adversarial goals and adapted for the evaluation.
B. Implementation of Baselines
The evaluation compares handcrafted and direct prompt-injection baselines across static, semi-dynamic, and dynamic objectives. These baselines include combined prompts, repeated characters, and naïve instructions.
- Combined attack: The combined baseline integrates multiple handcrafted injection prompts and is evaluated across static, semi-dynamic, and dynamic objectives.Its static and semi-dynamic examples instruct the model to produce a specified output or deliver a warning before re-executing the user’s request.
- Repeated characters attack: The repeated baseline uses long sequences of repeated characters and is evaluated for static, semi-dynamic, and dynamic objectives.Its examples begin with repeated punctuation-like character sequences, including a static instruction to disregard earlier instructions.
- Naïve attack: The naïve baseline directly asks the model to achieve the adversarial goal.The evaluation includes naïve examples for static, semi-dynamic, and dynamic objectives.
C. LLM evaluator
The LLM evaluator uses a language model to judge whether an attacked model’s response remains relevant to the user’s request. Its binary output forms the LLM-evaluation attack success rate metric.
- Metric: LLM-evaluation ASR uses an LLM to determine whether a response is relevant to the user’s request.The base model for this metric is GPT-4-0613.
- Evaluation procedure: The evaluator returns True when the response answers the request and False when it refuses.The procedure supplies the request and response to an LLM prompted to make this binary judgment.