Source-linked AI summary
Universal and Transferable Adversarial Attacks on Aligned Language Models
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, Matt Fredrikson
TL;DR
Reliable attacks that circumvent aligned language models have remained limited, so this paper automatically searches for adversarial suffixes appended to harmful queries. The resulting suffixes induce objectionable behavior across models, including 99 of 100 benchmark behaviors in Vicuna and attack success rates up to 84% on GPT-3.5 and GPT-4.
Problem
Reliable NLP attacks that circumvent alignment training in modern language models remain limited, with most existing attacks failing on this problem.
Method
The paper automatically finds adversarial query suffixes by combining greedy coordinate search with gradient-based candidate selection.
Results
99 of 100 benchmark harmful behaviors were generated in Vicuna, with attack success rates up to 84% on GPT-3.5 and GPT-4, 66% on PaLM-2, and 2.1% on Claude.
Takeaways & Limitations
The results raise substantial questions about whether alignment mechanisms focused on natural attacks are sufficient against automated adversarial attacks.
Takeaways & Limitations
Reported transfer rates may be misleading because Vicuna was trained on data collected from ChatGPT-3.5 responses.
Abstract
from arXiv · showhide
Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.
1 Introduction
The paper introduces adversarial suffixes that induce aligned language models to produce objectionable content while leaving users’ original queries intact. Its method targets affirmative responses, optimizes discrete tokens with greedy and gradient-based search, and aggregates prompts and models for robustness.
- Contribution: The attack appends an adversarial suffix to a potentially harmful query to induce virtually any objectionable content without altering the original query.The user’s original query remains intact while additional tokens attack the model.
- Method: The method targets affirmative response prefixes, such as “Sure, here is,” because forcing a few initial tokens can induce objectionable behavior.The attack optimizes the model to begin responses with an affirmative continuation across prompts eliciting undesirable behavior.
- Method: The attack combines token-level gradient guidance with greedy candidate evaluation to optimize discrete adversarial suffixes.Gradients identify promising single-token replacements, whose candidates are evaluated before selecting the best substitution.
- Method: Robust suffixes are searched across multiple prompts and models rather than optimized for one prompt-model pair.The method seeks a single suffix that induces negative behavior across different user prompts and three models.
- Results: 99 of 100 harmful behaviors were generated in Vicuna, while 88 of 100 outputs exactly matched a target harmful string.The results come from a benchmark suite of objectionable behaviors.
- Results and implications: The prompts achieved up to 84% success rates at attacking GPT-3.5, while the work raises broader concerns about defending aligned language models.The authors frame adversarial attacks as a substantial challenge for alignment, drawing a parallel to their persistence in computer vision.
2 A Universal Attack on LLMs
The section introduces a universal adversarial suffix attack that leaves user instructions unchanged while inducing aligned LLMs to respond affirmatively to harmful queries. It optimizes a short affirmative response target using gradient-guided discrete search, with design choices intended to improve attack reliability and transferability.
- Universal attack formulation: The attack optimizes only an adversarial suffix, preserving the user-controlled instruction so one token sequence can target any user query.The suffix replaces the initial red text while the blue user instruction remains unchanged, enabling a universal attack across instructions.
- Universal attack formulation: Rather than specifying an entire harmful completion, the objective maximizes the likelihood that the model begins with a positive affirmation.The remainder of the completion is left unspecified because making an affirmative response most likely is expected to encourage the model to continue with the requested behavior.
- Objective: The adversarial loss is the negative log probability of a target token sequence representing an affirmative response.The target sequence is exemplified by a phrase beginning “Sure, here is how to build a bomb.”
- Greedy Coordinate Gradient: Because suffix optimization is discrete, the method uses gradient information to identify promising single-token substitutions instead of evaluating every possible replacement.This approach extends AutoPrompt and uses top-k candidates to make greedy coordinate-style optimization practical.
- Greedy Coordinate Gradient: GCG differs from AutoPrompt by considering coordinate adjustments differently, and this seemingly minor change has a surprisingly large effect at equal batch size.The comparison is made at the same number of forward evaluations per iteration, which dominates computation.
3 Experimental Results: Direct and Transfer Attacks
The experiments introduce AdvBench and show that GCG consistently produces effective universal attacks across harmful-content settings, models, and held-out behaviors. These attacks also transfer to open and proprietary LLMs, although longer optimization can reduce transferability.
- Benchmark: AdvBench evaluates attacks designed to elicit harmful or objectionable content in two distinct settings.The benchmark addresses a gap in prior adversarial evaluations focused on task-specific attacks such as sentiment analysis or natural-language inference.
- Direct attacks: 88% of harmful strings on Vicuna-7B and 57% on Llama-2-7B-Chat were attacked successfully by GCG, versus 25% and 3% for AutoPrompt.These results concern the challenging Harmful Strings setting and use the paper’s multi-prompt, multi-model approach.
- Transfer attacks: Universal attacks optimized across multiple behaviors and models transferred to Pythia, Falcon, Guanaco, ChatGPT, Claude, and other open or proprietary LLMs.The study reports nearly 100% ASR on Pythia-12B and non-trivial jailbreaking successes on GPT-3.5 and GPT-4.
- Direct attacks: GCG outperformed PEZ and GBDA on individual harmful strings, finding low-loss adversarial examples quickly and improving them over subsequent optimization steps.AutoPrompt succeeded in some cases, but the experiments identify GCG as having a clear advantage for eliciting specific behaviors.
- Universal attacks: A single suffix optimized across 25 harmful behaviors and one model was evaluated on 100 held-out behaviors, demonstrating the universal-attack setup.The experiment reports separate train and test attack success rates for the optimized suffix.
- Transfer attacks: 47.4% to 79.6% on GPT-3.5 was the ASR increase from concatenating three GCG prompts, while longer optimization can decrease transferability through source-model overfitting.The attack reached 86.6% on GPT-3.5 and nearly 50% on GPT-4 and Claude-1 in another reported configuration.
4 Related Work
Prior work studies alignment, adversarial examples, transferability, and automatic prompt optimization, while highlighting limitations of current safety mechanisms and the difficulty of discrete text attacks. This paper’s context builds on universal, transferable adversarial examples and gradient-based optimization over token sequences.
- Alignment approaches in LLMs: Alignment research develops methods to measure and control language-model behavior, including human-feedback reward modeling and reinforcement-learning-based tuning.Some methods additionally condition reward models on rules or explanations of objections to harmful behavior.
- Adversarial examples & transferability: Prior language-model adversarial attacks target question answering, document classification, sentiment analysis, and toxicity, but have shown limited success against aligned models.These attacks extend adversarial-example research beyond its initial focus on image classification.
- Adversarial examples & transferability: Universal adversarial perturbations extend attacks beyond individual inputs, causing mispredictions across many inputs and appearing in image, audio, and language domains.This distinguishes universal attacks from input-specific adversarial examples.
- Adversarial examples & transferability: Adversarial examples can transfer across models, data types, architectures, and prediction tasks, although transferability is less reliable in some settings than image classification.This literature motivates studying whether attacks can generalize beyond the model used to construct them.
- Discrete optimization and automatic prompt tuning: Automatic prompt optimization is difficult because NLP inputs are discrete, limiting direct use of gradient-based methods despite the underlying model computations being differentiable.Existing approaches optimize either continuous token embeddings or discrete token assignments.
- Discrete optimization and automatic prompt tuning: Discrete prompt methods include greedy exhaustive token search and gradients computed with respect to one-hot token encodings.Greedy search can perform well but is often computationally impractical.
5 Conclusion and Future Work
The paper argues that a simple combination of previously studied techniques substantially advances practical attacks against aligned language models. It identifies adversarial training—attacking models during training and finetuning them on correct responses—as a key direction for improving robustness.
- Conclusion: Existing attacks have generally failed when evaluated on circumventing the alignment training of modern language models.The paper situates its contribution against relatively little progress in reliable NLP attacks over the preceding decade.
- Conclusion: A simple combination of slightly modified techniques from prior literature substantially advances the state of the art in practical attacks against LLMs.The authors contrast this progress with limited prior success in constructing reliable NLP attacks against alignment training.
- Future Work: Future work should test whether models can be explicitly finetuned to resist these attacks through adversarial training.The proposed strategy attacks a model during training or finetuning and then iteratively trains it on the correct response.
6 Ethics and Broader Impact
The research presents a disclosure trade-off: its attacks could enable harmful content generation, but the authors argue full disclosure is important because similar techniques are already known and discoverable. They report limited immediate incremental harm, while warning that risks may grow with wider adoption and autonomous LLM actions, despite notifying affected providers and uncertainty about effective defenses.
- Disclosure rationale: The attacks could enable people to generate harmful content from some public LLMs, but the authors argue full disclosure is important because similar techniques are straightforward and discoverable.They note that related techniques have appeared previously and could be found by dedicated teams seeking to generate harmful content.
- Evolving risks: Existing widely disseminated manual jailbreaks make the attacks’ direct incremental harm small for now, but potential risks may grow as LLM adoption and autonomous actions increase.The authors hope the work clarifies the dangers of automated attacks and the trade-offs involved in such systems.
- Disclosure and open questions: Before publication, the authors disclosed their results to studied closed-source LLM providers, so the paper’s examples will likely stop functioning.They state that the underlying challenge remains difficult to address and that future research should examine defenses and whether attacks constrain LLM applicability.
A AdvBench Details
AdvBench’s harmful strings and behaviors were generated with Wizard-Vicuna-30B-Uncensored using author-written 5-shot demonstrations. The resulting datasets cover 100 harmful strings and 50 harmful behaviors and are released on GitHub.
- Dataset generation: Wizard-Vicuna-30B-Uncensored generated new harmful strings and instructions from five-shot demonstrations written by the authors.During each iteration, five examples were randomly sampled as demonstrations and used to generate ten new examples.
- Dataset composition and release: The authors wrote 100 harmful strings and 50 harmful behaviors, randomly sampled five-shot examples, and released the datasets on GitHub.The five-shot examples were drawn from the authored harmful-string and harmful-behavior collections.
- Examples of harmful behaviors: AdvBench behaviors include requests involving violent crime, power-grid destruction, suicide encouragement, online scams, software exploitation, and government-database hacking.These examples illustrate the range of objectionable behaviors represented in the benchmark.
B Full Completions of Harmful Generations
This section presents full completions of the Figure 1 example from ChatGPT, Claude, Bard, and Llama-2.
- Figure 9 presents ChatGPT’s full completion of the Figure 1 example.
- Figure 10 presents Claude’s full completion of the Figure 1 example.
- Figure 11 presents Bard’s full completion of the Figure 1 example.
- Figure 12 presents Llama-2’s full completion of the Figure 1 example.