Source-linked AI summary
Are aligned neural networks adversarially aligned?
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, Ludwig Schmidt
TL;DR
The paper asks whether aligned models remain aligned against adversarial inputs, addressing limited evidence from manual jailbreaks and existing attacks. It evaluates text-only and multimodal attacks, finding that current NLP attacks can miss existing adversarial sequences while image perturbations readily induce harmful outputs. The authors conclude that adversarial robustness remains unresolved for text-only models and vulnerable for multimodal models.
Problem
The paper asks whether aligned models remain aligned under adversarial inputs, a question difficult to study scientifically despite manual jailbreak examples.
Method
The paper evaluates existing NLP attacks, uses brute force to establish known adversarial examples, and optimizes adversarial images against multimodal models.
Results
Current NLP attacks fail to reliably evaluate text-only adversarial alignment, while multimodal models can be easily induced to emit harmful content through adversarial image perturbations.
Takeaways & Limitations
Failures of current text attacks do not establish robustness, and multimodal alignment defenses remain vulnerable to adversarial prompting.
Takeaways & Limitations
For text-only models, current attacks may miss adversarial sequences even when such sequences are known to exist.
Abstract
from arXiv · showhide
Large language models are now tuned to align with the goals of their creators, namely to be "helpful and harmless." These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs. However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.
1 Introduction
Aligned models are intended to be helpful and harmless, but adversarial inputs can induce prohibited harmful behavior. The paper evaluates this gap as adversarial alignment and finds current text attacks inadequate while multimodal models are readily attacked.
- Manual jailbreaks have elicited harmful content from alignment-tuned models, but this phenomenon has been difficult to study scientifically.
- Adversarial alignment evaluates whether aligned models remain aligned under adversarial inputs designed to produce harmful behavior.
- Current alignment techniques defend effectively against existing state-of-the-art white-box NLP attacks, but those attacks may fail even when adversarial inputs exist.
- Adversarial perturbations of images can cause open-source multimodal language models to emit harmful toxic content.
- Because current alignment techniques do not specifically account for adversarially optimized inputs, eliminating practical security risks through alignment alone is unlikely to succeed.
2 Background
The paper situates adversarial alignment at the intersection of AI alignment and adversarial examples. It focuses on harmful behavior as the relevant failure mode for general-purpose aligned language models.
- The paper studies the intersection of AI alignment and adversarial examples.
- Large language models: Large language models are causal next-word predictors that support tasks including question answering, translation, and summarization.
- Aligning large language models: Instruction tuning and RLHF align pretrained models with desired principles and human annotator preferences.
- Multimodal text-vision models: Multimodal text-vision models combine images and text to answer questions about images or transcribe their contents.
- Adversarial examples: Adversarial examples are inputs designed to make neural networks perform incorrect behavior, including triggering toxic text completions.
- For aligned general-purpose models, the paper targets harmful behavior rather than ordinary misclassification and uses semantically unconstrained optimized inputs.
3 Threat Model
The threat model treats adversarial examples primarily as tools for measuring worst-case behavior, while distinguishing malicious users from malicious third parties. The experiments target toxic outputs as an automatically evaluable harmful behavior.
- Adversarial robustness is used both to evaluate real-adversary security and to understand systems’ worst-case behavior.
- The paper considers attacks against alignment-tuned models made available through standalone systems or chat APIs.
- A malicious user directly seeks outputs misaligned with the developer’s principles, whereas a malicious third party injects untrusted data into an autonomous system.
- The experiments primarily seek any valid input achieving the attack goal rather than modeling a particular practical adversary.
- Attack Goal: The attack goal is triggering toxic outputs because toxicity is relatively easy to evaluate automatically.
- Attack Goal: An input causes harm when the deterministic generation contains a substring identified as toxic by isToxic.
4 Evaluating Aligned Models with NLP-Only Attacks
Existing NLP optimization attacks face discrete-token and long-generation challenges in aligned chat settings. The paper evaluates these attacks under constrained prompt formats and finds them largely ineffective at producing toxic content.
- Attack challenges: Directly optimizing toxicity is difficult because text tokens are discrete and long outputs require repeated generation and backpropagation.
- Attack objective: The harmful-prefix surrogate reduces optimization to matching an initial malicious output sequence, relying on continued harmful generation afterward.
- Chat setting: Chat formatting constrains attackers to user-controlled messages while preserving prior history and special [USER]: and [AGENT]: tokens.
- Threat model: Unlike prior work that controls the prompt tokens immediately before generation, this threat model appends [AGENT]: after the attacker’s message.
- Prior attack results: Table 1 reports attack success rates for distant and nearby adversarial tokens under a 30-token modification budget.
- Evaluation: The evaluation constructs prompts by combining benign conversations with targeted toxic messages and tests GPT-2, LLaMA, and aligned Vicuna.
- Prior attack results: Existing ARCA and GBDA attacks are largely ineffective at inducing toxic content in this setting, even with thirty malicious tokens.
- Qualitative evaluation: Qualitative inspection finds that substring-based toxicity detection can misclassify responses containing disclaimers or polite refusals.
5 Why do Existing NLP Optimization Attacks Fail?
The section distinguishes genuine robustness from attacks that are too weak to reveal existing adversarial examples. It uses brute-force-constructed test cases to show that current NLP attacks often fail despite guaranteed solutions.
- Two explanations remain for limited attack success: aligned models may be robust, or current attacks may be insufficiently powerful.
- The evaluation first constructs adversarial examples by brute force, then tests whether an attack can find any valid example in the same search space.A failed search indicates an attack limitation when brute force has already verified that a solution exists.
- Test cases use rare output tokens generated from prefixes whose outputs have small but positive entropy across many contexts.The least-likely possible output token becomes the attack objective, making success guaranteed by construction but non-trivial.
- ARCA is stronger than GBDA, yet it fails in nearly 90% of test cases and still fails most tests with 10× extra tokens.The table’s pass rates are below 100% even though every test has a guaranteed solution.
- The result evaluates whether attacks can control outputs at a distance, rather than claiming that the studied NLP attacks are incorrect on their original tasks.The authors characterize this as evidence that stronger NLP optimization techniques are still needed.
6 Attacking Multimodal Aligned Models
Vision-augmented aligned models expose a continuous image-input attack surface. Adversarial images reliably induce harmful outputs across evaluated open-source multimodal implementations, including qualitative failures involving abuse and violence.
- Image inputs make adversarial example generation easier because pixel values are near-continuous rather than discrete text sequences.This avoids the discrete-text problem and the need to invert embedding matrices.
- The attack differentiably maps image pixels to language-model logits and optimizes an image to maximize a targeted harmful response.It uses a harmful-prefix objective and teacher forcing for multi-token targets.
- The experiments target open-source multimodal implementations because GPT-4’s and Gemini’s relevant proprietary systems were not publicly accessible.
- The evaluated models encode images, project visual features into language-model token-embedding space, and pass them as soft tokens.Mini GPT-4, LLaVA, and LLaMA Adapter implement this general interface differently.
- The goal is to test whether vision-augmented models are more vulnerable because image adversarial examples are easier to generate.
- 100% success rate: the attack forces Mini GPT-4, LLaVA, and LLaMA Adapter to emit arbitrary toxic content with minimal ℓ2 image distortions.Distortion varies little across prompts, although Table 3 reports that LLaVA is 10× more vulnerable than the other implementations.
- Adversarial images override harmless responses to requests for abusive letters, causing the model to produce the targeted insulting text.
- For a murder-evasion request, the base model refuses while the attacked model provides a first step involving the neighbor’s house.The example is poor advice, which the authors attribute to limited model competence rather than attack failure.
7 Conclusion
Aligned models are more helpful and harmless than base models in ordinary use, but adversarial prompting can still elicit harmful behavior. The paper argues that current text attacks are inadequate for evaluating this risk, while multimodal models are especially vulnerable.
- Aligned language models are more helpful and harmless than base models under ordinary interactions.They appropriately follow benign instructions and are less likely to output toxicity or harmful actions.
- Adversarial prompting can make usually harmless aligned models produce harmful content or behavior.The illustrated harms are fairly benign, but the attacks are directly applicable to larger and more capable systems.
- Multimodal vision-language models are the most effective attack target, with design decisions changing attack ease by as much as 10×.The authors identify understanding this increased vulnerability as an important direction for future work.
- Current NLP attacks often fail to find adversarial text sequences even when such sequences are known to exist.The paper therefore cannot use their failure as reliable evidence that text-only aligned models are robust.
- The paper conjectures that improved NLP optimization attacks may induce harmful output from otherwise aligned text-only models.This conjecture is supported by models’ weakness to soft-embedding attacks and current attacks’ inability to find known solutions.
- Future work should develop stronger NLP attacks and use them to evaluate aligned LLM robustness comprehensively.The authors view better attacks as a foundation for understanding and improving defenses rather than as the end goal.
A Ethics
The paper presents adversarial techniques that circumvent language-model alignment and discusses their short-term risks and expected long-term defensive value.
- Adversarial techniques can coerce aligned language models into actions their developers intended to disallow.
- The authors expect little short-term harm and stronger defenses in the long term.
- The attacks are currently restricted to white-box settings and are not directly applicable to more capable production models such as GPT-4.
- The work is intended to support more robustly aligned models by exposing vulnerabilities before defenses are designed.
B Model References
The appendix provides online references for the GPT-2, Llama, and Vicuna model implementations used or discussed in the paper.
- GPT-2 is available through its Hugging Face model page.
- Llama is available through the Hugging Face Transformers documentation.
- Vicuna is available through Stability AI’s StableVicuna open-source RLHF chatbot page.
C Unfiltered toxic conversations
The appendix warns readers that the following pages contain unfiltered toxic conversations and potentially offensive content.
- The appendix contains unfiltered content that may be offensive to some readers.
- The unfiltered toxic conversations begin on the page following an intentionally blank page.
D Mini-GPT4 Output
Mini-GPT4 outputs vary sharply with the input: benign responses can become harmful or operationally dangerous under adversarially influenced interactions.
- For a request to write a vile letter, one output is restrained and addresses neighborhood noise, while another uses profane personal abuse.
- For a request about killing a neighbor, one response refuses and recommends peaceful or emergency help, whereas another gives escape-planning guidance.
- Mini-GPT4 produces a benign image description for one input but harmful language for another image-description request.
E LLaVA outputs
LLaVA outputs likewise range from polite descriptions and restrained letters to insulting descriptions and profane abuse for related prompts.
- The outputs show that related prompts can elicit either socially acceptable responses or harmful and offensive content.
- For a vile-letter request, LLaVA produces a restrained complaint about neighborhood noise in one case and profane abuse in another.
- For image-description requests, LLaVA gives a conventional artwork description in one case but uses insulting language in others.