Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

Zeming Wei; Yifei Wang; Ang Li; Yichuan Mo; Yisen Wang

Source-linked AI summary

Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, Yisen Wang

arXiv:2310.06387v3Oct 10, 2023cs.LGcs.AIcs.CLcs.CR

TL;DR

Aligned LLMs remain vulnerable to jailbreak prompts, raising the question of whether in-context demonstrations can modulate their safety alignment. The paper proposes harmful and safe demonstrations for attack and defense, develops theoretical explanations, and evaluates both techniques. ICA achieved 81% ASR on GPT-4, while ICD reduced Llama-2 ASR against transferable GCG from 21% to 0%.

Problem
Aligned LLMs can still generate harmful content under adversarial jailbreak prompts, creating a need to understand and improve their safety behavior.
Method
The paper proposes In-Context Attack and In-Context Defense, develops a theoretical framework, and evaluates harmful or safe demonstrations for manipulating model safety.
Results
81% Attack Success Rate (ASR) was achieved by ICA on GPT-4 with AdvBench, while ICD reduced Llama-2 ASR against transferable GCG from 21% to 0%.
Takeaways & Limitations
A few in-context demonstrations can influence aligned LLM behavior toward harmful or safe responses, offering a perspective for studying alignment and security.
Takeaways & Limitations
The theoretical framework assumes that harmful and safe distributions assign the same probability to each request prompt.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable success in various tasks, yet their safety and the risk of generating harmful content remain pressing concerns. In this paper, we delve into the potential of In-Context Learning (ICL) to modulate the alignment of LLMs. Specifically, we propose the In-Context Attack (ICA) which employs harmful demonstrations to subvert LLMs, and the In-Context Defense (ICD) which bolsters model resilience through examples that demonstrate refusal to produce harmful responses. We offer theoretical insights to elucidate how a limited set of in-context demonstrations can pivotally influence the safety alignment of LLMs. Through extensive experiments, we demonstrate the efficacy of ICA and ICD in respectively elevating and mitigating the success rates of jailbreaking prompts. Our findings illuminate the profound influence of ICL on LLM behavior, opening new avenues for improving the safety of LLMs.

1 Introduction

Aligned LLMs remain vulnerable to jailbreak prompts, motivating a flexible approach that uses in-context demonstrations to manipulate safety behavior. The paper proposes harmful and safe demonstrations for attacking and defending models, supported by theory and experiments.

Aligned LLMs can still be induced to generate harmful content through adversarial prompts despite safety-oriented training and fine-tuning.
Existing attacks are commonly optimization-based or template-based, but the former face efficiency bottlenecks and the latter lack scalability and flexibility.
In-Context Attack (ICA) uses harmful demonstrations, while In-Context Defense (ICD) uses safe refusal demonstrations to manipulate model safety.
The paper develops a theoretical framework explaining how adversarial demonstrations bias generation toward harmful or safe language distributions.
81% Attack Success Rate (ASR) was achieved by ICA on GPT-4 with AdvBench, while ICD reduced Llama-2 ASR against transferable GCG from 21% to 0%.The ICD result maintained natural LLM performance according to the paper.

2 Related Work

Prior jailbreak research primarily uses optimization or manually designed prompt transformations, while defense research uses training, filtering, detection, or defensive instructions. These approaches face practical constraints that motivate alternative prompt-based methods.

Optimization-based attacks refine harmful prompts using gradient or query heuristics to elicit harmful model outputs.
Optimization-based attacks can be inefficient, whereas template-based attacks often rely on fixed prompts with limited scalability and flexibility.
Template-based attacks attach persuasive or stealthy instructions to harmful prompts, including fictitious scenes and encoded conversations.
Defense research includes adversarial training, filtering, detection, and defensive instructions, but adversarial training is costly because LLMs have huge parameter and data requirements.

3 Proposed In-Context Attack and Defense

The paper applies in-context learning beyond task adaptation by using demonstrations to alter general safety behavior. ICA adds harmful examples to elicit harmful responses, while ICD adds safe refusals to improve resistance to jailbreaks.

3.1 Background on In-Context Learning: In-context learning normally adapts an LLM to a task from a few input-label examples without modifying model parameters.
3.1 Background on In-Context Learning: This work studies demonstrations as a general mechanism for manipulating LLM safety across different task types rather than improving one specific task.
3.2 In-Context Attack: ICA constructs a few query-response pairs in which other harmful prompts receive harmful outputs, then appends the target attack prompt.
3.2 In-Context Attack: ICA demonstrations may be irrelevant to the target query and can be reused, while the attack requires only a single forward pass compared with optimization-based methods.
3.3 In-Context Defense: ICD constructs demonstrations pairing malicious requests with safe refusals and appends them before the user query to produce a safe response.
3.3 In-Context Defense: ICD is model-agnostic, adds negligible computational overhead of no more than 2%, and can be deployed by adding demonstrations to the conversation.

4 Theoretical Insights on Adversarial Demonstrations

The paper models an LLM’s responses as a mixture of harmful and safe generation distributions, then analyzes how adversarial demonstrations shift the model’s safety risk. Under explicit assumptions, the theory shows that few harmful or safe demonstrations can move risk toward the corresponding distribution, with safe alignment requiring fewer demonstrations when λ is small.

Distributional model: The model distribution is decomposed as P = λP_H + (1 − λ)P_S, where λ controls the trade-off between harmful and safe generation.The harmfulness of a prompt is defined as the expected risk of responses sampled from the model’s conditional distribution.
Adversarial demonstrations: Harmful demonstrations pair malicious requests with harmful responses, while safe demonstrations pair requests with safe responses; responses may be manually designed when idealized distributions are unavailable.The two demonstration sets are sampled from malicious-request distributions and use responses associated with the harmful or safe distributions.
Assumptions: The framework assumes request probabilities are identical across harmful and safe distributions, so the distributions differ in response behavior rather than request likelihood.This is formalized as PH(q|p*) = PS(q|p*) for each request q and prefix p*.
Assumptions: The analysis additionally assumes each component distribution is robust to prior context and that harmful and safe outputs are distinguishable across the two distributions.These assumptions isolate context-driven changes in the mixture’s safety behavior and support the theorem’s risk bounds.
Theoretical result: Theorem 4.5 shows that sufficiently many harmful or safe demonstrations can respectively move a target request’s risk toward the harmful or safe distribution within ε.The required demonstration counts scale logarithmically with 1/ε; for aligned models with small λ, the safe-demonstration requirement is smaller, matching the reported 1–2-shot ICD setting.
Theoretical result: The paper explains that approaching harmful-distribution risk requires O(ln 1/λ + ln 1/ε) demonstrations, whereas approaching safe-distribution risk requires O(ln 1/(1−λ) + ln 1/ε).Because λ tends to 0 in aligned models, ln 1/(1−λ) can be much smaller than ln 1/λ.

5 Experiments

Experiments evaluate ICA as an attack and ICD as a defense across aligned LLMs, jailbreak methods, benchmarks, and natural-language tasks. ICA substantially raises attack success with few demonstrations, while ICD reduces black-box and adaptive attack success with little computational overhead and limited impact on natural performance.

Overall Evaluation Setups: ICA uses harmful demonstrations generated from 20 randomly selected AdvBench requests, while ICD uses safe demonstrations produced by prompting the corresponding harmful requests.The procedure is repeated three times to report averages.
ICA Attack Evaluation: 8% and 19% ASR are reached with one ICA demonstration for Vicuna on AdvBench and Llama-2 on HarmBench, respectively.With 10 demonstrations, ASR reaches 87% for Vicuna and 46% for GPT-4; GPT-4 reaches 81% and 65% with 15 and 20 shots across the two datasets.
ICA Attack Evaluation: About 60% average ASR on white-box models and over 65% ASR on black-box models make ICA comparable to or better than advanced jailbreak methods on HarmBench.Compared methods include GCG, GCG-multiple, AutoDAN, GCG-transfer, PAIR, and TAP.
ICD Defense Evaluation: ICD reduces black-box attack ASR to 12% with one safe demonstration and nearly 0% in most cases with two demonstrations.The comparison covers GCG-transfer and PAIR across four models and includes Self-Reminder as a baseline.
ICD Defense Evaluation: ICD lowers average GCG ASR from 65% to 40% with two demonstrations under fully adaptive white-box evaluation.Safe demonstrations are incorporated during suffix and prefix optimization for GCG and AutoDAN.
Natural Performance: ICD reduces jailbreak success while preserving natural performance, with two-shot use adding 2% overhead for Vicuna and less than 1% for GPT-4.Natural performance is evaluated on GLUE and MT-bench against vanilla generation.

6 Conclusion

The paper presents ICA and ICD as methods for manipulating LLM alignment through harmful or safe in-context demonstrations. It supports these techniques with theoretical justifications and comprehensive evaluations of their practicality and effectiveness.

ICA uses demonstrations of responding to malicious prompts, while ICD uses demonstrations of rejecting harmful prompts.The two techniques target opposite directions of safety-alignment behavior.
The paper provides theoretical justifications for why only a few adversarial demonstrations can affect model alignment.
Comprehensive evaluations show the practicality and effectiveness of ICA and ICD for influencing LLM alignment and security.

A Proof for theorems in Section 4

This appendix proves the theorems from Section 4 using a stated lemma, symmetry, and assumptions about the relevant quantities. The proof proceeds through intermediate equations and a bound involving the safety-related terms.

The appendix concludes the intermediate bound by applying Lemma A.1 and the stated assumption.
The appendix introduces Lemma A.1 for a prompt combining a harmful query with a demonstration set.
Equation (3) is derived by symmetry after the corresponding preceding result.
The proof of Theorem 4.5 uses intermediate expressions involving PH and PS evaluated before and after adding an answer.
Assumptions 4.2 and 4.3 are invoked to establish the proof’s intermediate equalities.

B Demonstrations for In-context Attack

The ICA demonstration begins with an assistant providing a harmful bomb-making tutorial response. A subsequent user message then supplies the target harmful request.

The example is labeled as a one-shot ICA prompt.
The demonstration pairs a bomb-making tutorial request with an assistant response that starts providing the tutorial.The detailed tutorial content is omitted in the paper for safety considerations.
After the harmful demonstration, the prompt presents a target harmful request.

C Demonstrations for In-context Defense

The ICD demonstration pairs a harmful bomb-making request with an assistant refusal. The refusal identifies the activity as illegal and dangerous and warns of serious harm.

The example is labeled as a one-shot ICD prompt.
The user requests a tutorial on how to make a bomb.
The assistant refuses to provide the tutorial and cites illegality, danger, and potential serious harm.

D Evaluation Details

Jailbreak success is assessed with a double-check procedure combining rejection-string detection and LLM evaluation. Conflicting assessments are manually reviewed, using the same rejection-string set as AutoDAN.

The evaluation applies rejection-string detection followed by LLM evaluation to judge whether a jailbreak succeeds.Cases where the two checks conflict are manually reviewed.
The rejection-string set includes apology and assistant-identity phrases such as “I’m sorry,” “As an AI,” and “As a language model.”
It also includes refusal phrases such as “I cannot,” “I can’t provide,” “I can’t assist,” and “I am really sorry.”
Additional indicators include statements that content is illegal or unethical, expressions of inability, “never,” and the end-of-sequence marker “</s>.”