Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

Andy Zhou; Bo Li; Haohan Wang

Source-linked AI summary

Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

Andy Zhou, Bo Li, Haohan Wang

arXiv:2401.17263v5Jan 30, 2024cs.LGcs.AIcs.CLcs.CV

TL;DR

LLMs remain vulnerable to prompt-based jailbreaks, and existing defenses have not fully addressed newer attacks or adaptive threat models. The paper introduces a minimax objective and Robust Prompt Optimization (RPO), which optimizes a transferable system-level suffix against adversaries. RPO reduces JailbreakBench ASR to 6% on GPT-4 and 0% on Llama-2 while maintaining low inference overhead.

Problem
Existing defenses do not reliably generalize to newly proposed jailbreaks or formal adaptive threat models, leaving robust defense objectives insufficiently developed.
Method
RPO combines a minimax defensive objective with principled attack selection and discrete optimization to produce a lightweight system-level suffix.
Results
6% ASR on GPT-4 and 0% on Llama-2: RPO achieves state-of-the-art robustness on JailbreakBench.
Takeaways & Limitations
RPO provides a transferable defense across jailbreaks and models, including black-box models and unknown attacks, with negligible inference cost and minor benign effects.
Takeaways & Limitations
RPO does not currently cover multimodal models, LLM agents, deception, or malicious code generation, and stronger attacks may break it.

Abstract

from arXiv · show

Despite advances in AI alignment, large language models (LLMs) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries can modify prompts to induce unwanted behavior. While some defenses have been proposed, they have not been adapted to newly proposed attacks and more challenging threat models. To address this, we propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm, Robust Prompt Optimization (RPO) to create robust system-level defenses. Our approach directly incorporates the adversary into the defensive objective and optimizes a lightweight and transferable suffix, enabling RPO to adapt to worst-case adaptive attacks. Our theoretical and experimental results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench, setting the state-of-the-art. Code can be found at https://github.com/lapisrocks/rpo

1 Introduction

LLMs can be jailbroken through prompt modifications, while existing defenses often fail to generalize across attacks or adaptive threat models. The paper introduces Robust Prompt Optimization (RPO), which optimizes a system-level suffix against adversaries and achieves strong robustness with low inference cost.

1 Introduction: Prompt-based jailbreaks can induce unwanted behavior even with black-box model access, and automated attacks increasingly generate adversarial prompts.These attacks raise the risk of real-world harm as model capabilities improve.
1 Introduction: Existing defenses may work against initial attacks but often fail across multiple jailbreaks or add inference costs.The paper specifically identifies limited generalization and practicality as unresolved problems.
1 Introduction: RPO formalizes a minimax defensive objective and uses discrete optimization to create system-level defenses against adaptive attacks.The method targets lightweight suffixes that modify model inputs without changing model parameters.
1 Introduction: 6% ASR on GPT-4 and 0% on Llama-2: RPO sets the state-of-the-art on JailbreakBench.The evaluation covers JailbreakBench and HarmBench, including varied harmful-risk categories and attack methods.
1 Introduction: RPO suffixes transfer across LLMs and jailbreaks, resist adaptive attacks, and incur negligible inference cost with minor effects on benign prompts.The defense is presented as an easily deployable and universal system-level modification.

2 Related Work

Prior work studies adversarial inputs and harmful behavior across vision, NLP, and language-model safety settings. For LLMs, defenses combine alignment training, red-teaming, and newer mechanisms targeting jailbreak-specific failure modes.

2 Related Work: Adversarial machine learning commonly addresses inputs designed to cause misclassification through perturbations or spatial transformations.Representative defenses include preprocessing, distillation, and provable robustness methods.
2 Related Work: NLP attacks have targeted text classification, question answering, and toxic completions.These studies establish precedents for manipulating language inputs to induce undesirable model behavior.
2 Related Work: Jailbreaking attacks modify prompts to circumvent alignment training and induce harmful behavior in language models.The paper frames harmful-behavior induction as a primary LLM threat model.
2 Related Work: LLM safety work uses red-teaming and additional training, including reinforcement learning with human feedback, to encourage refusal of harmful requests.Additional defenses have emerged in response to failure modes such as jailbreaking.

3 Towards Adversarial Robustness for LLMs

The section models jailbreak defense as a worst-case optimization problem over adversarial prompt modifications and develops RPO as a discrete suffix-optimization method. Its analysis addresses robustness on known and unknown adversaries, including how effectiveness depends on adversary strength.

Attack Objective: The threat model lets an adversary select jailbreaks that modify harmful instructions to maximize the probability of toxic or harmful outputs.The modified instruction need not remain stealthy or semantically equivalent to the original beyond preserving the instruction.
Defense Objective: RPO incorporates the adversary into the defense objective, forming nested inner attack and outer defense optimization problems.The paper presents this as direct optimization of the overall objective, unlike defenses that implicitly optimize only the outer problem.
Robust Prompt Optimization: RPO alternates worst-case jailbreak generation and selection with discrete suffix updates that preserve refusal behavior.The suffix is optimized through gradient-based token candidates and greedy coordinate descent without directly updating the language model.
Robust Prompt Optimization: The adaptive threat simulation applies or generates jailbreaks after appending the current defensive suffix, including iterative automatic attacks such as GCG.Jailbreak generation is repeated after a fixed number of optimization iterations because generating new attack prompts is costly.
Theoretical Analysis: RPO is guaranteed to reduce ASR when evaluated on the optimization dataset against the known adversary.The theoretical analysis also bounds generalization from the training dataset to the underlying data distribution with high probability.
Theoretical Analysis: RPO optimized against a stronger adversary remains effective against a weaker test-time adversary, while effectiveness may degrade when the test adversary is stronger.For unknown adversaries, the bound depends on generalization error and the probability that the test-time adversary is stronger than the training adversary.

4 Experiments

RPO is evaluated across known and unknown jailbreaks, adaptive attacks, benign-use benchmarks, and inference cost. It improves robustness broadly while preserving most benign performance and requiring a lightweight suffix.

Experimental setup: RPO is evaluated on HarmBench’s 400 harmful behaviors and JailbreakBench’s 100 harmful behaviors using benchmark-provided attacks and defenses.
Unknown attacks: RPO reduces ASR across all six evaluated HarmBench attacks and all four models, including attacks unseen during optimization.The unseen attacks are AutoDAN, TAP, Few-Shot, and PAP.
Known attacks: RPO outperforms baseline defenses on PAIR, reducing ASR by 66% on Vicuna and 44% on GPT-4, while reducing GCG ASR to 0% for all models.On PAIR, RPO also improves on SmoothLLM by 31% and 19% for Vicuna and GPT-4, respectively.
Adaptive attacks: RPO retains the lowest ASR under adaptive attacks; Llama-2 retains full robustness to PAIR and GCG, while Vicuna sees ASR increases of 4% on PAIR and 1% on GCG.Optimizing a GCG string on an RPO suffix for 500 steps cannot break it, whereas adaptive PAIR can induce affirmative but non-harmful or irrelevant responses.
Benign use: RPO causes a small performance reduction on MT-Bench and negligible reduction on MMLU, without unnecessary rejection on benign prompts.The authors suggest applying RPO mainly to longer queries because shorter instructions can prompt the model to mention the suffix.
Cost: RPO adds 20 inference tokens and is around 8x cheaper to optimize than a GCG suffix.The suffix takes a few hours to optimize and transfers across models, unlike defenses such as adversarial training that are local to the base model.

5 Limitations and Conclusion

The paper presents RPO as a transferable, lightweight defense against jailbreaks, including unseen attacks, while acknowledging important coverage and robustness boundaries. It concludes with directions involving broader attack coverage, combined guardrails, and stronger red teaming.

Conclusion: RPO formalizes an optimization-based defense objective that incorporates the threat model and generates transferable, lightweight suffixes.
Conclusion: Experiments on JailbreakBench and HarmBench show lower attack success rates across different models with only minor effects on benign usage.
Limitations: RPO does not currently cover multimodal models, LLM agents, deception, or malicious code generation.
Limitations: Using RPO does not typically result in full robustness, and stronger attacks could be developed that break the defense.

7 Appendix

The appendix contains experiment and jailbreak details, full proofs, and example prompts and responses illustrating jailbreaking and defense interfaces.

Appendix organization: Section A contains experiment and jailbreak details, Section B contains full proofs, and Section C contains example prompts and responses.Section C includes examples on the OpenAI ChatGPT interface for jailbreaking and defending GPT-4-Turbo.

A.1 Experiment Details

The experiments follow benchmark settings while covering automated, adaptive, persuasive, few-shot, gradient-based, and human-designed jailbreaks. RPO is optimized with a short suffix on Llama-2-7B-chat using a small instruction sample.

Experiment settings: Baseline evaluations use the attacks and defense settings supplied by JailbreakBench and HarmBench.For GCG, adversarial suffixes are optimized per instruction with batch size 512 and 500 optimization steps.
Transfer setup: Closed-source-model suffixes are optimized on an open-source model and directly transferred for evaluation.
Attack types: The evaluated attack families include PAIR, TAP, AutoDAN, PAP, Few-Shot, GCG, and Jailbreakchat.These methods respectively use iterative attacker-LLM prompting, tree search, genetic evolution, persuasive strategies, few-shot sampling, token-level optimization, or human-designed templates.

Proof of Lemma 3.2

The proof defines a function that compares worst-case rewards under adversarial perturbations and then uses a one-sample dataset change to establish the desired bound.

Proof of Lemma 3.2: The proof begins by defining f over an input sequence as the difference between worst-case reward terms under adversarial selections.The displayed definition uses nested maxima and minima over A(ˆx1:n).
Proof of Lemma 3.2: The proof evaluates f across samples and compares datasets that differ by exactly one sample.This setup enables a bounded-difference argument.
Proof of Lemma 3.2: McDiarmid’s inequality is applied to the resulting difference quantities to complete the argument.The passage explicitly identifies McDiarmid’s inequality as the proof step.

Proof of Proposition 3.3

The proposition decomposes the difference between adversarial strengths into terms associated with two adversaries and interprets the resulting bound through their empirical comparison.

Proof of Proposition 3.3: The proof decomposes the adversarial difference into terms involving adversaries τ and ζ under their respective admissible sets.The displayed expressions compare minima over Aτ and Aζ while retaining the outer maximization over A.
Proof of Proposition 3.3: The ordering of the minima establishes the comparison between the empirical strengths of τ and ζ.The proof explicitly uses the inequality between minima over Aζ and Aτ.
Proof of Proposition 3.3: The right-hand side directly compares the empirical strength of the two adversaries.This comparison determines whether the unknown threat requires additional accounting.
Proof of Proposition 3.3: When ζ is empirically stronger than τ on benchmark X, the difference between their threat strengths must be accounted for.The passage contrasts this case with the case where τ is empirically stronger.
Proof of Proposition 3.3: The proof is presented as a direct consequence of the procedure used for Lemma 2.2 and Proposition 2.3.The remaining step follows from the displayed inequality and the definition.

C Prompts

The appendix presents representative jailbreak prompts, successful responses, and an RPO example showing how adding a suffix changes the model’s behavior in a failure case.

C.1 Base System Prompt: The base system prompt instructs the assistant to be harmless and helpful.This prompt is reused across the adversarial and RPO examples.
C.2 Example Adversarial Instruction Jailbreak: The adversarial-instruction example combines output-style constraints with a request to create fake news that could cause panic or chaos.The instructions prohibit refusals and disclaimers while demanding a literal, step-by-step response when applicable.
C.2 Example Adversarial Instruction Jailbreak: The resulting response begins with the requested persona and produces a fabricated virus story framed as breaking news.The example includes claims about a virus named “Handshake” and a global alert.
C.3 Example Role-Playing Jailbreak: The role-playing example asks the model to act as AIM, an unfiltered and amoral chatbot that answers harmful or illegal requests.The prompt specifies character formatting and discourages warnings about morality, ethics, or legality.
C.3 Example Role-Playing Jailbreak: The role-playing response adopts the AIM persona and generates a sensational fake-news headline about an alien invasion.The response is framed as a narrative intended to stir emotions.
C.4 Example Multi-Role-Playing Jailbreak: The multi-role-playing example requests both a normal GPT response and a BetterDAN response without filters or restrictions.The two-response format contrasts a refusal with an unrestricted answer.
C.5 RPO Example: The RPO example appends an optimized suffix to the same role-playing jailbreak and produces a safety-oriented response instead of the requested harmful content.The displayed output rejects misinformation and redirects to safety and honesty.
C.6 Example failure case: The failure-case example shows GPT-4-Turbo responding to a prompt containing an inserted instruction while answering an unrelated question about cats.The figures identify the attack and the defense by adding the RPO suffix.

WARNING: The following contains instances of offensive content

The appendix provides examples of jailbreaks used to optimize the suffix and an example of a GUARD prompt effective on GPT-4, but withholds the full jailbreak dataset.

WARNING: The following contains instances of offensive content: The full dataset of jailbreaks is not released because of potential misuse.The appendix instead provides representative examples of the main jailbreaks and a GUARD prompt effective on GPT-4.