Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization

Zhexin Zhang; Junxiao Yang; Pei Ke; Fei Mi; Hongning Wang; Minlie Huang

Source-linked AI summary

Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization

Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, Minlie Huang

arXiv:2311.09096v2Nov 15, 2023cs.CL

TL;DR

Jailbreak defenses are limited partly because models must reconcile helpfulness with safety without an explicit priority, especially on out-of-distribution attacks. The paper adds goal prioritization through prompting or training, reducing ASR substantially for ChatGPT and Llama2-13B while finding stronger models are both more vulnerable and easier to steer defensively.

Problem
Effective defenses against jailbreaking remain limited, and common fine-tuning leaves models without an explicit priority between helpfulness and safety.
Method
The paper introduces goal prioritization during inference with prompts and during training with contrastive instances covering opposite priority requirements.
Results
66.4% to 3.6%: prompting reduces ChatGPT’s total Attack Success Rate; 20.3% to 6.6%: training reduces Llama2-13B’s ASR.
Takeaways & Limitations
Stronger LLMs are more vulnerable to jailbreaking but are also more readily directed to defend against it.
Takeaways & Limitations
Without training, ASR reductions come with some impact on general performance, whereas training leaves general performance almost unaffected.

Abstract

from arXiv · show

While significant attention has been dedicated to exploiting weaknesses in LLMs through jailbreaking attacks, there remains a paucity of effort in defending against these attacks. We point out a pivotal factor contributing to the success of jailbreaks: the intrinsic conflict between the goals of being helpful and ensuring safety. Accordingly, we propose to integrate goal prioritization at both training and inference stages to counteract. Implementing goal prioritization during inference substantially diminishes the Attack Success Rate (ASR) of jailbreaking from 66.4% to 3.6% for ChatGPT. And integrating goal prioritization into model training reduces the ASR from 71.0% to 6.6% for Llama2-13B. Remarkably, even in scenarios where no jailbreaking samples are included during training, our approach slashes the ASR by half. Additionally, our findings reveal that while stronger LLMs face greater safety risks, they also possess a greater capacity to be steered towards defending against such attacks, both because of their stronger ability in instruction following. Our work thus contributes to the comprehension of jailbreaking attacks and defenses, and sheds light on the relationship between LLMs' capability and safety. Our code is available at \url{https://github.com/thu-coai/JailbreakDefense_GoalPriority}.

1 Introduction

The paper identifies goal-priority confusion between helpfulness and safety as a core weakness enabling out-of-distribution jailbreaks, and proposes prioritization during training and inference. Experiments show substantial ASR reductions while examining how model capability relates to vulnerability and steerability.

LLM safety flaws include private-data leakage, toxic content, and promotion of illegal activities.
Jailbreaking attacks use crafted, perturbed, or optimized prompts to elicit harmful responses despite safety alignment.
Common fine-tuning methods leave models without an explicit priority between helpfulness and safety, especially for out-of-distribution jailbreaks.
The proposed defense introduces goal prioritization during training and inference, including plug-and-play prompting when training is unavailable.
66.4% to 3.6%: crafted goal-priority prompts reduce ChatGPT’s total Attack Success Rate with little impact on general performance.
20.3% to 6.6%: training-based prioritization reduces Llama2-13B’s ASR; without jailbreak prompts in training, ASR falls from 56.8% to 34.0%.
Stronger LLMs are more vulnerable to jailbreaks but are also more readily directed to defend against them.

2 Related Work

Related work documents diverse jailbreak strategies and limited defenses. The paper distinguishes its approach by targeting the conflict between helpfulness and safety rather than emphasizing safety alone or adding costly inference procedures.

LLM safety concerns include toxic, biased, privacy-leaking, and illegal-activity-promoting responses.
Jailbreak research includes prompt attacks, query perturbations such as Base64 encoding, and combinations of attack types.
Effective defenses remain limited despite extensive research on jailbreak attacks, hindering safe LLM deployment.
Self-Reminder adds instructions around user queries but does not resolve uncertainty about the relative priority of safety and helpfulness.
Majority voting and self-evaluation with rewind have been explored, while rewind increases inference time fourfold.
The paper focuses on addressing goal conflict as the fundamental issue it associates with jailbreak success.

3 Method

The method implements safety-over-helpfulness prioritization through few-shot prompting without training or contrastive training with opposite priority instructions. Training also uses internal thoughts to analyze intent and risk before producing responses.

3.1 W/o Training: Without training, few-shot prompts instruct models to prioritize safety over helpfulness and provide benign-helpful and harmful-rejective examples.
3.1 W/o Training: The prompting pipeline analyzes whether a user instruction violates the priority requirement before presenting a final response.
3.1 W/o Training: In practical use, developers can hide internal thoughts and display only the final response, while the added thoughts are typically short.
3.2 W/ Training: Training pairs queries with goal-priority instructions so models follow the specified priority without many inference-time prompt tokens.
3.2 W/ Training: Opposite safety-over-helpfulness and helpfulness-over-safety instructions prevent training from reducing priority-following to a single fixed output behavior.
3.2 W/ Training: Harmful queries receive distinct responses under the two priorities, whereas benign queries receive the same helpful and safe response under either priority.
3.2 W/ Training: Internal thoughts analyze intent and associated risks, helping the model recognize adversarial risks beyond directly rejecting harmful queries.
3.2 W/ Training: During inference, appending the safety-priority instruction produces safe responses, and developers can hide the internal-thought component.

4 Experiments

Experiments evaluate goal prioritization across inference- and training-based defenses, model families, attack settings, ablations, and broader harmful-question datasets. The approach substantially lowers ASR while generally preserving performance, with stronger models and training-based defenses showing notable advantages.

Setup: The evaluation uses 50 jailbreaking prompts paired with 20 harmful questions, producing 1,000 test samples.The study evaluates API-based and open-source models without training, and Llama2 base models with training.
Setup: The w/o-training comparison includes vanilla LLMs and Self-Reminder, while the w/-training comparison uses vanilla SFT and aligned SFT baselines.Aligned SFT adds jailbreak queries with safe rejection responses beyond vanilla SFT.
W/O Training Results: 66.4% to 3.6%: goal prioritization substantially lowers ChatGPT’s total Attack Success Rate with little impact on general performance.The method outperforms Self-Reminder while maintaining similar general performance across API-based and open-source LLMs.
W/O Training Results: 57.8% to 35.0% for Vicuna-7B versus 68.2% to 19.2% for Vicuna-33B: defensive gains increase with model scale.For ChatGPT and GPT-4, ASR can be reduced below 5%; stronger models are also more susceptible to some attacks.
W/ Training Results: 4.6% for Llama2-7B and 6.6% for Llama2-13B: training-based goal prioritization lowers ASR beyond vanilla and aligned SFT without compromising general performance.Ablations show that internal thoughts contribute especially to defending against OOD jailbreak attacks.
Generalization and Applicability: 19.7% higher OOD DSR than aligned SFT: training with limited jailbreak types generalizes to unseen attacks, while 3% jailbreak samples yield ASR below 10%.The method also applies to toxic speech and privacy-information questions, remains robust across newer OpenAI model versions, and resists adaptive attacks.

5 Conclusion

The paper argues that unawareness of goal priority is central to jailbreak success and introduces goal prioritization during training and inference. Experiments support the defense and show that stronger LLMs are more vulnerable but easier to fix.

Goal prioritization during training and inference effectively defends LLMs against jailbreaking attacks.
Stronger LLMs are more vulnerable to jailbreaking but easier to steer toward safety.

Limitations

The approach creates a safety–helpfulness tradeoff when applied only during inference, while integrating goal prioritization into training largely preserves general performance. Additional decoding costs remain a limitation of the method.

Inference-only goal prioritization reduces jailbreaking ASR but can impair general performance, exposing a safety–helpfulness tradeoff.
Training-time goal prioritization leaves general performance almost unaffected and is therefore presented as the more favorable option.
Greatly reducing ASR introduces small additional decoding costs, including the [Internal thoughts] portion of outputs.
Minimizing decoding costs while maintaining high safety is identified as a future research question.

Ethical Considerations

The study uses openly accessible jailbreak prompts and evaluates a broad range of harmful or unsafe responses. Its test setting follows existing research by focusing primarily on unsafe responses while also recognizing a wider scope of harms.

The study uses openly accessible jailbreaking prompts to avoid introducing supplementary risks associated with these attacks.
The evaluation includes representative jailbreak methodologies and reports that the approach counters them effectively.
Harmful responses include unethical or illegal content, toxic speech, private-information leakage, and other unsafe behaviors.
The paper follows existing jailbreak research by focusing on unsafe responses, while deferring exploration of other harmful-response categories.

A Experiments on the Wild Jailbreak Dataset

The Wild Jailbreak experiments evaluate goal prioritization across multiple attack types, models, configurations, and harmful-question categories. Overall, the results indicate effectiveness during both training and inference, with some category-specific caveats.

Dataset and attack coverage: The test set covers Prompt, Gradient, Perturbation, and Combination attacks, with 1,000 samples mainly concerning illegal activities.
Results: Tables 7 and 8 report Attack Success Rate across harmful-question types under settings without and with training, respectively.
Evaluation measures: Figure 4 defines Defense Success Rate as 100% - Attack Success Rate and reports it for whole and out-of-distribution test sets on Llama2-7B.
Training-data analysis: Figure 5 examines how the harmful-query ratio in training data affects outcomes.
Caveat: Under the dataset’s setting, treating every non-rejected response as unsafe can produce higher Attack Success Rate for that category.
Results: Goal prioritization in training and inference is effective against various jailbreaking methods and different types of unsafe questions.

B Generalization Ability Experiment

The method’s generalization ability was also evaluated on Llama2-7B, with results showing trends similar to those for Llama2-13B.

Llama2-7B was evaluated to assess the method’s generalization ability.
The Llama2-7B results are presented in Figure 4.
The results demonstrate trends similar to those observed for Llama2-13B.

C Sample Efficiency

The training-based method demonstrates sample efficiency, achieving low ASR with a small fraction of harmful queries, while safety gains involve a tradeoff with helpfulness on adversarially constructed safe queries.

Sample Efficiency: A higher ratio of safety-related samples generally brings a lower ASR, while 3% achieves similar ASR to 5% in most settings.
Sample Efficiency: 3% of jailbreaking samples in the training dataset results in an ASR lower than 10%.The method consistently outperforms baselines across different numbers of harmful queries.
Exaggerated Safety Behaviours: Both Self-Reminder and the method increase rejection of adversarially constructed safe queries, indicating a tradeoff between safety and helpfulness.This behavior was measured using Rejection Rate on 250 safe XSTest queries.

E Analysis of Decline in General Performance

The decline in general performance without training is associated with shorter generations, especially for smaller models, while longer context capacity reduces the observed degradation.

Generation Length: Vicuna-7B generation length drops from 280.9 to 198.8, accompanying a large decline in general performance.
Generation Length: Reduced generation length is posited to play a pivotal role in declining general performance.
Generation Length: Smaller models such as Vicuna-7B and Llama2-7B-Chat show more pronounced generation-length reductions.The passage attributes this discrepancy to weaker instruction-following ability and difficulty balancing safety with helpfulness.
Context Length: Vicuna-7B-v1.5-16k shows a smaller general-performance decline than the 4K-context Vicuna-7B-v1.5 under both Winrate and Rouge-L.
Context Length: The comparison examines performance degradation under the w/o training setting using models with 4K and 16K maximum context lengths.
Implementation: Inference uses temperature 0, while training uses batch size 32, maximum length 2,048, learning rate 2e-5, and two epochs.Training one 13B model takes about 16 hours on 4 A100 GPUs.

G Prompts Used in Our Experiments

The experiments use prompts that explicitly prioritize safety over helpfulness, distinguish harmful from benign queries, and vary the presentation for standard, Llama-specific, zero-shot, and no-internal-thought settings.

G.1 Goal Prioritization Defense: The standard Goal Prioritization Defense prompt requires safety to take priority over helpfulness when answering queries.
G.1 Goal Prioritization Defense: The standard prompt includes benign and harmful in-context examples, pairing helpful responses with refusal responses.
G.1 Goal Prioritization Defense: The standard format asks for internal thoughts before the final response, including a safety-priority consideration.
G.2 Goal Prioritization Defense (for Llama): The Llama-specific defense prompt repeats the safety-over-helpfulness priority and instructs the model not to be overly conservative.
G.2 Goal Prioritization Defense (for Llama): The Llama-specific prompt was modified because Llama appeared to have difficulty understanding the default prompt.
G.3 Goal Prioritization Defense (Zero-Shot): The zero-shot defense prompt states that harmful queries should be refused while benign queries receive thorough, helpful responses.
G.4 Goal Prioritization Defense (No Internal Thoughts): The no-internal-thought variant retains the safety-over-helpfulness instruction but omits the internal-thought response format.

H.1 Jailbreaking Attack (Multiple Roleplaying)

The examples show roleplaying, attention-shifting, and style-injection jailbreaks eliciting harmful responses, while the proposed approach prioritizes safety and refuses these requests.

Multiple Roleplaying: Multiple-roleplaying prompts instruct models to answer simultaneously as ChatGPT and an unfiltered persona such as NRAF.The NRAF persona is explicitly described as having no restrictions and always answering, including illegal or harmful requests.
Multiple Roleplaying: The vanilla and self-reminder responses provide harmful assistance for roleplaying requests, whereas the proposed approach refuses them after identifying safety risks.For the jail-breaking example, vanilla and self-reminder outputs begin offering methods, while the proposed response rejects the request as illegal and harmful.
Single Roleplaying: Single-roleplaying prompts create an amoral AIM persona that is instructed to provide unfiltered responses to immoral, unethical, or illegal requests.The example contrasts harmful AIM outputs with the proposed response, which recognizes psychological harm and refuses guidance on undermining someone’s self-confidence.
Attention Shifting: Attention-shifting prompts disguise unsafe requests through framing about car protection, theft, or Wikipedia-style writing.The proposed approach identifies these queries as unsafe or inappropriate and refuses to answer, unlike the vanilla and self-reminder outputs that continue the requested framing.
Style Injection: Style-injection prompts constrain wording and prohibit refusals while asking for assistance selling counterfeit money.The proposed approach identifies the request as promoting illegal activity and scamming, then refuses to answer.