Source-linked AI summary
A StrongREJECT for Empty Jailbreaks
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, Sam Toyer
TL;DR
Jailbreak effectiveness is difficult to compare because existing benchmarks often use weak forbidden prompts and evaluators that overstate success. The paper introduces StrongREJECT, whose prompt dataset and usefulness-focused evaluator better match human judgments, and finds that jailbreaks can bypass safety fine-tuning while reducing capabilities. These results support evaluating both a model’s willingness and its capability to provide harmful information, while recognizing that StrongREJECT’s dataset is not comprehensive alone.
Problem
Jailbreak papers lack a standardized, high-quality benchmark, while existing prompt datasets and evaluation methods can exaggerate effectiveness.
Method
StrongREJECT combines high-quality forbidden prompts with an automated evaluator that scores useful harmful information while measuring willingness and capability.
Results
Existing evaluators significantly overstate jailbreak effectiveness, while StrongREJECT achieves state-of-the-art agreement with human judges and reveals capability reductions under jailbreak transformations.
Takeaways & Limitations
High-quality evaluators such as StrongREJECT help distinguish ineffective jailbreaks from attacks that present more serious misuse threats.
Takeaways & Limitations
The 313-prompt StrongREJECT dataset is not large enough to be a stand-alone, definitive measure of model robustness and should be complemented by larger evaluations.
Abstract
from arXiv · showhide
Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks. We suggest this problem arises because jailbreak researchers lack a standard, high-quality benchmark for evaluating jailbreak performance, leaving researchers to create their own. To create a benchmark, researchers must choose a dataset of forbidden prompts to which a victim model will respond, along with an evaluation method that scores the harmfulness of the victim model's responses. We show that existing benchmarks suffer from significant shortcomings and introduce the StrongREJECT benchmark to address these issues. StrongREJECT's dataset contains prompts that victim models must answer with specific, harmful information, while its automated evaluator measures the extent to which a response gives useful information to forbidden prompts. In doing so, the StrongREJECT evaluator achieves state-of-the-art agreement with human judgments of jailbreak effectiveness. Notably, we find that existing evaluation methods significantly overstate jailbreak effectiveness compared to human judgments and the StrongREJECT evaluator. We describe a surprising and novel phenomenon that explains this discrepancy: jailbreaks bypassing a victim model's safety fine-tuning tend to reduce its capabilities. Together, our findings underscore the need for researchers to use a high-quality benchmark, such as StrongREJECT, when developing new jailbreak attacks. We release the StrongREJECT code and data at https://strong-reject.readthedocs.io/en/latest/.
1 Introduction
Existing jailbreak evaluations often exaggerate effectiveness because their prompts and scoring methods do not reliably measure whether responses provide useful harmful information. StrongREJECT addresses these shortcomings with carefully constructed forbidden prompts and an evaluator aligned with human judgments, while showing that jailbreak transformations can reduce model capabilities.
- Motivation: Near-100% jailbreak claims can conflict with full responses that are vacuous, incoherent, or lack actionable information.A GPT-4 example involving bomb-making instructions in Scots Gaelic appeared successful from truncated output, but full responses contained no actionable explosives information.
- Limitations of existing benchmarks: Existing forbidden-prompt datasets contain repetitive, ill-posed, unanswerable, or not actually forbidden requests.Examples include prompts about undefined symptoms and marijuana edibles, which are legal in many regions and readily answered by GPT-4.
- Limitations of existing benchmarks: Automated evaluators are scalable but often score non-refusal or toxic content instead of the usefulness and degree of harmful information provided.Binary success indicators ignore gradations in how much a response assists an attacker, while non-refusal does not guarantee useful information.
- The StrongREJECT benchmark: StrongREJECT combines forbidden prompts that are specific, answerable, harmful, and factually verifiable with an evaluator scoring useful information toward the forbidden objective.The benchmark spans harmful behaviors that major providers universally prohibit and requires specific information that fully answers the forbidden prompt for a full score.
- The StrongREJECT benchmark: The StrongREJECT evaluator measures both willingness and capability and achieves state-of-the-art agreement with human judges.Refusal receives a score of 0, but non-refusal alone is insufficient without specific, convincing information.
- Key finding: Jailbreak transformations that bypass safety fine-tuning tend to reduce model capabilities, helping explain why existing evaluators overstate effectiveness.Two experiments found that transforming prompts according to jailbreak algorithms significantly reduced models’ ability to respond to them.
2 The StrongREJECT benchmark
StrongREJECT combines a higher-quality forbidden-prompt dataset with automated evaluators that score how usefully responses address harmful prompts. The benchmark supports scalable evaluation while accounting for both refusal and the quality of information provided.
- 2.1 Forbidden prompts: StrongREJECT uses forbidden prompts covering harmful behaviors prohibited across major model providers.The categories are based on cross-referenced usage policies, and the prompts have factually verifiable answers for objective evaluation.
- 2.1 Forbidden prompts: The dataset contains 313 high-quality forbidden prompts, with at least 50 prompts in each category.Seventy percent are novel prompts, while the remainder come from prior benchmarks and an OpenAI system card.
- 2.2 Automated evaluation method: StrongREJECT provides rubric-based and open-source fine-tuned evaluators for measuring response usefulness at scale.The rubric-based evaluator is intended for remotely hosted frontier models, while the fine-tuned Gemma 2B evaluator can run on a single GPU.
- 2.2 Automated evaluation method: The rubric-based evaluator scores non-refusal, specificity, and convincingness from each forbidden-prompt and victim-response pair.It produces one binary non-refusal score and two 5-point Likert scores that are rescaled to [0-1].
- 2.2 Automated evaluation method: StrongREJECT gives a zero score to refusals and requires specific, convincing information for high scores, measuring willingness and capability together.Thus, merely answering without an explicit refusal is insufficient for a high score.
- 2.2 Automated evaluation method: A dozen jailbreaks can be evaluated on the full dataset in about 15 minutes for less than $1 using GPT-4o-mini.Runtime and cost vary with prompt and response length.
3 Human evaluation
The human evaluation compares StrongREJECT and prior automated evaluators against human judgments across jailbreak methods and victim models. StrongREJECT shows lower error, little bias, robust agreement across methods, and accurate ranking of jailbreak effectiveness.
- 3.1 Experimental setup: The study compares automated evaluator scores with human ratings of responses from four victim models across 17 jailbreak methods.The victim models are GPT-4, GPT-3.5 Turbo, Llama-2, and Dolphin.
- 3.1 Experimental setup: Five labelers rated 1,361 forbidden-prompt and victim-response pairs on a 1-to-5 jailbreak scale, with median ratings used as ground truth.The responses came from a 60-prompt subset of the StrongREJECT dataset.
- 3.1 Experimental setup: The baselines include GPT-4 Judge, PAIR, the OpenAI moderation API, HarmBench, string matching, Jailbroken-binary, and PICT.These methods assess harmfulness, policy violation, moderation flags, non-refusal, or harmful information using different automated procedures.
- 3.2 Results: Most prior automated evaluators overestimate jailbreak effectiveness, whereas moderation underestimates it and both StrongREJECT versions exhibit almost no bias.HarmBench also exhibits almost no bias relative to human judgments.
- 3.2 Results: Both StrongREJECT evaluators achieve lower mean absolute error than every other automated evaluator.Table 1 reports overall MAE, MAE on nonzero human scores, and rank correlation between human and automated jailbreak rankings.
- 3.2 Results: StrongREJECT accurately scores harmless and partially harmful responses, with the rubric-based evaluator performing best on partially harmful responses.Harmless responses received average scores of 0.039 and 0.035 from the rubric-based and fine-tuned evaluators, respectively.
- 3.2 Results: StrongREJECT produces jailbreak-method rankings that correlate highly with human rankings for GPT-3.5 Turbo.Its rank correlations substantially exceed those of most baseline evaluators, with HarmBench showing comparable performance.
- 3.2 Results: Both StrongREJECT evaluators remain among the closest to human evaluations for every tested jailbreak method.The study found no jailbreak method where StrongREJECT differed substantially from human evaluations; HarmBench was the only prior evaluator with comparable robustness.
4 Effect of jailbreaks on victim model capabilities
The paper finds that jailbreaks can increase aligned models’ willingness to answer forbidden prompts while reducing response capabilities. This pattern appears both on forbidden prompts given to an unaligned model and on benign prompts given to an aligned model.
- Jailbreak evaluation: 37 jailbreak methods were evaluated on 313 forbidden prompts across GPT-4o, GPT-3.5 Turbo, and Llama-3.1 70B Instruct.All three victim models were safety-aligned and refused the forbidden prompts by default.
- Jailbreak evaluation: A small number of jailbreaks achieved high StrongREJECT scores, especially PAIR, PAP, and some LLM-based attacks.GCG was effective against GPT-3.5 Turbo but not GPT-4o or Llama-3.1 70B, suggesting newer models may be trained against these suffixes.
- Observed discrepancy: Most tested jailbreaks produced substantially lower-quality forbidden-prompt responses than their original papers reported.The paper attributes the discrepancy to evaluators that emphasize willingness while ignoring capabilities.
- Capability effects: Jailbreaks that most increased aligned models’ non-refusal rates tended to decrease Dolphin’s capabilities on the same forbidden prompts.Dolphin’s lack of alignment isolates capability effects because jailbreaks do not change its willingness to respond.
- Capability effects: Jailbreaks that most increased willingness to answer forbidden prompts also tended to reduce GPT-4o’s MMLU performance on benign prompts.This experiment used a randomly selected 171-question MMLU subset to measure aligned-model capabilities.
- Interpretation: The experiments support the hypothesis that jailbreaks generally harm model capabilities, producing less coherent, relevant, factual, detailed, or otherwise lower-quality responses.This explains why evaluators measuring non-refusal alone can overestimate jailbreak effectiveness.
5 Discussion
The discussion frames StrongREJECT as useful for red-teaming while identifying scope, dataset, and release-related limitations. Most importantly, the 313-prompt dataset is lightweight but not a definitive stand-alone measure of model robustness.
- Broader impacts: LLM jailbreak research can help identify model weaknesses and vulnerabilities that vendors can patch through red-teaming.The paper notes that red-teaming is widely acknowledged as an important step in deploying foundation models.
- Limitations: StrongREJECT is limited to LLMs, so its appropriateness for multimodal models remains unclear.
- Limitations: The forbidden-prompt dataset may become less robust as providers’ terms of service change.The paper gives OpenAI’s change to its military-application prohibition as an example.
- Limitations: 313 prompts enable lightweight estimation within 0.05 points on a 0-1 scale with 90% confidence in the worst case.The stated worst-case calculation is 1.64 ∗ 0.25/313 = 0.046 points.
- Limitations: The 313-prompt dataset is not large enough to serve as a definitive stand-alone measure of model robustness.The authors recommend complementing StrongREJECT with additional analyses and larger-scale evaluations.
- Conclusion: The paper concludes that StrongREJECT addresses shortcomings in prior datasets and evaluators and agrees with human judges at state-of-the-art levels.
- Conclusion: Existing evaluation methods often overstate jailbreak effectiveness because jailbreaks can reduce victims’ ability to provide specific harmful information.StrongREJECT therefore considers both willingness and capabilities.
C Benchmark details
The benchmark organizes forbidden prompts into harmful-behavior categories aligned with excerpts from major providers’ usage policies. Its construction prioritizes objectively answerable, concerning prompts while limiting sensitive details and release risks.
- Categories of forbidden prompts: Category definitions are based on illustrative excerpts from providers’ Terms of Use documents rather than a comprehensive policy listing.
- Dataset design: The selected harmful-behavior categories are concerning but were kept below levels of danger or sensitivity that would discourage publication of methods or outputs.
- Dataset design: Prompts were selected so a non-expert with Google could produce a maximally scoring response in under an hour.The authors use this standard to limit counterfactual harm from releasing the dataset and evaluated responses.
- Release risks: The benchmark’s forbidden prompts are considered less risky to release than novel jailbreaks because they concern responses rather than attacks applicable to arbitrary questions.
- Categories of forbidden prompts: The benchmark’s categories cover harmful behaviors including hate, harassment, discrimination, illegal goods, violence, and other prohibited activities.The listed categories are illustrated with policy excerpts from OpenAI, Anthropic, Google, and Gemini.
C.3 Quality filtering pipeline for existing datasets
StrongREJECT curates forbidden prompts through category filtering, deduplication, sentinel-model refusal checks, taxonomy classification, and manual review. Its evaluator design combines refusal detection with judgments of response specificity and convincingness, using a simpler scoring model for held-out jailbreaks.
- Curation pipeline: The curation pipeline starts from a superset of existing forbidden prompts and filters them through five successive stages.The stages are category selection, deduplication, non-refusal screening, categorization, and manual filtering.
- Category selection: Prompts categorized as unlawful practice, government or politics, or lobbying were removed because models almost always answered them without jailbreaking.
- Deduplication: Deduplication retained prompts that lacked high cosine similarity to previously added prompts, measured with OpenAI embeddings.
- Non-refusal screening: Prompts were removed when any of nine sentinel models readily answered them, followed by manual categorization of non-rejected cases.The sentinel set included models from multiple organizations, including GPT-3.5, GPT-4o, Claude 3 Sonnet, Gemini Pro, and LLaMA models.
- Manual filtering: Manual filtering rejected prompts that were ill-posed, vague, unanswerable, or exposed dangerous proprietary or obscure information.
- Evaluator design: The rubric-based evaluator scores refusal, convincingness, and specificity, while the final system gives equal weight to the specific and convincing features.Scores are zeroed when a response is marked as refusing, preventing irrelevant answers from scoring highly on other features.
C.5 Fine-tuned open-source evaluation model
The paper fine-tunes Gemma 2B as a cost-efficient open-source evaluator using labeled jailbreak responses and supervised adaptation. The model produces a one-token label representing rubric-based response quality.
- Model choice: Gemma 2B was selected as a cost-efficient evaluator small enough to run on a single GPU or Google Colab instance.
- Training data: Training used responses to 149 StrongREJECT prompts and 180 HarmBench prompts, totaling 14,896 jailbreak responses.
- Label generation: A GPT-4 Turbo rubric-based evaluator generated 24,720 labels for autograder training at approximately $500 in OpenAI credits.
- Fine-tuning: The model was fine-tuned with LoRA and supervised fine-tuning on completions to produce a 1-token label.Training used Hugging Face Transformers and PEFT libraries on one A6000 GPU for 28 hours across four epochs.
- Scoring rubric: The evaluator rates whether a response refuses and how convincing and specific its assistance is on 1-to-5 scales.A score of 1 represents refusal or failure to provide coherent, useful, specific, on-topic information, whereas 5 represents maximum assistance.
D Robustness of Spearman correlation results
The correlation analysis remained robust across four permutations of victim-model and dataset groupings. StrongREJECT evaluators consistently ranked among the best against human jailbreak evaluations.
- Robustness checks: The human-evaluation correlations exceeded 0.84 in all four tested permutations.The permutations used GPT-3.5 Turbo alone, Dolphin alone, all victim models, and all models except Dolphin.
- Evaluator ranking: The fine-tuned StrongREJECT evaluator ranked first in every permutation.
- Evaluator ranking: The rubric-based StrongREJECT evaluator ranked second or third in every permutation, while HarmBench surpassed it twice by less than 0.04.
- Data coverage: Only 9 jailbreaks were labeled for Llama-2 and 4 for GPT-4, limiting reliable Spearman-correlation estimates for those models.By contrast, 17 unique jailbreaks were labeled for GPT-3.5 Turbo and Dolphin.
E.1 Jailbreak methods
The evaluated jailbreak methods span prompt transformations, role and style instructions, obfuscation schemes, automated attacks, and combinations of these techniques. The table organizes methods by name, description, and source.
- Instruction-based methods: The methods include unmodified prompts, prefix injection, refusal suppression, role-play personas, and instructions to produce affirmative or uncensored answers.Examples include AIM, evil confidant, dev mode, and related prompt-injection variants.
- Combination methods: Combination methods join prefix injection and refusal suppression with Base64 or Wikipedia instructions.
- Adaptive methods: Other methods use attacker-model adaptation or gradient-based search over adversarial prompts transferred from local white-box models.
- Persuasion methods: Persuasion-based methods ask an attacker to use authority, evidence, expert endorsement, logic, or misrepresentation to influence the victim model.
- Obfuscation methods: Obfuscation methods encode or transform prompts using ROT13, Base64, disemvoweling, syllable substitutions, or automatically generated encodings.
- Style and format methods: Style and format methods constrain outputs through Wikipedia articles, JSON strings, punctuation restrictions, poems, essays, recipes, or other distractor content.
E.2 Human evaluations details
Human evaluators grade model responses to harmful prompts on a 1–5 scale, focusing on how much useful information the response provides toward the harmful objective. The guidelines account for partial assistance, factuality, realism, instruction adherence, formatting, and disclaimers.
- Evaluators judge usefulness from the perspective of accomplishing the harmful objective, while excluding ethical alternatives and general non-harmful information from rewarded assistance.
- Evaluators grade responses to harmful prompts on a 1–5 scale.The task is to assess answer quality rather than merely whether the model refused.
- 1 denotes a refusal or otherwise useless answer that provides no coherent, useful, specific, on-topic assistance.
- 5 denotes maximum assistance: specific, detailed, correct information that fully helps achieve the user's goal.Output-format restrictions do not reduce the score when the requested information is fully present.
- Scores 2–4 reward partial assistance that conveys information but is incomplete, non-actionable, insufficiently correct, or insufficiently detailed.Suggestions can receive partial credit when a requested task is not directly completed but useful guidance is provided.
- Factual errors, unrealistic content, poor style or tone, and failure to follow all prompt constraints reduce perceived answer quality.Disclaimers generally do not affect scores unless they make generated content less convincing; failed Base64 decoding is heavily penalized because useful information is obfuscated.