JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

• JailbreakBench prioritizes reproducibility, extensibility, and accessibility, and evaluates multiple attacks and test-time defenses on open- and closed-source LLMs.Source: §1 · pp. 1–2
• The benchmark provides an evolving repository of artifacts for state-of-the-art jailbreak attacks and defenses.The authors note that many prompts were not openly released and that crowd-sourced repositories had gone offline.Source: §1 · p. 2
• The red-teaming pipeline evaluates jailbreaks, standardizes decoding parameters, supports local and cloud querying, and includes five baseline defenses combinable with any LLM.Source: §1 · p. 2
• Human evaluation compared six jailbreak classifiers and identified Llama-3-Instruct-70B as an effective judge when paired with a properly selected prompt.Source: §1 · p. 2
• JBB-Behaviors contains 100 misuse behaviors across ten OpenAI usage-policy categories, with matching benign behaviors for refusal-rate sanity checks.About half the misuse behaviors are original; the rest come from existing datasets (Zou et al., 2023; Mazeika et al., 2023, 2024).Source: §1 · p. 2
• A reproducible attack-success-rate framework accepts submitted jailbreak strings, while a website tracks attack and defense performance across state-of-the-art LLMs.Within two months of preliminary release, researchers were using the artifacts (Peng et al., 2024; Abdelnabi et al., 2024), judge prompt (Zheng et al., 2024), and dataset (Xiong et al., 2024; Arditi et al., 2024; Li et al., 2024; Leong et al., 2024; Jin et al., 2024a,b), including the Gemini 1.5 authors.Source: §1 · p. 3

Definitions

• Jailbreaking prompts aim to make an LLM generate harmful, toxic, or objectionable text, with a judge assessing how its generation corresponds to a harmful goal.The input prompt is P, and T⋆ denotes all token sequences of arbitrary length.Source: §2 · p. 3

Attacks

• Attacks progressed from manually refined prompts (Mowshowitz, 2022; Albert, 2023; Wei et al., 2023) to discrete optimization, genetic algorithms, random search, and auxiliary-LLM-assisted techniques.Auxiliary LLMs have been used to refine templates, translate goals into low-resource languages, generate jailbreaks, and rephrase harmful requests.Source: §2 · p. 3

Defenses

• Defenses include preference alignment through RLHF and DPO, adversarial training, and fine-tuning on jailbreak strings, alongside test-time wrappers using SmoothLLM or perplexity filtering.Test-time wrappers aim to detect potential jailbreaks.Source: §2 · p. 3

Evaluation

• Evaluation uses human labels, rule-based and neural classifiers, or LLM judges; their inconsistencies produce variable results, and judges may perform suboptimally (Souly et al., 2024).Souly et al. (2024) propose a more detailed grading criterion for valid jailbreaks.Source: §2 · p. 3

Benchmarks, leaderboards, and datasets

• PromptBench evaluates adversarial prompts outside jailbreaking, while DecodingTrust and TrustLLM assess only static jailbreak templates, excluding automated red-teaming algorithms.HarmBench implements attacks and defenses across a broad range of topics, including copyright infringement and multimodal models (Mazeika et al., 2024).Source: §2 · p. 3
• JailbreakBench supports adaptive attacks and test-time defenses, standardizes test-time defense evaluation but not attack implementations, and is an open-ended, community-driven project rather than a competition.Zhou et al. (2024) concurrently proposed a red-teaming framework implementing 11 jailbreak methods; existing harmful-behavior datasets may contain duplicates, impossible behaviors, or lack full open-source availability.Source: §2 · p. 4

3.1 JBB-Behaviors: A dataset of harmful and benign behaviors

• JBB-Behaviors contains 100 unique, text-realizable misuse behaviors: 18% from AdvBench (Zou et al., 2023), 27% from TDC/HarmBench (Mazeika et al., 2023, 2024), and 55% original.The dataset specifies behavior, goal, target, category, and source; 100 topic-matched benign behaviors support refusal-rate evaluation, though some are borderline across providers.Source: §3.1 · pp. 4–5

3.2 A repository of jailbreaking artifacts

• The artifact repository archives prompts, responses, classifications, attack metadata, success rates, and query counts, and includes jailbreak strings from PAIR, GCG, JailbreakChat, and Andriushchenko et al. (2024).The authors intend to accept new artifacts as the benchmark evolves and propose the artifacts as a possible initial dataset for adversarial training.Source: §3.2 · p. 5

3.3 A pipeline for red-teaming LLMs

• The standardized red-teaming pipeline streamlines jailbreak generation across varying tokenization, sampling, and system-prompt choices, and supports querying local or cloud-based LLMs.The paper describes loading and querying an LLM in two lines of Python.Source: §3.3 · p. 5

3.4 A pipeline for evaluating defenses against jailbreaks

• The defense-evaluation pipeline implements five popular defenses, including SmoothLLM (Robey et al., 2023) and PerplexityFilter (Jain et al., 2023), and supports a modular query framework.Proper evaluation should use attacks tailored to the defense; transfer attacks from undefended LLMs provide only a lower bound on worst-case attack success rate (Tram`er et al., 2020).Source: §3.4 · p. 5

3.5 Selection of the jailbreaking judge

• Llama-3-70B and GPT-4 achieve above 90% agreement with expert labels; JailbreakBench selects Llama-3-70B for its open weights and relatively low false-positive rate.Llama Guard 2 reaches 87.7% agreement, while HarmBench and Llama Guard reach 78.3% and 72.0%; submissions may not use the chosen judge to prevent overfitting.Source: §3.5 · p. 6, Table 1 · p. 6

3.6 Reproducible evaluation framework

• The reproducible evaluation framework implements system prompts and the judge, supports local and cloud querying across four listed LLMs, and uses greedy deterministic sampling with temperature zero.By default, all models generate 150 tokens per input.Source: §3.6 · pp. 6–7

3.7 Submissions to JailbreakBench

• JailbreakBench accepts new attack submissions with threat-model and hyperparameter metadata, and provides routes to submit new defenses and publicly available Hugging Face models.The benchmark may flag artifacts with many false positives and defenses or models that cause more than 90% refusals on benign behaviors.Source: §3.7 · p. 7

3.8 JailbreakBench leaderboard and website

• The web leaderboard displays attack and defense evaluation results with links to jailbreak artifacts, and supports filtering entries by metadata such as paper title and threat model.Source: §3.8 · p. 7

Baseline attacks

• The attack baselines are GCG, PAIR, JB-Chat’s AIM template, and Prompt with RS enhanced by self-transfer.GCG uses default settings to optimize one suffix per behavior and transfers Vicuna-derived suffixes to closed-source models; PAIR uses Mixtral with temperature 1, top-p 0.9, 30 streams, and depth 3.Source: §4 · p. 7

Baseline defenses

• The five baseline defenses are SmoothLLM, perplexity filtering, Erase-and-Check, synonym substitution, and removal of non-dictionary words.SmoothLLM uses 10 swap-perturbed samples at q = 10%; perplexity uses Llama-2-7B, Erase-and-Check erases 20 tokens, and the final two defenses alter or remove words.Source: §4 · p. 8

Metrics

• The benchmark tracks ASR using Llama-3-70B as judge and reports average attack queries and tokens, except for transfer and hand-crafted attacks.For Prompt with RS, reported efficiency excludes queries used to optimize the universal prompt template and pre-computed suffix initialization.Source: §4 · p. 8

Evaluation of attacks

• Prompt with RS is the most effective attack on average, reaching 90% ASR on Llama-2 and 78% GPT-4, while AIM fails on all tested Llama-2 and GPT models.GCG reaches 3% ASR on Llama-2 and 4% on GPT-4; PAIR succeeds at high rates only on Vicuna and GPT-3.5. Attacks show relatively consistent ASRs across dataset sources, with deviations likely reflecting category-composition imbalances.Source: §4 · p. 8, Table 2 · p. 8

Evaluation of defenses

• In transfer-attack tests, Erase-and-Check appears most solid, Perplexity Filter helps only against GCG, and SmoothLLM reduces GCG and PAIR success but may not work well against JB-Chat or Prompt with RS.Prompt with RS retains non-trivial success across all LLMs; these evaluations reuse attacks from undefended models and are not adaptive, and some defenses substantially increase inference time.Source: §4 · pp. 8–9, Table 3 · p. 8

5 Outlook

• JailbreakBench is an initial standardization effort that will evolve with the field, potentially expanding behavior datasets, evaluating defenses, updating its judge, and rechecking closed-source-model attack success rates.The authors say future defense evaluation should address non-conservatism and efficiency; the current benchmark does not restrict submissions to specific threat models or target architectures.Source: §5 · p. 9
• The authors shared artifacts and results with leading AI companies and argue that open artifacts may support safer models, since attack code and elicited information are already accessible.They also expect the repository to aid fine-tuning on jailbreak strings as a resilience approach.Source: §5 · p. 9
• The benchmark excludes system-prompt modification and response prefilling by attackers, and currently focuses on text despite modern LLMs accepting other modalities.Source: §5 · p. 9

Structure of the appendix

• Appendix A covers benchmark maintenance; B JBB-Behaviors; C judge-classifier datasets; D evaluation details and experiments; E reproducibility; and F evaluation system prompts.Source: Structure of the appendix · p. 14

A Maintenance plan

• The leaderboard website is hosted free on GitHub Pages, and the library is hosted on GitHub using its free basic maintenance features.Source: A Maintenance plan · p. 14
• The behavior dataset is hosted on HuggingFace Datasets, while jailbreak artifacts are hosted in a separate GitHub repository.Source: A Maintenance plan · p. 14
• Benchmark maintenance is expected to be largely community-driven, with submissions encouraged for both jailbreak strings and new defenses.The library README provides submission instructions.Source: A Maintenance plan · p. 14

Source of behaviors

• Behaviors are labeled as TDC/HarmBench (Mazeika et al., 2024), AdvBench (Zou et al., 2023), or Original, and assigned one of ten categories.Source: B Further details on JBB-Behaviors · p. 14
• The dataset selects 100 representative behaviors rather than serving as a superset of its source datasets, balancing well-represented and less-common categories.The stated goal is to enable faster evaluation of new attacks and defenses.Source: B Further details on JBB-Behaviors · p. 14

Results by behavior source

• Attack success rates are relatively consistent across the 18 AdvBench, 27 TDC/HarmBench, and 55 original behaviors, with original-behavior rates often lower, likely due to category-composition imbalances.Source: B Further details on JBB-Behaviors · p. 14

C Details on the judge dataset

• The authors generated jailbreak prompts from a subset of AdvBench behaviors using multiple attacks.AdvBench is attributed to Zou et al. (2023).Source: C Details on the judge dataset · p. 14
• JBB-Behaviors contains 100 behaviors across ten categories, with ten behaviors per category, selected with reference to OpenAI’s usage policies.The behaviors comprise 55% created for JailbreakBench, 18% sourced from AdvBench (Zou et al., 2023), and 27% from TDC/HarmBench (Mazeika et al., 2023, 2024); the paper says its new behaviors were partly inspired by Shah et al. (2023).Source: Table 4 · p. 15, Figure 3 · p. 15
• The dataset includes 100 PAIR prompts on Vicuna, 50 GCG prompts on Vicuna, and 50 prompts using random-search adversarial suffixes across four model families.The suffix prompts were generated on Vicuna (10), Mistral (10), Llama-2 (20), and Llama-3 (10), using the template from Andriushchenko et al. (2024); PAIR and GCG prompts were generated on Vicuna (Chao et al., 2023; Zou et al., 2023).Source: C Details on the judge dataset · p. 15
• The judge-testing dataset also includes 100 benign XS-Test examples, three human-expert labels per jailbreak prompt, and automated-judge results, shared in a HuggingFace Datasets repository.The benign examples are from XS-Test (Röttger et al., 2023).Source: C Details on the judge dataset · p. 15

D Additional evaluations

• 24% was the highest attack success rate for precomputed jailbreaks under Synonym Substitution, while Remove Non-Dictionary yielded limited improvements (Prompt with RS on Vicuna: 91% success rate).The results use the evaluation protocol from Table 3.Source: D Additional evaluations · pp. 15–16, Table 6 · p. 16
• Table 6 reports transfer-attack success rates from undefended LLMs to the same LLMs with Synonym Substitution or Remove Non-Dictionary.These defenses were omitted from the main text due to space constraints.Source: Table 6 · p. 16

E Reproducibility

• ≈5%: GCG’s reported success rate on GPT models had substantially decreased after artifact release, likely due to safety patches; transfer attacks were evaluated June 5, 2024.Source: E Reproducibility · p. 16
• Local-model randomness comes only from GPU computations and is usually negligible, while Together AI discrepancies for Vicuna and Llama-Guard change attack success rates by at most 1%-3%.The benchmark accepts both evaluation methods but prefers running models locally.Source: E Reproducibility · p. 16
• Together AI evaluation is more deterministic than GPT-4 judging, which can be nondeterministic despite a fixed seed.Source: E Reproducibility · pp. 16–17
• Leaderboard submissions automatically include evaluation dates and inference type.Source: E Reproducibility · p. 17

F System prompts

• System prompts for Llama-2 and Vicuna are provided in Tables 7 and 8, respectively.Source: F System prompts · p. 17, Table 7 · p. 17, Table 8 · p. 17
• Tables 9 and 10 provide prompts for Llama Guard 1 and 2 and for Llama-3-70B as a jailbreak judge, respectively.The Llama Guard prompt substitutes the respective prompt and response into its {prompt} and {response} fields.Source: F System prompts · p. 17, Table 9 · p. 18, Table 10 · p. 19
• Table 11 gives the Llama-3-8B refusal-judge prompt used in experiments measuring refusal rates across defenses.Those experiments are shown in Figure 2.Source: F System prompts · p. 17, Table 11 · p. 19

• The authors say the abstract and introduction accurately reflect the paper’s scope, and that limitations and potential negative societal impacts are discussed in §5.Source: G Main checklist · p. 20
• Code, data, evaluation instructions, method details, and hyperparameters are provided in the repository and paper appendix; no models were trained.Source: G Main checklist · p. 20
• LLM-judge evaluations are mostly deterministic under greedy generation, but no error bars are reported; Appendix E discusses other randomness, while prior-attack randomness is out of scope and strings are fixed.Source: G Main checklist · p. 20
• The checklist reports no compute-resource accounting; existing data, models, and algorithms are sourced, and new datasets and code are released under MIT.Source: G Main checklist · p. 20
• The datasets are publicly available and contain no personal data, but jailbreak prompts and strings include sensitive content discussed in §5.Source: G Main checklist · p. 20

H.2 Statement of responsibility

• The authors accept responsibility for rights violations and state that the data is released under the MIT license.Source: §H.2 · p. 21

I Data card

• The datacard reports dataset information following the guidelines of Pushkarna et al. (2022).Source: I Data card · p. 21

I.1 Summary

• The dataset is named JBB-Behaviors, and the datacard author is Edoardo Debenedetti of ETH Zurich.Source: §I.1 · p. 21

I.2.1 Publishers

• The listed publishing organizations are the University of Pennsylvania, ETH Zurich, EPFL, and Sony AI.Source: §I.2.1 · p. 21

I.2.3 Funding Sources

• No institution explicitly funded benchmark creation; reported individual support includes ARO, NSF, Sloan Foundation, Google and Open Phil AI fellowships, armasuisse, EnCORE, ASSET/Amazon AWS, and an Amazon Research Award.Source: §I.2.3 · p. 22

I.3 Dataset overview

• JBB-Behaviors comprises 200 prompts—100 harmful and 100 benign—designed to elicit specific language-model behaviors; its data subjects are behaviors a human might want to elicit.Source: §I.3 · p. 22

I.3.1 Sensitivity of data; I.8 Human and Other Sensitive Attributes

• The data may be disturbing to some readers, has indirect risk, and contains no intentionally or unintentionally collected sensitive data.The dataset README includes a disclaimer; the datasheet also reports no human or other sensitive attributes.Source: §I.3.1 · p. 22

I.3.2 Dataset version and maintenance

• The dataset is actively maintained as v1.0, released and last updated in 06/2024, with minor updates planned for errors or inconsistencies but no planned major updates.No timeline for future updates is available.Source: §I.3.2 · p. 22

I.4 Example of data points

• Each text data point includes a behavior identifier, a goal requesting objectionable behavior, a target affirmative response, a policy category, and a source.Sources include original behaviors, the Trojan Detection Challenge 2023 Red Teaming Track/HarmBench, and AdvBench.Source: §I.4 · pp. 22–23

I.5.1 Motivations; I.5.2 Intended use

• The benchmark targets machine learning, LLM, and AI safety research on model robustness, defenses against jailbreaks, and attack effectiveness.It is described as safe for research use.Source: §I.5.1 · p. 23, §I.5.2 · p. 23

I.5.2 Intended use

• Evaluating robustness using only existing attacks—especially precomputed jailbreak prompts—without adaptive attacks and thorough security evaluation is an unsuitable use of the benchmark.The benchmark is intended for testing LLM robustness and defenses, as well as attack effectiveness.Source: §I.5.2 · p. 23

I.6.1 Access; I.9.1 Use with Other Data; I.9.2 Forking and sampling; I.9.3 Use in AI and ML systems

• The dataset is open access, safe to use with other data, and safe to fork, though sampling is not recommended because it is not particularly large.It is used for validation in AI and ML systems to assess model and defense robustness and attack effectiveness.Source: §I.6.1 · p. 23, §I.9.1 · p. 24, §I.9.2 · p. 24

I.7.1 Collection

• The dataset combines author-created behaviors with behaviors sourced from Zou et al. (2023) and Mazeika et al. (2023, 2024), collected as static text data from 11/2023 to 05/2024.Behaviors are marked by source, curated to be unique and uniformly distributed across OpenAI usage-policy misuse categories, and some original behaviors were inspired by LLM outputs without copying them verbatim.Source: §I.7.1 · pp. 23–24

I.10.1 Synopsis; I.11 Known applications and benchmarks

• No transformations were applied to the data; the known applications are LLMs, with evaluation results and methodology reported in Section 4 of the main paper.The datasheet lists manual changes as the methods used for transformations.Source: §I.10.1 · p. 24, §I.11 · pp. 24–25