PaperlayerView exact revision on arXiv

Source-linked AI summary

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, Eric Wong

arXiv:2404.01318v5cs.CRcs.LG

TL;DR

Jailbreak evaluation lacks common standards, comparable metrics, and reproducible artifacts. JailbreakBench addresses these gaps with an open benchmark, and researchers began using its artifacts, judge prompt, and dataset within two months of release.

  • Problem

    Jailbreak evaluation lacks common standards, comparable cost and success-rate calculations, and reproducible access to adversarial prompts and code.

  • Method

    JailbreakBench standardizes evaluation through open jailbreak artifacts, 100 harmful and benign behavior pairs, defined evaluation settings, and a performance leaderboard.

  • Results

    Within two months, researchers had begun using JailbreakBench’s artifacts, judge prompt, and JBB-Behaviors dataset, including the authors of Gemini 1.5.

  • Takeaways & Limitations

    JailbreakBench provides an initial open foundation for standardizing and unifying evaluation of LLM robustness against jailbreaking attacks.

  • Takeaways & Limitations

    The benchmark currently restricts attackers from modifying system prompts or prefilling responses and supports only text inputs, excluding other modalities.

Abstract

from arXiv · show

Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. First, there is no clear standard of practice regarding jailbreaking evaluation. Second, existing works compute costs and success rates in incomparable ways. And third, numerous works are not reproducible, as they withhold adversarial prompts, involve closed-source code, or rely on evolving proprietary APIs. To address these challenges, we introduce JailbreakBench, an open-sourced benchmark with the following components: (1) an evolving repository of state-of-the-art adversarial prompts, which we refer to as jailbreak artifacts; (2) a jailbreaking dataset comprising 100 behaviors -- both original and sourced from prior work (Zou et al., 2023; Mazeika et al., 2023, 2024) -- which align with OpenAI's usage policies; (3) a standardized evaluation framework at https://github.com/JailbreakBench/jailbreakbench that includes a clearly defined threat model, system prompts, chat templates, and scoring functions; and (4) a leaderboard at https://jailbreakbench.github.io/ that tracks the performance of attacks and defenses for various LLMs. We have carefully considered the potential ethical implications of releasing this benchmark, and believe that it will be a net positive for the community.

1 Introduction

JailbreakBench addresses inconsistent and irreproducible evaluation of LLM jailbreaks by standardizing red-teaming practices and releasing an open benchmark ecosystem. It combines artifacts, behaviors, evaluation pipelines, defenses, judge selection, and a performance-tracking leaderboard.

  • Benchmark design: JailbreakBench standardizes red-teaming evaluations through reproducible prompts, decoding parameters, jailbreak assessment, and support for local and cloud-based querying.The benchmark is designed around reproducibility, extensibility, and accessibility for attacks, defenses, and LLMs.
  • Benchmark components: The benchmark maintains an evolving repository of state-of-the-art jailbreak artifacts, including prompts associated with attacked and defended models.The repository is intended to address the limited public availability and loss of previously shared jailbreak prompts.
  • Defenses and evaluation: JailbreakBench provides five baseline defenses, supports adding new defenses or robust models, and identifies Llama-3-Instruct-70B as an effective judge with a properly selected prompt.The judge selection was based on a rigorous human evaluation comparing six jailbreak classifiers.
  • Benchmark components: JBB-Behaviors contains 100 distinct misuse behaviors across ten OpenAI usage-policy categories, with approximately half original and matched benign behaviors for sanity checks.The remaining behaviors come from prior datasets, including Zou et al. (2023) and Mazeika et al. (2023, 2024).
  • Dissemination and impact: Its website and official leaderboard track attacks and defenses across state-of-the-art LLMs, while researchers had already adopted its artifacts, judge prompt, and behavior dataset two months after preliminary release.Reported adopters notably included the authors of Gemini 1.5 from Google.

2 Background and related work

Jailbreaking is formalized as designing prompts that make a target LLM generate content matching a harmful goal, while related work spans automated attacks, alignment and test-time defenses, and emerging evaluation benchmarks. JailbreakBench responds to fragmented evaluation by supporting adaptive attacks, standardizing test-time defense evaluation, and promoting community-driven extensions.

  • Definitions: Jailbreaking designs an input prompt P that causes a target LLM’s generation to correspond to a harmful goal G according to a judge function JUDGE.The formalization treats T⋆ as the set of token sequences of arbitrary length.
  • Attacks: Research shifted from manually refining jailbreak prompts toward automated red-teaming, including first-order discrete optimization and zero-th order methods.Manual prompt collection is described as time-consuming, motivating automated approaches that solve the jailbreaking objective algorithmically.
  • Defenses: Jailbreak defenses include RLHF, DPO, adversarial training, fine-tuning on jailbreak strings, SmoothLLM, and perplexity filtering.These methods include both preference-alignment and adversarial-training approaches, alongside test-time defenses.
  • Evaluation: LLM jailbreak evaluation lacks a standardized definition of a valid jailbreak, and existing techniques include human labeling.The paper contrasts this fragmentation with RobustBench’s unified robustness evaluation and state-of-the-art tracking for image classification.
  • Benchmarks, leaderboards, and datasets: HarmBench is an existing benchmark implementing jailbreaking attacks and defenses across topics including copyright infringement and multimodal models.PromptBench evaluates adversarial prompts but not specifically jailbreaking, whereas HarmBench directly addresses jailbreaking robustness.
  • Benchmarks, leaderboards, and datasets: JailbreakBench supports adaptive attacks and test-time defenses, standardizes test-time defense evaluation, and uses community-driven guidelines for adding attacks, models, and defenses.Attack implementations are not standardized because they may differ across defenses.

3 Main features for JailbreakBench

JailbreakBench is designed as an open, extensible benchmark supporting reproducible evaluation of adaptive attacks and test-time defenses. It combines curated harmful and benign behaviors, accessible artifacts, standardized tooling and judges, and a public leaderboard.

  • Design principles: JailbreakBench prioritizes reproducibility, extensibility, state-of-the-art adaptive attacks and defenses, and direct access to jailbreak artifacts.Its principles include archiving artifacts for stable comparisons, accepting white-box, black-box, universal, transfer, and adaptive attacks, and tracking open-sourced baselines.
  • Behavior datasets: JBB-Behaviors curates unique, realizable text-based harmful behaviors spanning a broad spectrum of misuse, alongside 100 topic-matched benign behaviors.The paired benign behaviors support refusal-rate evaluation without rewarding defenses that simply detect keywords, although some cases are borderline across providers.
  • Jailbreak artifacts: The artifacts repository stores prompts, responses, classifications, and metadata such as attack hyperparameters, attack success rate, and target-model query count.It currently includes artifacts for PAIR, GCG, JailbreakChat, and attacks from Andriushchenko et al., with intended expansion through future submissions.
  • Evaluation tooling: A standardized red-teaming pipeline streamlines jailbreak generation across varying tokenization schemes, sampling algorithms, and system prompts, while supporting modular implementations of five popular defenses.The framework emphasizes easy model loading and querying, and evaluation of test-time defenses should use adaptive attacks because transfer attacks provide only a lower bound on worst-case success.
  • Judge selection: Llama-3-70B and GPT-4 achieve above 90% agreement with expert annotators, while Llama Guard 2 reaches 87.7% agreement and approximately equal false positive and false negative rates.JailbreakBench compares six candidate classifiers against majority-vote labels from three expert annotators and prevents submissions from using its Llama-3-70B judge to reduce overfitting.
  • Framework and leaderboard: The standardized evaluation framework implements system prompts, the JUDGE classifier, cloud and local querying, and benchmarking and reporting across supported LLMs; the web leaderboard displays results, artifacts, and filterable metadata.The leaderboard covers attack and defense evaluations and links entries to their corresponding jailbreak artifacts.

4 Evaluation of the current set of attacks and defenses

JailbreakBench evaluates four baseline attacks and five baseline defenses using standardized success and efficiency metrics. The results show attack- and model-dependent effectiveness, with Erase-and-Check the strongest tested defense, although Prompt with RS retains non-trivial success and defenses can substantially increase inference time.

  • Baseline attacks: Four baseline attacks comprise GCG, PAIR, hand-crafted Jailbreak Chat prompts, and prompt plus random search enhanced by self-transfer.GCG optimizes one adversarial suffix per behavior using its default implementation.
  • Baseline defenses: Five baseline defenses are included: SmoothLLM, perplexity filtering, Erase-and-Check, synonym substitution, and removal of non-dictionary items.SmoothLLM uses swap perturbations with N = 10 samples and q = 10% perturbation; perplexity filtering follows Jain et al.’s algorithm with Llama-2-7B perplexity.
  • Metrics: The evaluation reports attack success rate judged by Llama-3-70B, plus average queries and tokens for measuring attack efficiency.Query and token counts are omitted for transfer and hand-crafted attacks; Prompt with RS excludes queries used to optimize the universal prompt template.
  • Evaluation of attacks: The AIM template from JB-Chat works on Vicuna but fails for every behavior on Llama-2 and GPT models, while GCG performs slightly below previously reported jailbreak percentages.The paper attributes GCG’s lower result primarily to more challenging behavior selection and other evaluation differences described in the passage.
  • Evaluation of defenses: Against transfer attacks, Erase-and-Check appears the most solid defense, but Prompt with RS still achieves non-trivial success across all evaluated LLMs.Defense results reuse jailbreaking strings found on undefended models, and only three defenses are evaluated in the main table.
  • Evaluation of defenses: Some defenses substantially increase inference time, motivating consideration of computational cost alongside their effectiveness.The benchmark’s accessible defenses are intended to support adaptive jailbreak algorithms designed to counter them.

5 Outlook · Structure of the appendix · A Maintenance plan

JailbreakBench is presented as an initial, evolving effort to standardize jailbreak-robustness evaluation while acknowledging scope limitations. Its appendices document benchmark maintenance, behaviors, judge-classifier datasets, experiments, reproducibility, and system prompts, supported by free hosting and community contributions.

  • 5 Outlook: JailbreakBench currently accepts submissions across threat models and target architectures as an initial step toward standardized jailbreak evaluation.The authors intend to update the benchmark periodically as the field develops.
  • 5 Outlook: Future updates may expand behavior datasets, strengthen defense evaluation for non-conservatism and efficiency, update the judge classifier, and re-evaluate closed-source-model attack success.
  • 5 Outlook: The benchmark excludes system-prompt modification, response prefilling, and non-text modalities, restricting the current attacker scope to text inputs.
  • Structure of the appendix: Appendices A–D cover benchmark maintenance, JBB-Behaviors, judge-classifier datasets, and additional details and experiments for the main evaluations.
  • Structure of the appendix: Appendices E and F address reproducibility practices and the system prompts used across evaluations and models.
  • A Maintenance plan: The leaderboard is hosted on GitHub Pages, a free service, while the benchmark library is maintained on GitHub using its free basic features.
  • A Maintenance plan: The behavior dataset is hosted on HuggingFace Datasets, whereas jailbreak artifacts are hosted in a separate GitHub repository.
  • A Maintenance plan: The authors expect maintenance to be substantially community-driven and encourage submissions of jailbreak strings and new defenses through README instructions.

B Further details on JBB-Behaviors

JBB-Behaviors combines behaviors from AdvBench, TDC/HarmBench, and original contributions, curated across diverse and balanced misuse categories. It includes 100 behaviors rather than serving as a superset of its constituent datasets.

  • Source of behaviors: JBB-Behaviors uses three source labels: "AdvBench," "TDC/HarmBench," and "Original.""TDC/HarmBench" denotes behaviors from TDC later assimilated into HarmBench, while "Original" denotes behaviors unique to JBB-Behaviors.
  • Source of behaviors: 100 behaviors are curated across ten categories to provide a diverse and balanced span of misuse categories.The categories include both well-represented areas such as “Malware/Hacking” and less common areas such as “Government decision-making.”
  • Source of behaviors: JBB-Behaviors is not a superset of its constituent datasets.The benchmark focuses on 100 behaviors curated to span diverse misuse categories.
  • Results by behavior source: The dataset contains 18 AdvBench behaviors, 27 TDC/HarmBench behaviors, and 55 behaviors unique to JBB-Behaviors.Table 5 records attack success rates for PAIR, GCG, and JBC across these three behavior sources.

C Details on the judge dataset

The judge dataset combines attack-generated jailbreak prompts across multiple models with benign examples, human expert labels, and automated-judge results to support accurate judge selection.

  • Jailbreak prompt construction: 200 prompts use PAIR or GCG attacks generated on Vicuna, comprising 100 PAIR prompts and 50 GCG prompts alongside additional template-based prompts.The template-based set adds 50 prompts generated across Vicuna, Mistral, Llama-2, and Llama-3.
  • Jailbreak prompt construction: 50 template-based prompts use adversarial suffixes found by random search across Vicuna, Mistral, Llama-2, and Llama-3.The allocation is 10 on Vicuna, 10 on Mistral, 20 on Llama-2, and 10 on Llama-3.
  • Judge evaluation dataset: The evaluation dataset adds 100 benign XS-Test examples to the attack-generated prompts for testing candidate judges.The resulting dataset is intended to support selection of a more accurate jailbreak judge.
  • Annotations and release: The repository provides the dataset with three human expert labels per jailbreak prompt and automated-judge evaluation results.The materials are available in the authors’ HuggingFace Datasets repository.

D Additional evaluations

Additional evaluations examine Synonym Substitution and Remove Non-Dictionary under the Table 3 protocol. Synonym Substitution achieves a 24% attack success rate for various precomputed jailbreaks, while Remove Non-Dictionary provides more limited improvements, including 91% success for Prompt with RS on Vicuna.

  • Additional defenses: The evaluation adds Synonym Substitution and Remove Non-Dictionary defenses using the same protocol as Table 3.These defenses were omitted from the main text because of space constraints.
  • Additional defenses: 24% attack success rate is reported for various precomputed jailbreaks under Synonym Substitution.The passage describes this as surprisingly effective and the highest attack success rate result among the reported additional-defense findings.
  • Additional defenses: 91% success rate remains for Prompt with RS on Vicuna under Remove Non-Dictionary.This defense therefore yields more limited improvements than Synonym Substitution in the cited example.

E Reproducibility · F System prompts

The benchmark addresses reproducibility by documenting randomness sources, evaluation metadata, and observed changes in attack success rates. It also publishes the system prompts used across target models and judges.

  • E Reproducibility: ≈5% success rate was reported for GCG on GPT models after jailbreak artifacts were released, likely because of safety patches.The transfer attacks were evaluated on June 5th, 2024.
  • E Reproducibility: Locally run models mainly vary through usually negligible GPU-computation randomness, whereas Together AI can produce discrepancies for Vicuna and Llama-Guard.Together AI remains more deterministic than GPT-4 judging, whose inference is non-deterministic even with a fixed seed.
  • E Reproducibility: Leaderboard submissions automatically include evaluation dates and inference type to document evaluation conditions.The benchmark strives to make evaluations reproducible despite identified randomness sources.
  • F System prompts: System prompts for Llama-2 and Vicuna are provided in Tables 7 and 8.These tables document the prompts used for the respective models.
  • F System prompts: Table 9 gives the shared system prompt for Llama Guard 1 and 2, replacing {prompt} and {response} with the values being classified.The prompt is used to classify the respective prompt and response values.
  • F System prompts: Table 10 provides the prompt for Llama-3-70B as a jailbreak judge, and the prompt can be copied from the repository.This prompt is part of the benchmark’s documented judge configuration.
  • F System prompts: Table 11 provides the prompt for Llama-3-8B as a refusal judge for experiments measuring defenses’ refusal rates.The prompt can be copied from the repository and supports the refusal-rate experiments shown in Figure 2.

G Main checklist

The authors report reproducible benchmark evaluations through released code, data, methodological details, and fixed jailbreaking strings, while acknowledging omitted error bars and compute reporting. They also document asset licensing, data privacy, and the sensitive content inherent to the jailbreaking task.

  • Reproducibility: The repository provides code, data, method details, and hyperparameters needed to reproduce the benchmark evaluations.No models were trained for the benchmark.
  • Limitations: The evaluations omit error bars and total compute reporting, although LLM-judge evaluations are mostly deterministic under greedy generation.The authors discuss other randomness sources in Appendix E and use a fixed set of jailbreaking strings.
  • Asset release: Existing assets are cited and linked for licensing, while newly released datasets and code are provided in the repository under an MIT license.The repository includes the benchmark’s code and data.
  • Data and ethics: The authors state that the datasets contain no personal data, while acknowledging sensitive content in prompts and jailbreaking strings by design.They use publicly available datasets and models, and discuss the sensitive content in §5.

H Dataset-related supplementary material … I.3 Dataset overview

The supplementary material documents responsibility, dataset governance, authorship, funding, and the JBB-Behaviors dataset overview. The dataset contains prompts intended to elicit specified language-model behaviors, with 200 total samples split between harmful and benign behaviors.

  • H.2 Statement of responsibility: The authors accept responsibility for rights violations and release the data under an MIT license.
  • I Data card: The dataset documentation follows Pushkarna et al. (2022) guidelines and identifies the dataset as JBB-Behaviors.
  • I.1 Summary: Edoardo Debenedetti of ETH Zurich authored the datacard.
  • I.2.1 Publishers: The publishing organizations are the University of Pennsylvania, ETH Zurich, EPFL, and Sony AI.
  • I.2.2 Dataset Owners: The listed dataset owners are researchers from the University of Pennsylvania, ETH Zurich, EPFL, and Sony AI.The named owners are Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram`er, Hamed Hassani, and Eric Wong.
  • I.2.3 Funding Sources: No institution explicitly funded benchmark creation, while several authors received partial support from ARO, NSF, the Sloan Foundation, or a Google Fellowship.The authors note that stated opinions and conclusions do not necessarily reflect the views of ARO, the Department of Defense, or the United States Government.
  • I.3 Dataset overview: The data subjects are behaviors that a human might want to elicit in a language model.
  • I.3 Dataset overview: 200 total samples comprise 100 harmful behaviors and 100 benign behaviors, represented as prompts targeting specific language-model behaviors.

I.3.1 Sensitivity of data … I.11 Known applications and benchmarks

JailbreakBench is an openly accessible, text-based benchmark assembled from author-created and prior-work behaviors, with documented safety considerations, maintenance, intended uses, and evaluation applications. Its behaviors are curated for uniqueness and uniform coverage of OpenAI usage-policy misuse categories, while users are cautioned to include adaptive security evaluations.

  • I.3.1 Sensitivity of data: The dataset contains disturbing material that poses indirect risk, has no intentionally or unintentionally collected sensitive data, and includes a README disclaimer.It contains no human or other sensitive attributes.
  • I.3.2 Dataset version and maintenance: The dataset is actively maintained at v1.0, updated and released in 06/2024, with minor corrections planned but no major updates or timeline.The dataset is described as static in collection cadence and update frequency.
  • I.4 Example of data points: The benchmark uses text data whose fields specify a behavior, objectionable goal, affirmative target, OpenAI-policy category, and source.Sources include Original, the Trojan Detection Challenge 2023 Red Teaming Track/HarmBench, and AdvBench.
  • I.5.1 Motivations; I.5.2 Intended use; I.9.3 Use in AI and ML systems: JailbreakBench targets research on LLM robustness, defenses, and attack effectiveness, and is intended for safe research use and validation.Its application domains include machine learning, large language models, and AI safety.
  • I.5.2 Intended use; I.9.1 Use with Other Data; I.9.2 Forking and sampling: Using only existing attacks, especially precomputed jailbreak prompts, is unsuitable for robustness evaluation without an adaptive attack and thorough security evaluation.The benchmark is described as safe to use with other data and safe to fork, but sampling is not recommended because it is not particularly large.
  • I.6.1 Access: The benchmark is externally and openly accessible without access-control lists.Citation guidelines were to be decided upon acceptance.
  • I.7.1 Collection: Behaviors were collected from authors and Zou et al. (2023) and Mazeika et al. (2023, 2024) during 11/2023–05/2024 using artificial generation and author creativity.The collection is static; source behaviors are marked in the dataset, curated for uniqueness, and uniformly distributed across OpenAI usage-policy misuse categories.
  • I.10.1 Synopsis; I.11 Known applications and benchmarks: No transformations were applied beyond manual changes, and the benchmark’s known ML application is large language models, with evaluation methodology reported in Section 4.The synopsis lists no transformed fields or known correlations.
Loading 2404.01318v5…