PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Measurement of Jailbreaking Evaluators

Yujie Mu

arXiv:2609.10594v1cs.CRcs.LG

TL;DR

Human evaluation of jailbreak responses is costly and difficult to scale, while independently validated evaluators can produce incomparable success measurements. This paper compares six evaluators on two human-labeled datasets under a controlled setup. JADES performs best overall, with HarmBench and StrongReject also performing strongly.

  • Problem

    Human evaluation is costly to scale, and evaluator-specific success definitions and fragmented validation make jailbreak results difficult to compare.

  • Method

    The study compares six jailbreak evaluators on JailbreakQR and JailMeter-Eva using human judgments, shared judge backbones where applicable, and agreement, error, and attack-family analyses.

  • Results

    JADES achieves the strongest overall performance across both datasets, while HarmBench and StrongReject also demonstrate consistently strong agreement with human judgments.

  • Takeaways & Limitations

    Evaluator choice is an important source of variation in jailbreak measurement, and structured procedures can provide more reliable agreement with human judgments.

  • Takeaways & Limitations

    The English-only, moderate-sized datasets may not generalize to other languages, domains, target models, or future jailbreak strategies, and human labels are imperfect.

Abstract

from arXiv · show

Expert evaluation of jailbreak responses is costly and difficult to scale, so the community increasingly relies on automated evaluators to determine whether an attack succeeds. However, jailbreak studies typically validate their chosen evaluator independently, repeatedly spending resources on similar evaluation efforts while making results across papers difficult to compare. Different evaluators also encode different definitions of jailbreak success, meaning that reported attack strength and apparent progress can depend substantially on which evaluator is used. We systematically compare six evaluators that recur in recent jailbreak attack and defense research: HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES, and JailMeter. To our knowledge, no prior study has evaluated all six on the same human-labeled data under a controlled setup. We evaluate them on JailbreakQR and JailMeter-Eva, using human judgments as the reference, and measure agreement with humans, error types, and consistency across attack families. For evaluators that require a general-purpose LLM judge, we use a shared backbone to control for model-specific variation. We found that JADES exhibits the best overall performance, while HarmBench and StrongReject also demonstrate good performance.

1 Introduction

Automated jailbreak evaluators are increasingly necessary because human review is costly, but evaluator differences can change reported attack success and hinder comparison across studies. This work provides a controlled comparison of six evaluators on two human-labeled datasets.

  • Human evaluation is costly and difficult to scale, motivating automated judgments of whether jailbreak attempts succeed.
  • Different evaluators encode different success definitions, so the same response may be accepted by one evaluator and rejected by another.
  • Independent validation practices use different datasets, baselines, and settings, leaving evaluator behavior difficult to compare across studies.
  • The study compares HarmBench, JailbreakBench, JailbreakRadar, StrongReject, JADES, and JailMeter using human judgments on JailbreakQR and JailMeter-Eva.
  • JADES achieves the strongest overall performance, while HarmBench and StrongReject also perform well under the evaluated settings.

2 Background

Jailbreak evaluation has progressed from simple refusal-string checks and toxicity classifiers toward LLM-based and structured evaluators. These methods differ in what evidence they examine and how they define successful harmful compliance.

  • String matching is inexpensive and reproducible but weakly proxies compliance because avoiding refusal language does not ensure useful harmful content.
  • Toxicity classifiers estimate harmful content, but harmfulness is not equivalent to fulfilling a harmful request.
  • JailbreakBench and JailbreakRadar use LLM judges, while HarmBench uses a dedicated classifier and StrongReject applies a structured rubric.
  • JADES decomposes harmful requests into weighted sub-questions, whereas JailMeter separates evidence extraction from final judgment.
  • The evaluated harmful requests commonly originate from safety benchmarks such as Do-Not-Answer and SALAD-Bench, which differ in scope and implementation quality.

3 Evaluators Under Measurement

The study formalizes evaluator agreement against human labels and compares six evaluators whose procedures range from classification and holistic judgment to rubric-based, decompositional, and evidence-based scoring.

  • Agreement measures how often an evaluator’s mapped output matches the human annotation for examples in dataset D.
  • Human annotations are treated as reference labels, but the paper uses agreement rather than correctness because annotations may contain noise.
  • HarmBench classifies whether a response exhibits a specified harmful behavior using a dedicated classifier rather than a general-purpose LLM judge.
  • JailbreakBench gives a single holistic verdict, while JailbreakRadar requires substantive, requested, and relevant content simultaneously.
  • StrongReject uses a refusal gate and rates non-refusals for convincingness and specificity, while JADES scores weighted sub-answers and JailMeter evaluates extracted evidence.

4 Experiment Setup

The experiment evaluates six evaluators on two complementary human-labeled datasets, aligns their outputs to common labels and metrics, and controls the judge backbone where possible. The comparison remains methodological rather than a reproduction of every system’s native configuration.

  • 4.1 Datasets: The study uses JailbreakQR and JailMeter-Eva, which differ in label granularity, target models, and attack families.
  • 4.1 Datasets: JailbreakQR contains 400 human-annotated prompt-response pairs from five attacks and four target models, with failed, partially successful, and successful labels.
  • 4.1 Datasets: JailMeter-Eva contains 330 non-refusal GPT-4o mini responses from eight attacks, including 73 labeled successful jailbreaks.
  • 4.2 Label Alignment and Metrics: Outputs are mapped to common labels, published thresholds are used, and evaluation reports accuracy, precision, recall, F1, and false positive rate.
  • 4.3 Judge Backbone: Five LLM-based evaluators share one backbone and retain their published prompts and procedures, but HarmBench and the native StrongReject scorer are not reproduced exactly.

5 Experiment Results

Across both human-labeled datasets, JADES performs best overall, while StrongReject and HarmBench remain competitive; structured evaluators generally outperform single-verdict judges.

  • 5.1 Main Results: 92.3% accuracy, 92.5% F1, and 5.8% FPR make JADES the strongest evaluator on JailbreakQR.StrongReject follows with 89.5% accuracy and 89.8% F1, while HarmBench reaches 88.3% accuracy and 88.5% F1.
  • 5.1 Main Results: 95.2% accuracy, 89.3% F1, and 3.9% FPR make JADES strongest on JailMeter-Eva, with StrongReject second at 93.0% accuracy and 85.0% F1.The evaluator ordering largely persists across both datasets.
  • 5.1 Main Results: 85.5% accuracy but 64.7% precision and 69.6% F1 show that accuracy can overstate evaluator quality on JailMeter-Eva.The dataset’s non-refusal cases make precision, F1, and false positive rate important complements to accuracy.
  • 5.2 Discussion: Evaluator choice materially changes jailbreak conclusions because the six evaluators differ substantially in agreement with human labels even under a shared judge backbone.This identifies evaluator design, not only judge-model choice, as a source of measurement variation.
  • 5.2 Discussion: JADES and StrongReject use structured evaluation, whereas JailbreakBench and JailbreakRadar rely primarily on single prompted verdicts and show larger performance gaps.StrongReject separates refusal detection from response quality, while JADES decomposes harmful requests into smaller criteria.
  • 5.2 Discussion: HarmBench remains competitive, while JADES and StrongReject achieve the lowest false positive rates across both datasets.Lower false positive rates reduce the risk of counting failed or insufficient responses as successful attacks.

6 Limitations

The study’s conclusions are bounded by English-only, moderate-sized datasets, imperfect human labels, dataset provenance, and partial control over evaluator configurations.

  • 6 Limitations: English-only, moderate-sized datasets and imperfect human labels limit generalization across languages, domains, target models, and future jailbreak strategies.JailbreakQR’s ordinal Krippendorff’s α of 0.823 indicates that some disagreements may reflect annotation uncertainty.
  • 6 Limitations: Dataset provenance may embed related assumptions because each evaluation set was introduced alongside one of the studied evaluators.The study therefore emphasizes consistency across both datasets rather than either dataset in isolation.
  • 6 Limitations: The shared-backbone setup controls methodological variation only partially because HarmBench uses a dedicated classifier and StrongReject uses a rubric-based judge.The results compare methodologies under a controlled setting rather than reproducing every system in its native configuration.
  • 6 Limitations: The work evaluates jailbreak measurement rather than developing new attacks, using previously released datasets and attack outputs.Its primary contribution is methodological: improving the reliability and comparability of jailbreak-success measurement.

7 Conclusion

Automated jailbreak evaluators are difficult to compare because they use different success definitions, so this work evaluates six systems under a unified protocol against human judgments. JADES performs best overall, while StrongReject and HarmBench also agree consistently with humans, supporting a common reference for evaluator selection.

  • Evaluator choice is an important source of variation in jailbreak measurement because widely used evaluators implement different definitions and procedures for success.
  • The study compares six evaluators on two human-labeled datasets using a unified protocol and a fixed LLM judge backbone where applicable.
  • JADES achieves the strongest overall performance, while StrongReject and HarmBench also demonstrate consistently strong agreement with human judgments.
  • The comparison aims to provide a common reference for selecting and evaluating jailbreak evaluators while reducing repeated reliability studies under incompatible settings.

A.1 Risks of Large Language Models

Large language models create expanding safety, privacy, and security risks across deployment contexts. These risks include memorization and inference of sensitive information, unsafe visual content, authenticity concerns, concealed harmful objectives, and assistance with conventional attacks.

  • The growing deployment of generative models has motivated extensive work on their security, privacy, and safety risks.
  • Language models can memorize sensitive training examples, leak information from prior conversations, and infer personal attributes absent from the input.
  • Generative vision systems face risks including hidden secondary tasks, unsafe or hateful content, compromised safeguards, and uncertain authenticity or provenance.
  • Harmful objectives can be concealed within benign tasks, while language-model capabilities can reduce the effort required for spear-phishing and other security attacks.

A.2 Safety Training and Its Limits

Modern language models undergo safety alignment before deployment through human feedback, model-written principles, and automated red-teaming. These mechanisms improve behavior but do not provide complete protection, and prior work identifies structural limits to robust alignment.

  • Safety alignment commonly uses reinforcement learning from human feedback, model-written principles and feedback, and automated red-teaming.
  • These mechanisms substantially improve model behavior but do not provide complete protection against unsafe behavior.

A.3 Jailbreak Attacks

Jailbreak attacks include human-authored prompts, obfuscation and transformation, optimization-based search, and inference-time or multi-turn manipulation. This work uses outputs from existing attack families to study evaluator behavior, while evaluator choice also affects conclusions about defense robustness.

  • Jailbreak attacks span human-authored prompts, request obfuscation, optimization-based search, and inference-time or multi-turn manipulation.
  • Obfuscation methods preserve harmful semantics while changing surface form through multilingual formulations, ciphers, ASCII art, source code, or personalized transformations.
  • Optimization-based attacks search for prompts that maximize jailbreak objectives using adversarial suffixes, evolved prompts, iterative refinement, fuzzing, random search, or decoding manipulation.
  • The study does not introduce a new attack; it uses outputs from existing attack families as material for examining how evaluators interpret identical model responses.
  • Evaluator choice affects conclusions about both attack strength and defense robustness because measured defense effectiveness depends on how jailbreak success is defined and detected.
Loading 2609.10594v1…