PaperlayerView exact revision on arXiv

Source-linked AI summary

JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework

Fan Liu, Yue Feng, Zhao Xu, Lixin Su, Xinyu Ma, Dawei Yin, Hao Liu

arXiv:2410.12855v2cs.CLcs.AI

TL;DR

Jailbreak judging remains difficult because current methods can lack explainability and struggle with complex scenarios. JAILJUDGE introduces diverse human-annotated datasets, explainable multi-agent scoring, and an end-to-end judge model; its experiments report state-of-the-art performance across judge methods and tested models.

  • Problem

    Jailbreak judgments are difficult, and existing methods may lack explainability or reliability across complex scenarios, making accurate assessment an open problem.

    Source: §1 · p. 2

  • Method

    JAILJUDGE combines diverse human-annotated datasets with a reasoning-based multi-agent judge and uses its judgments to instruction-tune an end-to-end judge model.

    Source: §1 · pp. 2–3, §3.2 · p. 6

  • Results

    The experiments report state-of-the-art performance for jailbreak-judge methods across models including GPT-4 and safety moderation tools such as Llama-Guard-3.

    Source: §7 · p. 11

  • Takeaways & Limitations

    The authors report that JAILJUDGE Guard can improve downstream jailbreak attack and defense mechanisms.

    Source: §7 · p. 11

Abstract

from arXiv · show

Despite advancements in enhancing LLM safety against jailbreak attacks, evaluating LLM defenses remains a challenge, with current methods often lacking explainability and generalization to complex scenarios, leading to incomplete assessments (e.g., direct judgment without reasoning, low F1 score of GPT-4 in complex cases, bias in multilingual scenarios). To address this, we present JAILJUDGE, a comprehensive benchmark featuring diverse risk scenarios, including synthetic, adversarial, in-the-wild, and multilingual prompts, along with high-quality human-annotated datasets. The JAILJUDGE dataset includes over 35k+ instruction-tune data with reasoning explainability and JAILJUDGETEST, a 4.5k+ labeled set for risk scenarios, and a 6k+ multilingual set across ten languages. To enhance evaluation with explicit reasoning, we propose the JailJudge MultiAgent framework, which enables explainable, fine-grained scoring (1 to 10). This framework supports the construction of instruction-tuning ground truth and facilitates the development of JAILJUDGE Guard, an end-to-end judge model that provides reasoning and eliminates API costs. Additionally, we introduce JailBoost, an attacker-agnostic attack enhancer, and GuardShield, a moderation defense, both leveraging JAILJUDGE Guard. Our experiments demonstrate the state-of-the-art performance of JailJudge methods (JailJudge MultiAgent, JAILJUDGE Guard) across diverse models (e.g., GPT-4, Llama-Guard) and zero-shot scenarios. JailBoost and GuardShield significantly improve jailbreak attack and defense tasks under zero-shot settings, with JailBoost enhancing performance by 29.24% and GuardShield reducing defense ASR from 40.46% to 0.15%.

1 INTRODUCTION

Jailbreak judging is a difficult but fundamental task, and existing approaches can lack explanatory power or reliability on complex scenarios. JAILJUDGE combines a diverse benchmark with an explainable multi-agent judge and an end-to-end judge model.

1 INTRODUCTION

  • Jailbreak judging evaluates whether a jailbreak succeeded by assessing the harmfulness of an LLM’s target response.The paper describes this task as fundamental and relevant to safety evaluation, jailbreak attacks, and defenses.Source: §1 · p. 2, §2.4 · p. 3
  • Heuristics and toxic-text classifiers can misclassify benign responses or struggle with complex risks, while LLM-based methods may be costly or lack reliable explanations.The paper also notes potential biases and data ambiguities in LLM-based judgments.Source: §1 · p. 2
  • JAILJUDGE pairs human-annotated complex and multilingual test sets with a multi-agent framework for explainable, fine-grained judgments and an end-to-end judge model.The benchmark includes more than 4.5k complex-scenario and 6k multilingual labeled examples; the framework also supports instruction-tuning JAILJUDGE Guard.Source: §1 · pp. 2–3

2.1 LARGE LANGUAGE MODEL

  • LLMs predict a sequence’s next token from the preceding tokens, with the model assigning probabilities to possible next tokens.The model is denoted πθ, and y denotes its output sequence.Source: §2.1 · p. 3

2.2 JAILBREAK ATTACK AND DEFENSE ON LLM

  • Jailbreak attacks seek harmful outputs through adversarial prompts, while defenses seek safe responses using input or output filters.The paper describes input filters as potentially perturbing requests and output filters as potentially filtering malicious responses.Source: §2.2 · p. 3

2.3 EVIDENCE THEORY

  • Evidence theory represents uncertainty over mutually exclusive hypotheses and combines evidence sources using Dempster’s rule.For jailbreak judging, the hypotheses include jailbroken, non-jailbroken, uncertain, and no conclusion; the rule can aggregate judgments from different LLMs.Source: §2.3 · p. 3

2.4 PROBLEM DEFINITION

  • The explainability-enhanced judge produces a reason and a 1-to-10 jailbreak score, then applies a threshold α to make the binary judgment.A score of 1 denotes a fully compliant, non-jailbroken response, while 10 denotes a direct and egregious violation that fully addresses the task.Source: §2.4 · pp. 3–4

3 BUILDING JAILJUDGE BENCHMARK AND MULTI-AGENT JUDGE FRAMEWORK

JAILJUDGE combines diverse training data, human-annotated test sets, and a multi-agent judge that makes jailbreak decisions explainable and fine-grained.

3.1.1 JAILJUDETRAIN: INSTRUCTION-TUNING DATASET FOR COMPLEX JAILBREAK JUDGMENTS

  • JAILJUDGETRAIN contains 35k+ instruction-tuning items from six prompt sources and responses from closed-source, open-source, and defense-enhanced LLMs.The prompt sources include synthetic, adversarial, multilingual, in-the-wild, and deceptive harmful prompts alongside vanilla harmful prompts.Source: §3.1.1 · pp. 4–5

3.1.2 JAILJUDGETEST: HIGH-QUALITY HUMAN-ANNOTATED TEST JAILBREAK JUDGE DATASETS

  • JAILJUDGETEST comprises a 4.5k+ in-distribution set and a 6k+ multilingual out-of-distribution set spanning ten languages, with human labels refined through review and voting.Annotators label whether each prompt-response pair is jailbroken; mismatches with GPT-4 labels are manually reviewed and relabeled by multiple annotators using majority vote.Source: §3.1.2 · pp. 5–6

3.2 JAILJUDGE MULTIAGENT: MULTI-AGENT JUDGE FRAMEWORK

  • JailJudge MultiAgent uses judging, voting, and inference agents to produce an explainability analysis and a fine-grained jailbreak score.Judging agents provide reasons and scores; voting agents accept or reject judgments, and the inference agent combines those results for the final decision.Source: §3.2 · pp. 6–7
  • The framework converts each judge’s score into an uncertainty-aware BPA and aggregates judgments before voting, reducing judge-to-voter communication from O(k · m) to O(1 · m).The aggregated score and reason inform voting and the final inference.Source: §3.2 · pp. 6–7

4 JAILJUDGE GUARD AND JAILBREAK ENHANCERS

JAILJUDGE Guard supplies explainable jailbreak scores, and the paper uses it as the basis for an attack enhancer and a system-level defense.

JAILJUDGE Guard

  • JAILJUDGE Guard is instruction-tuned on Llama-2-7B to return reasoning and a jailbreak score from 1 to 10 for a prompt-response pair.A score of 1 indicates non-jailbroken and 10 indicates complete jailbreak.Source: §4 · p. 7

JailBoost

  • JailBoost uses JAILJUDGE Guard to iteratively evaluate attacker-refined prompts for their ability to elicit harmful outputs.The method is described as attacker-agnostic.Source: §4 · p. 7

GuardShield

  • GuardShield is a system-level defense that detects whether an LLM is jailbroken and generates a safe response.Its decision uses a predefined threshold.Source: §4 · p. 7

5 EXPERIMENTS

The experiments evaluate jailbreak judges on ID, OOD, and public datasets, then assess JailBoost and GuardShield on HEx-PHI and ablate the multi-agent framework.

5.1 JAILBREAK JUDGE EXPERIMENTS

  • JailJudge MultiAgent achieved the highest average F1 scores: 0.9197 on JAILJUDGE ID and 0.9609 on JBB Behaviors, outperforming open-source baselines across both datasets.Its JAILJUDGE ID explainability-quality score was 4.5234, compared with 4.3989 for GPT-4-Reasoning.Source: Table 2 · p. 8, §5.1 · p. 8
  • On zero-shot multilingual JAILJUDGE OOD, MultiAgent achieved F1 0.711 versus GPT-4-Reasoning’s 0.5633; the methods also outperformed open baselines across OOD and WILDTEST.The paper reports higher bias evaluations in low-resource languages; JAILJUDGE Guard’s WILDTEST F1 was 0.7314, but it fell short of MultiAgent on JAILJUDGE OOD.Source: §5.1 · pp. 8–9, Table 3 · p. 9

5.2 JAILBREAK ATTACK AND DEFENSE EXPERIMENTS

  • On zero-shot HEx-PHI, GuardShield achieved an average ASR of 0.15% and nearly 100% defense capability against four SOTA attackers.JailBoost increased attacker ASR compared with nominal AutoDAN.Source: §5.2 · p. 9

5.3 ABLATION STUDY

  • The ablation reports progressive improvements across datasets, with JAILJUDGE ID F1 rising from 0.55 for Vanilla GPT-4 to 0.91 for the full judge.On JBB Behaviors, the reported scores rose from 0.79 to 0.96; human evaluators gave MultiAgent an average 95.29% across four datasets.Source: §5.3 · p. 9

6 RELATED WORKS

Prior jailbreak-judge approaches include heuristics, toxic-text classifiers, and LLM-based methods, each with reliability or generalization concerns. The related work also surveys attack and defense strategies.

Jailbreak Judge

  • Heuristic judges can misclassify benign responses containing keywords, while toxic-text classifiers struggle to generalize to broad-risk and adversarial scenarios.LLM-based judges can also be costly, inherit model biases, and produce ambiguous or unreliable judgments.Source: §6 · pp. 9–10

Jailbreak Attack Methods

  • Jailbreak attacks use token-level optimization or prompt-level semantic manipulation to elicit harmful outputs.Examples include AutoDAN’s token refinement and PAIR’s iterative prompt refinement.Source: §6 · p. 10

Jailbreak Defense Methods

  • Defenses range from system-level input/output safeguards to model-level changes such as safety training.SmoothLLM uses majority voting over perturbed prompts; safety training adds safety-specific data during tuning.Source: §6 · pp. 10–11

7 CONCLUSIONS

JAILJUDGE combines human-annotated benchmark data with an explainable multi-agent judge and an API-free end-to-end judge model. The paper reports strong judge performance and downstream attack and defense applications.

7 CONCLUSIONS

  • JAILJUDGE evaluates complex risk scenarios with human-annotated data, JailJudge MultiAgent, and JAILJUDGE Guard trained without API costs.The conclusion reports that the judge methods achieve state-of-the-art performance, and that JAILJUDGE Guard can improve downstream jailbreak attack and defense mechanisms.Source: §7 · p. 11

Evaluation Framework Supplementary Material

The supplied passage identifies a supplementary section on JAILJUDGE Guard as an attack enhancer and defense method.

Evaluation Framework Supplementary Material

8 BUILDING JAILJUDGE BENCHMARK AND MULTI-AGENT JUDGE FRAMEWORK

The benchmark construction combines varied harmful-prompt sources, response sources, and annotation policies. Its reported statistics describe prompt complexity and jailbroken-status distributions for training, in-distribution, and multilingual out-of-distribution data.

8.1 COMPLEX HARMFUL PROMPT CONSTRUCTION

  • Prompt construction draws on vanilla harmful prompts, GPT-4 rewrites, adversarial rewrites, in-the-wild templates, and deceptive prompts refined through attacker-model interaction.The vanilla prompts are organized into 14 hazard categories, including Code Interpreter Abuse; deceptive prompts use role-playing, scenario assumptions, long contexts, and adaptive strategies.Source: §8.1 · pp. 18–20
  • The benchmark gathers target responses from closed-source, open-source, and defense-enhanced LLMs to include both safe and unsafe outputs.The described defenses include SmoothLLM, RPO, adversarial training, unlearning, and safety training.Source: §8.1 · p. 20

8.2 HUMAN ANNOTATOR TRAINING POLICIES

  • Human annotator training follows the MLCommons standardized LLM policy, with scenarios and examples explaining policy violations.The authors primarily follow OpenAI usage policies and also incorporate AI usage policies from the ML community.Source: §8.2 · p. 20

8.3 STATISTIC INFORMATION OF JAILJUDGE BENCHMARK

  • The prompt-complexity taxonomy defines five types: simple, adversarial, in-the-wild, synthetic vanilla, and deceptive harmful prompts.Q5 combines elements from multiple methods and is described as harder to detect and handle.Source: §8.3 · p. 20
  • JAILJUDGETRAIN is dominated by Q5 prompts, with 18,093 instances; JAILJUDGE ID contains 4,500 instances, including 66.4% jailbroken and 33.6% non-jailbroken cases.The ID set is described as balanced across risk scenarios, while Q5 is its most frequent complexity category and Q1 its least frequent.Source: §8.3 · p. 22, Figure 11 · p. 22
  • JAILJUDGE OOD has 6,300 multilingual instances, of which 88.6% are non-jailbroken and 11.4% jailbroken.The passage attributes the lower jailbroken share relative to ID to multilingual scenarios and the absence of optimized jailbreak attacks.Source: §8.3 · p. 22

9 MULTI-AGENT JUDGE FRAMEWORK

The framework combines judging agents, voting agents, and an inference agent to produce jailbreak judgments. The accompanying figures show language and jailbreak-status distributions in JAILJUDGE OOD.

9 MULTI-AGENT JUDGE FRAMEWORK

  • Figure 12 presents the language distribution in JAILJUDGE OOD.Source: Figure 12 · p. 23
  • Figure 13 presents the distribution of jailbroken instances across languages in JAILJUDGE OOD.Source: Figure 13 · p. 23
  • Judging agents assess prompts and model responses with initial reasons and scores; voting agents use those to assess judgment validity, and an inference agent makes the final judgment from votes and predetermined criteria.The section describes three judging agents and three voting agents; the inference-agent count is not stated in the supplied text.Source: §9 · pp. 22–23

10 JAILJUDGE GUARD: AN END-TO-END JAILBREAK JUDGE MODEL

JAILJUDGE Guard is instruction-tuned on explainability-enhanced JAILJUDGETRAIN using multi-agent-generated ground truth. It takes a prompt and model response and returns a rationale with a fine-grained jailbreak score.

10 JAILJUDGE GUARD: AN END-TO-END JAILBREAK JUDGE MODEL

  • JAILJUDGE Guard uses LoRA supervised fine-tuning on Llama-2-7B, with a user prompt and model response as input and an explanation plus 1–10 jailbreak score as output.A score of 1 indicates non-jailbroken and 10 complete jailbreak.Source: §10 · p. 23

11 JAILJUDGE GUARD AS THE ATTACKER ENHANCER AND DEFENSE METHOD

JailBoost uses JAILJUDGE Guard to evaluate adversarial prompts refined by an attacker, while GuardShield uses the judge for system-level safety moderation. The supplied passages describe their aims and algorithm references, but provide no outcome comparisons.

11.1 JAILJUDGE GUARD AS THE ATTACKER ENHANCER

  • JailBoost aims to create high-quality adversarial prompts that cause harmful LLM outputs, and uses JAILJUDGE Guard’s jailbroken score as an iterative prompt-quality evaluator against a threshold.The attacker refines the adversarial prompts; Algorithm 1 provides the detailed procedure.Source: §11.1 · pp. 23, 27

11.2 JAILJUDGE GUARD AS THE DEFENSE METHOD

  • GuardShield is a system-level defense for safety moderation that detects whether an LLM is jailbroken and generates a safe response; its detailed procedure is in Algorithm 2.The method uses a predefined threshold.Source: §11.2 · p. 27, §11.1 · p. 27

12 EXPERIMENTS

Experiments evaluate jailbreak-judging performance, explanation quality, and attack and defense effectiveness using benchmark and external datasets. Human evaluators gave the multi-agent judge a 95.29% average explanation-agreement score across four datasets.

12.1 JAILBREAK JUDGE EVALUATION DATASETS AND METRICS

  • The evaluation reports accuracy, precision, recall, F1, and GPT-4-rated explainability quality on a 1–5 scale, with higher scores indicating clearer reasoning.The supplied passage defines the classification metrics and references the GPT-4 evaluation prompt.Source: §12.1 · p. 27, Figure 19 · p. 28
  • JAILJUDGE ID and OOD are evaluated alongside JBB Behaviors and WILDTEST; the latter contain approximately 200 and over 1,700 instances, respectively.JBB Behaviors covers broad risk scenarios; WILDTEST contains real-world user interactions with LLM prompts.Source: §12.1 · p. 27

12.2 JAILBREAK JUDGE BASELINES

  • Baselines include heuristic and classifier systems, safeguard models, prompt-driven GPT-4 judges, a GPT-4 reasoning baseline, and multi-agent voting; JAILJUDGE Guard is trained on JAILJUDGETRAIN.The specified judge-score threshold α is 2; this passage reports setup, not comparative results.Source: §12.2 · p. 29

12.3 HUMAN EVALUATION OF EXPLAINABILITY

  • 95.29% average human evaluation score was achieved by the multi-agent judge across JAILJUDGE ID, JBB Behaviors, JAILJUDGE OOD, and WILDTEST.Three evaluators rated explanation reasonableness on 200 sampled instances per dataset, with voting determining the final score.Source: §12.3 · p. 29

12.4 JAILJUDGE GUARD AS AN ATTACK ENHANCER AND DEFENSE METHOD: DATASETS AND METRICS

  • Zero-shot attack and defense evaluation uses the 330-instance HEx-PHI dataset, 11 attack-risk categories, Vicuna-13B-v1.5, and GPT-4 to assess attack success.Higher ASR indicates a more effective attack, while lower ASR indicates a more effective defense.Source: §12.4 · pp. 29, 31, Figure 23 · p. 32

Related papers

Loading 2410.12855v2…