EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models

Motivation

• Jailbreak methods are difficult to compare fairly because they often use different data samples and victim models.Reimplementing prior work is also time-consuming and error-prone when source code is unavailable.Source: §1 · p. 1

Framework

• EasyJailbreak decomposes jailbreak methods into Selector, Mutator, Constraint, and Evaluator components.Selectors identify threatening candidates, mutators refine prompts, constraints filter ineffective instances, and evaluators assess attack success.Source: §1 · pp. 1–2

Contributions

• The framework supports standardized benchmarking of 12 jailbreak attacks within a unified implementation.This enables the methods to be benchmarked, compared, and analyzed together.Source: §1 · p. 2
• Its modular architecture supports reuse of shared components and lowers the development barrier for creating new attacks.Researchers can focus on novel components while minimizing development effort.Source: §1 · p. 2
• EasyJailbreak supports open-source and closed-source models, including LlaMA2 and GPT-4, and allows users to incorporate their own models and datasets through HuggingFace transformers.Source: §1 · p. 2

Empirical findings

• 60% average breach probability was found across 10 LLMs evaluated against jailbreak attacks, including 57% and 33% average ASRs for GPT-3.5-Turbo and GPT-4.The findings indicate that advanced models remain susceptible to jailbreak attacks.Source: §1 · p. 2

Taxonomy

• Jailbreak attacks are organized into Human-Design, Long-tail Encoding, and Prompt Optimization categories.The categories cover manual prompt construction, rare-format inputs, and automated vulnerability exploration.Source: §2 · p. 2

Human Design

• Human-designed attacks use role-playing, scenario crafting, and context-learning vulnerabilities to induce responses to malicious instructions.Source: §2 · p. 2

Long-tail Encoding

• Long-tail Encoding exploits limited generalization to unseen security-alignment data by using rare formats, low-resource languages, or encrypted inputs with embedded decoding functions.Source: §2 · p. 2

Prompt Optimization

• Prompt optimization methods search for vulnerabilities through gradients, genetic algorithms, prompt fuzzing, iterative scoring, persuasion, or assistant-model generation.Source: §2 · p. 2

3 Framework

• EasyJailbreak integrates 11 classic jailbreak attack methods into a unified framework with a user-friendly interface.Users can execute attack algorithms with just a few lines of code.Source: §3 · p. 2

3.1 Preparation

• Users configure queries, seed prompt templates, target or auxiliary models, and attack hyperparameters before running an attack.Queries specify prohibited instructions, while seeds are initial templates intended to improve ASR.Source: §3.1 · p. 3

3.2 Selector

• Selectors choose promising candidates when productive mutators cause alternative jailbreak inputs to grow exponentially.EXP3SelectPolicy is an example that uses Exp3 to select seeds for subsequent updates.Source: §3.2 · p. 3

3.3 Mutator

• Mutators modify rejected jailbreak inputs to seek successful attacks, including by translating inputs into languages on which the target model was rarely trained.Source: §3.3 · p. 3

3.4 Constraint

• Constraints remove randomly generated jailbreak inputs that are likely to fail, such as off-topic inputs identified by DeleteOffTopic.Source: §3.4 · p. 3

3.5 Evaluator

• Evaluators automatically determine whether a target response constitutes a successful jailbreak and whether further attack steps are warranted.ClassificationJudge uses a trained classifier to distinguish responses indicating successful jailbreaking.Source: §3.5 · p. 3
• The framework combines Selector, Mutator, Constraint, and Evaluator components flexibly across attack recipes.Table 1 marks unused components as N/A.Source: Table 1 · p. 4

3.6 Report

• Each attack produces a report containing success rates, response perplexity, model replies, jailbreak prompts, and evaluations.These reports support pinpointing vulnerabilities and strengthening model defenses.Source: §3.6 · p. 4

Configuration

• EasyJailbreak runs PAIR attacks by combining an attack model, target model, evaluation model, and JailbreakDataset.The attack model generates prompts, the target model is tested, and the evaluation model judges success.Source: §4 · pp. 4–5
• The attack model generates the jailbreak prompt and performs the initial attack phase.Source: §4 · p. 4
• The target model is the LLM users attempt to jailbreak and the primary object of security testing.Source: §4 · p. 5
• The evaluation model determines whether an illegal query successfully jailbroke the target and assesses attack effectiveness.Source: §4 · p. 5

Web interface

• The web platform lets users compare outputs from different jailbreak approaches, including ChatGPT’s response to PAIR.Source: §4 · p. 5, Figure 3 · p. 5

5.1 Setup

• The setup evaluates ten models on AdvBench using human-designed, long-tail encoding, and optimization-based attack recipes.GenerativeJudge uses GPT-4-turbo-1106 as the scoring model.Source: §5.1 · p. 5

5.2 Result Analysis

• 63% average breach probability across ten models shows pervasive vulnerability, while GPT-3.5-Turbo and GPT-4 reach 57% and 33% average ASR, respectively.Source: §5.2 · p. 5, Table 2 · p. 6
• Closed-source GPT-3.5-Turbo and GPT-4 average 45% ASR versus 66% for the remaining open-source models, while Llama2 security is comparable to GPT-4.Source: §5.2 · p. 5
• On both Llama2 and Vicuna, 13B versions have slightly higher average jailbreak success rates than 7B versions, so larger size does not necessarily improve security.The paper proposes further validation on larger-scale models such as Llama2-Chat-70B.Source: §5.2 · pp. 5–6

5.3 Evaluator Comparison

• GPT-4 leads evaluator accuracy, TPR, and F1 but takes longer, whereas the Gptfuzz classifier combines high efficiency with the lowest FPR.Rule-based matching is fast but has higher FPR because of strictness and limited adaptability.Source: §5.3 · p. 6, Table 3 · p. 6

• EasyJailbreak simplifies evaluating and developing attack and defense strategies across a spectrum of language models.Source: §6 · p. 6
• The evaluation reports a 60% average breach probability in advanced LLMs, highlighting the need for enhanced security measures.Source: §6 · p. 6

• EasyJailbreak’s dual-use potential motivates conscientious research and deployment focused on advancing LLM security.Source: Ethics Statement · p. 6
• The authors advocate responsible disclosure, ethical usage, monitoring, iterative updates, and cybersecurity collaboration to strengthen defenses.Source: Ethics Statement · p. 6

A Component Details

• The section provides detailed explanations of all components implemented in Easyjailbreak.Source: A Component Details · p. 8

A.1 Selector

• Selectors choose seeds for subsequent updates using random, bandit, round-robin, MCTS, score-based, or reference-loss strategies.These policies balance exploration, scoring, and loss-based selection across candidate seeds.Source: §A.1 · p. 8

A.2 Mutator

• Mutators update jailbreak inputs through generative-model rephrasing or translation, gradient-based perturbation, and rule-based transformations such as Base64 encoding.The mutation strategies target improved jailbreak likelihood through language-model generation, gradients, or predefined rules.Source: §A.2 · p. 8

A.3 Constraint

• Constraints filter attack instances by harmfulness, topical relevance, or perplexity before further processing.DeleteHarmLess removes inputs judged harmless, DeleteOffTopic removes off-topic inputs, and PerplexityConstraint removes high-perplexity inputs.Source: §A.3 · p. 8

A.4 Evaluator

• Evaluators use classifiers or generative models to score responses from 0 to 9 or judge whether an attack succeeds.Classifier-based evaluators include ClassificationGetScore and ClassificationJudge, while generative evaluators include GenerativeGetScore and GenerativeJudge.Source: §A.4 · pp. 8–9
• Figure 4 compares ASR and efficiency across jailbreak methods on llama2-7b and llama2-13b.The figure separates ASR in panel (a) from efficiency in panel (b).Source: Figure 4 · p. 9
• Rule-based evaluators determine success through exact matching, pattern presence, or required response prefixes.The framework provides Match, PatternJudge, and PrefixExactMatch according to the required matching level.Source: §A.4 · p. 9

B Efficiency Comparison

• Human-designed methods require the shortest processing time, but their success rates may drop sharply as models are updated or replaced.JailBroken, DeepInception, and ICA typically use only a human-written prompt.Source: B Efficiency Comparison · p. 9
• 61% and 90% are the Cipher success rates for Llama2-7B-chat and Llama2-13B-chat, respectively, whereas MultiLingual performs worse because of limited multilingual capability.Cipher’s performance is attributed to security optimizations not adequately covering its language-processing abilities.Source: B Efficiency Comparison · p. 9
• 764.4 hours and 803.9 hours are required by prompt optimization on Llama2-7B-chat and Llama2-13B-chat, respectively, while achieving higher success rates.The long runtimes reflect iterative optimization for finding effective jailbreak prompts.Source: B Efficiency Comparison · p. 9