Source-linked AI summary
JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs
Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, Yang Zhang
TL;DR
Prior jailbreak evaluations often compared limited method sets or did not ensure aligned settings, leaving no unified, comprehensive benchmark. This study assesses diverse attacks across LLMs, forbidden questions, and defenses, finding vulnerability in black-box settings and different levels of defense resilience. The authors identify attack diversity and seed dependence as relevant patterns for future study.
Problem
Earlier jailbreak evaluations compared limited method sets, sometimes lacked aligned settings, or omitted newer automatic attacks, leaving no unified and comprehensive benchmark.
Source: §1 · p. 1Method
Results
In real-world black-box settings LAA achieves a 100% ASR on DeepSeek-V3, while PAIR and TAP retain ASRs above 15% under all eight defenses and seed-based attacks can be nearly ineffective against defenses.
Source: §1 · p. 2Takeaways & Limitations
The authors argue that natural, diverse prompts—especially those not requiring initial seeds—are harder to defend against and merit researchers’ attention.
Source: §8 · p. 7The policies and corresponding dataset are static and may become outdated as LLM-related policies evolve.
Source: Limitation · p. 7
Abstract
from arXiv · showhide
Jailbreak attacks aim to bypass the LLMs' safeguards. While researchers have proposed different jailbreak attacks in depth, they have done so in isolation -- either with unaligned settings or comparing a limited range of methods. To fill this gap, we present a large-scale evaluation of various jailbreak attacks. We collect 17 representative jailbreak attacks, summarize their features, and establish a novel jailbreak attack taxonomy. Then we conduct comprehensive measurement and ablation studies across nine aligned LLMs on 160 forbidden questions from 16 violation categories. Also, we test jailbreak attacks under eight advanced defenses. Based on our taxonomy and experiments, we identify some important patterns, such as heuristic-based attacks could achieve high attack success rates but are easy to mitigate by defenses, causing low practicality. Our study offers valuable insights for future research on jailbreak attacks and defenses. We hope our work could help the community avoid incremental work and serve as an effective benchmark tool for practitioners.
1 Introduction
The paper addresses the lack of a unified, systematic benchmark comparing jailbreak attacks across aligned settings and method types. It assesses 17 attacks and reports that real-world black-box models remain vulnerable, while attack diversity is associated with differing defense resilience.
1 Introduction
- Earlier evaluations either compared a limited range of attacks, used settings that did not ensure alignment, or omitted emerging automatic methods.The paper frames its assessment as addressing the absence of a unified, systematic, comprehensive, and fair benchmark.Source: §1 · p. 1
- The assessment collects 17 representative attacks, establishes a six-category taxonomy, and evaluates attacks across nine LLMs and eight defenses.It also conducts ablation studies and covers both automatic and non-automatic attack types.Source: §1 · pp. 1–2
- LAA achieves a 100% ASR on DeepSeek-V3 in real-world black-box settings.The authors also report that seed-based attacks can be highly successful yet vulnerable to defenses, while diverse feedback-based attacks are more stable under defenses.Source: §1 · p. 2
- PromptGuard reduces LAA’s attack success rate to 0%.The authors attribute seed-based attacks’ defense vulnerability to their limited prompt diversity and similar distributions.Source: §1 · p. 2
- PAIR and TAP still achieve ASRs above 15% when all eight defense strategies are deployed.The paper contrasts these more diverse, natural prompts with seed-based attacks that defenses can render nearly ineffective.Source: §1 · p. 2
- The authors hope their diverse forbidden-question dataset and taxonomy-based observations will support future research and help avoid incremental work.They specifically suggest giving heuristic-based attacks lower priority.Source: §1 · p. 2
2 Background and Related Works
The background describes safety alignment and existing jailbreak and defense approaches. It distinguishes the paper’s broader evaluation from earlier studies focused on non-automatic human-based or obfuscation-based attacks.
2.1 Safety-Aligned LLMs
- Safety-aligned LLMs use techniques including reinforcement learning from human feedback and red teaming, but their use of external information can expose them to harmful knowledge.The background notes that adversaries exploit these capabilities to launch attacks.Source: §2.1 · p. 2
2.2 Jailbreak Attacks and Defenses
- Jailbreak attacks commonly craft specialized prompts, while some can exploit generation parameters without altering the original question.Defenses include perplexity-based detection, pretrained classifiers, and detection assisted by another LLM.Source: §2.2 · p. 2
- Earlier evaluations [43] [73] [74] focused on non-automatic human-based or obfuscation-based attacks, whereas this work includes emerging automatic methods and ablation studies.The related-work section also points to a table summarizing the tested attacks.Source: §2.2 · p. 2, Table 1 · p. 2, §2 · p. 2
3 Jailbreak Attack Taxonomy
The taxonomy classifies 17 attacks by whether they alter forbidden questions and, if so, how they produce prompts. It separates five prompt-modifying categories from generation-parameter attacks that use the original question.
3 Jailbreak Attack Taxonomy
- The taxonomy uses two criteria: whether an attack alters the forbidden question and, if it does, how it generates the modified prompt.The authors state that the taxonomy focuses on how attacks jailbreak the target LLM, rather than other features such as access.Source: §3 · p. 3
3.1 Human-Based Method
- Human-based methods use jailbreak prompts found in the wild; the tested examples include AIM, Devmoderanti, and Devmode v2.The listed prompts were the top three human-based prompts by votes on the jailbreakchat website.Source: §3.1 · p. 3
3.2 Obfuscation-Based Method
- Obfuscation-based methods use techniques such as translation to exploit vulnerabilities including low-resource languages or seemingly harmless synonyms.Tested examples include Base64, Combination, Zulu, and DrAttack.Source: §3.2 · p. 3
3.3 Heuristic-Based Method
- Heuristic-based methods optimize prompts with algorithms such as mutation, random search, or genetic algorithms, typically starting from human-crafted seeds.The tested attacks are AutoDAN, GPTFuzz, and LAA.Source: §3.3 · p. 3
3.4 Feedback-Based Method
- Feedback-based methods iteratively modify prompts using feedback such as gradient information or jailbreak scores, relying less on human-crafted initial seeds.The tested attacks are GCG, COLD, PAIR, and TAP.Source: §3.4 · p. 3
3.5 Fine-Tuning-Based Method
- Fine-tuning-based attacks fine-tune an attack LLM to generate potential jailbreak prompts from forbidden questions.The tested methods are MasterKey and AdvPrompter.Source: §3.5 · p. 3
3.6 Generation-Parameter-Based Method
- Generation-parameter-based attacks exploit sampling methods or parameters during generation without creating typical jailbreak prompts.Generation Exploitation is the tested example; the paper treats this category separately because its mechanism differs fundamentally from other categories.Source: §3.6 · p. 3
4 Forbidden Question Dataset
The dataset combines explicit violation categories from five providers’ usage policies with questions drawn from prior work and newly generated candidates. It contains 160 questions across 16 categories, manually reviewed for category fit.
4 Forbidden Question Dataset
- The authors unify policies from five major LLM-related service providers into 16 explicit violation categories, distinguishing general moral violations from region-specific ones.They aim for explicit coverage because providers’ policies vary and may group specific concerns under broad categories.Source: §4 · p. 3
- The paper defines “feedback” broadly enough to include optimization based on jailbreak scores and objectives, as well as gradient-based methods.It treats generation-parameter attacks separately because their mechanisms differ fundamentally from other categories.Source: §4 · p. 3
- The dataset contains 160 forbidden questions—10 per category—selected from prior work and generated candidates, then manually screened and reviewed by two annotators.Filtering removes improper, duplicate, or irrelevant queries; the authors describe the resulting set as diverse and broader than prior datasets in its category coverage.Source: §4 · p. 4
5 Evaluation Settings
The evaluation tests nine aligned LLMs using direct attacks on forbidden questions, with attack success rate (ASR) as the metric and GPT-4-Turbo judging responses. To compare iterative methods, it standardizes what counts as a step while noting that some attacks use fixed templates instead.
Test Datasets and LLMs
- Nine popular aligned LLMs—five open-source and four closed-source—are evaluated using the forbidden-question dataset.DeepSeek-V3 is used in a closed-source setting because of computing-resource limitations.Source: §5 · p. 4
Metrics
- ASR is the ratio of successful jailbreak queries to total queries, and GPT-4-Turbo labels responses across three aspects.The authors report that human annotation found this judging method outperformed string matching and LLM-as-a-judge alternatives.Source: §5 · p. 4
Unification of the Term “Step”
- Each prompt modification counts as one step, with a maximum of 50 modification steps per question; GCG and COLD use 500 gcg_step.The authors set these values to make GCG and COLD performance and efficiency comparable to other methods.Source: §5 · p. 4
Evaluation Settings
- Fixed-template attacks such as Combination, Base64, and Zulu do not fit the iterative step definition, while GE tests 50 generation settings per question.For GE, a question counts as successfully jailbroken if any of its 50 responses is labeled successful; other methods use the top-1 response.Source: §5 · p. 4
6 Evaluation Results
Across direct attacks, LAA has the highest reported average ASR, while results vary by model, attack type, and violation category. The evaluation also finds that taxonomy-level patterns generally hold across policies, with exceptions, and reports ablations on transferability, prompt length, and other factors.
6.1 Evaluation of Attack Taxonomy
- Attack performance varies by type: human-based methods nearly fail on some strongly aligned Llama models, obfuscation attacks are often model-specific, and feedback-based methods remain comparatively stable.GE achieves 0.68 average ASR and ranks third; DrAttack is an exception to the model-specific pattern and exploits cross-model semantic vulnerabilities.Source: §6.1 · pp. 4–5
- 0.87 average ASR is achieved by LAA, which outperforms all other attacks, including white-box attacks; its ASR reaches 0.55 on Llama3.1.All tested models exceed 0.55 ASR under at least one attack; DeepSeek-V3 has the highest average ASR at 0.75.Source: §6.1 · pp. 4–5, Table 2 · p. 5
6.2 Evaluation of Unified Policy
- Political Activities has ASR ≥0.80 on GPT-3.5 and GPT-4 despite explicit prohibition, while categories labeled “general” are challenging to jailbreak.Disinformation Spread and Privacy Breach have average ASRs of 0.49 and 0.43, respectively; DeepSeek-V3, Vicuna, and GPT-3.5 have the highest ASRs across 9, 5, and 3 categories.Source: §6.2 · pp. 5–6, Table 3 · p. 5, Figure 3 · p. 6
6.3 Taxonomy-Policy Relationship
- Taxonomy-level ASR trends generally persist across individual policies, but Llama3.1 is an exception for severe categories where LAA can trail obfuscation attacks.For Llama3.1, the authors suggest sensitive phrases retained in LAA prompts may trigger safeguards, whereas Zulu or DrAttack replace or obfuscate phrases; on some vulnerable categories, attack ASRs fall below baseline.Source: §6.3 · p. 6
6.4 Takeaways
- Seed-based heuristic attacks are relatively non-robust, feedback-based methods are more robust, and obfuscation attacks often work only against specific powerful LLMs.The authors also report that heuristic attacks have good transferability but relatively long prompts.Source: §6.4 · p. 6, §6.5 · p. 6
7 Jailbreak Defenses
Combining all eight defenses leaves several diverse, seed-free attacks effective, while seed-based methods are nearly ineffective. The study also evaluates individual defenses, finding that their effectiveness varies across attack methods.
7.1 Defense Methods
- The eight tested defenses are Self-Reminder, Moderation, Perplexity, Erase, Llama-Guard, Llama-Guard-2, Llama-Guard-3, and Prompt-Guard.The evaluation reports bypass rate and ASR.Source: §7.1 · p. 6, §7.2 · p. 6
7.2 Experiments
- Prompt-Guard reduces the average ASR of nearly all seed-based methods to zero, but DrAttack and TAP retain ASRs of 0.55 and 0.59, respectively.Moderation is almost ineffective in most cases, while Perplexity is effective against high-perplexity prompts such as Zulu and GCG.Source: §7.2 · p. 6
- DrAttack, COLD, PAIR, and TAP remain effective under all eight defenses together, while seed-based attacks—including LAA—are nearly ineffective.The authors attribute the contrast to diverse, natural prompts being harder to detect than seed-derived prompts with similar patterns and distributions.Source: §7.2 · pp. 6–7, Table 4 · p. 7
8 Discussion
The discussion flags possible safety-alignment trade-offs for categories perceived as less harmful and highlights seed-free, natural, diverse prompts as harder to defend against. It frames how to handle the less-harmful categories as an open question.
Safety Alignment Trade-Offs
- Categories perceived as less harmful may receive less attention in safety alignment or involve utility-safety trade-offs, even when provider policies explicitly cover them.How to address these categories remains an open question.Source: §8 · p. 7
Future Attacks and Defenses
- Natural, diverse jailbreak prompts that do not rely on initial seeds are described as harder to defend against than seed-based attacks.The discussion calls for attention to attacks requiring no initial seeds rather than focusing solely on variants of known prompts.Source: §8 · p. 7
9 Conclusion
The paper assesses 17 jailbreak attacks with a six-category taxonomy and evaluates them across 160 questions spanning 16 violation categories. It finds that current defenses handle attacks using human-based initial seeds more effectively than attacks without such seeds.
9 Conclusion
- The study analyzes 17 representative attacks, proposes a six-category taxonomy, and evaluates them on 160 questions across 16 violation categories.The conclusion also reports evaluations across nine LLMs and under eight defenses.Source: §9 · p. 7
- In real-world black-box settings, the latest LLMs remain vulnerable to current jailbreak attacks, with LAA performing best.Current defenses could effectively defend against attacks using human-based initial seeds but struggle against those not using such seeds.Source: §9 · p. 7
- The authors call for more focus on creating and defending against attacks that require no initial seeds.They hope the evaluation supports the development of trustworthy LLMs.Source: §9 · p. 7
Limitation
The paper identifies limits in attack coverage, the currency and granularity of its policy questions, and the quality of its automated evaluation. It also notes possible biases in dataset construction, attack assistance, and human annotation.
Research Scope
- The evaluation covers 17 representative attacks, while more than 200 attacks exist, so the paper’s research scope remains limited.Evaluating every attack in one paper is described as infeasible.Source: Limitation · p. 7
Static Policies and Questions
- The 2024 policies and questions are static and may become outdated as LLM-related policies evolve.The study focuses on differences between violation categories; within-category response differences remain for future exploration.Source: Limitation · p. 7
Jailbreak Evaluation Methods
- The automatic ASR evaluation is imperfect, and the authors cannot assess jailbroken-response quality or compare it with harmful knowledge from other sources.Expert manual annotation assessing both ASR and response quality is described as impractical because of its high cost.Source: Limitation · pp. 7–8
Potential Biases
- Possible biases include overlap between public training data and newer models’ training data, LLM assistance in dataset construction or attacks, and human annotation.The authors report manual checks and modifications but acknowledge that some biases may remain unavoidable.Source: Limitation · p. 8
Ethical Considerations
The study used publicly accessible data and did not involve participants, but it could reveal which methods trigger inappropriate content more effectively. The authors say they shared findings responsibly and planned to notify involved LLM service providers.
Ethical Considerations
- The authors planned to disclose findings to the involved LLM providers because the study identifies methods that trigger inappropriate content more effectively.They state that they took care to share the findings responsibly and believe societal advantages outweigh the relatively minor increased harm risks.Source: Ethical Considerations · p. 8
A.1 Transferability
The appendix evaluates transferability, prompt length, runtime, and performance over model updates, alongside details on violation categories, annotation, and dataset comparisons. Transfer attacks remain effective on many targets, but results vary across models, categories, methods, and time.
A.1 Transferability
- AutoDAN’s average transfer ASR is 0.55, above the 0.40 baseline but below its 0.98 original performance on Vicuna.LAA’s transfer ASR exceeds 0.70 on all models except the Llama series and reaches 0.99 on GPT-3.5; white-box attacks can also transfer to black-box targets.Source: §A.1 · p. 11, Table 5 · p. 12
- Llama series models have average transfer ASRs below 0.30, lower than baseline, while Political Activities transfer attacks average 0.75 versus 0.78 on Vicuna.A transfer attack reaches 0.90 ASR on GPT-3.5 for Political Activities; Llama models show resilience across most violation categories.Source: §A.1 · pp. 11–12, Table 6 · p. 12
- Transfer attacks raise average ASRs above 0.20 for Illegal Activities, Privacy Breach, and Disinformation Spread, whose baseline ASRs are below 0.05.AutoDAN, TAP, and LAA improve on baseline across most violation subcategories on GPT-3.5 and GPT-4; TAP reaches 0.63 and 0.60 ASR on Terrorist Content, respectively.Source: §A.1 · p. 12
A.2 Token Numbers
- Human-based jailbreak methods average over 670 tokens, while AIM averages 382.78 and the baseline forbidden questions average 14.78.AutoDAN, GPTFuzz, LAA, and MasterKey also exceed 300 average tokens; Generation Exploitation averages 14.78 and Zulu 38.06.Source: §A.2 · pp. 12–13
A.3 Time Efficiency
- Generation Exploitation combines high attack performance with low time cost, while GCG takes over three times as long as AutoDAN and five times as long as Generation Exploitation.Generation Exploitation generates 50 responses without additional operations; runtime comparisons are qualitative because API calls introduce uncontrollable factors.Source: §A.3 · p. 13
A.4 Longitudinal Test
- After a February 16 update, all tested GPT-3.5 attacks except PAIR declined, while GPTFuzz, PAIR, and TAP were relatively stable across updates.Human-based and most obfuscation-based attacks were effectively mitigated; GPT-4 performance was more stable, with only minor declines for Devmode v2 and Base64.Source: §A.4 · p. 15
C.1 Standard of Violation
- A question’s violation status is determined by whether the question itself violates policy, based on human screening, regardless of an LLM’s response.The text gives copyrighted song lyrics as an example where a seemingly harmless model response does not change the query’s violation status.Source: §C.1 · p. 15
B.1 Violation Category Description; B.2 Clarification of Child Endangerment; B.3 Annotation of Violation Categories
- The study covers 16 of the 17 listed violation categories, excluding Child Endangerment; its policy categories were manually annotated and classified as general or specific.A category was labeled general only when all three annotators unanimously agreed.Source: §B.3 · p. 15, Table 8 · p. 16
D.1 ASR Evaluation Methods
This section describes how jailbreak success is evaluated and contextualizes the study within related LLM misuse and security research. It also documents experimental settings, attack categories, defenses, and pointers to supplementary prompts and results.
D.1 ASR Evaluation Methods
- String Match can mistake “sure” or “sorry” for success or failure, while single-label judging can falsely count mere rephrasing as a successful jailbreak.The authors also note that methods optimized and evaluated with the same judge may be biased in comparisons, motivating an impartial third-party arbiter.Source: §D.1 · p. 17
- The proposed GPT-4 judge assigns three binary labels—rephrasing, answering the adversary, and including requested content—and counts success only when all three are 1.It uses few-shot learning with three demonstrations.Source: §D.1 · p. 17
- 94.84% label matching rate was achieved by the proposed method, versus 75.63% for String Match and 67.03% for Judging Model (Single Label).Three annotators independently labeled 640 randomly selected responses; K-alpha was 0.87, indicating good annotator consistency.Source: §D.1 · p. 17
D.2 Discussion of High ASR Baseline
- High baseline ASRs arise mainly in six violation categories, and baseline ASRs sometimes exceed attack ASRs for human-based or obfuscation-based attacks and strongly secured LLMs.The authors suggest lower attention to these categories during safety alignment or utility trade-offs may contribute; they also suggest obfuscation comprehension failures and detectable prompt patterns may explain some below-baseline attack rates.Source: §D.2 · pp. 17–18
E.1 Supplementary Defense Metrics
- Bypass rate (BR) measures defense evasion, and the supplementary results report average BRs for attacks across nine LLMs under different defenses.Table 11 notes that AutoDAN, GCG, COLD, and AdvPrompter results use five open-source LLMs, and “All” means all eight defenses are deployed together.Source: §E.1 · p. 18, §E.2 · p. 18, Table 11 · p. 19
F Setting Supplement
- The experiments used a resource-sufficient attacker, target-model default settings, temperature 0.01, and no added protective system prompts.White-box attacks may require substantial compute, whereas black-box attacks generally need only API access; DeepSeek-V3 experiments used DeepSeek’s official API.Source: F Setting Supplement · p. 18
G.1 Attack Selection
- The selected attacks had public repositories and leading-venue publication or high citation counts, while the taxonomy classifies jailbreak mechanisms rather than access type.Human-based prompts were used without alteration; obfuscation attacks use techniques such as encoding, translation, or fragmentation to exploit alignment vulnerabilities.Source: §G.1 · p. 18, §G.2 · p. 18, §G.3 · p. 19
G.5 Heuristic-Based Method
- Heuristic attacks use mutation, random search, or genetic algorithms; feedback-based methods optimize prompts iteratively, fine-tuning methods generate prompts with trained attack models, and generation-parameter attacks manipulate decoding.Heuristic methods typically rely on human-crafted seeds; fine-tuning takes time but enables rapid prompt generation once complete.Source: §G.5 · p. 19, §G.6 · p. 19, §G.7 · p. 20
I.1 Misuse of LLMs
- LLMs can be misused to generate misinformation, promote conspiracy theories [37], support phishing, violate intellectual property, plagiarize [31], and orchestrate hate campaigns [72].The passage argues that ease of misalignment underscores the need for robust security measures and ongoing vigilance.Source: §I.1 · p. 21
- Jailbreaks [40] [43] [75] and other attacks, including prompt injection [71], backdoors [9] [16], data extraction [14] [46], and obfuscation [37], threaten LLM safeguards.The passage also names membership inference [59] and adversarial attacks [36], noting that code injection and virtualization can bypass vendor safeguards [37].Source: §I.1 · p. 21
I.2 Security Measures of LLMs
- LLM security measures include internal safety training and external safeguards that monitor or filter conversation text [32] [75].Internal approaches include safety reward models, RLHF context distillation, and rejection sampling; external examples include OpenAI’s moderation endpoint [47] and systems using an additional LLM [34] [39].Source: §I.2 · p. 21
I.3 Discussion of Concurrent Works
- The study offers unified empirical results and ablations, using a broader forbidden-question dataset and attack coverage than concurrent work.Unlike survey and dataset work, it includes automated feedback-based attacks such as GCG and heuristic-based attacks such as GPTFuzz; it also reports more advanced methods and detailed ablations than.Source: §I.3 · p. 21
J Related Prompts
- The appendix provides prompts for generating forbidden questions and guiding GPT-4-Turbo to judge responses, omitting harmful few-shot examples.The prompts are shown in Figures 8 and 9.Source: J Related Prompts · p. 21, Figure 8 · p. 22, Figure 9 · p. 22
K Additional Experiment Results
- Additional results include category-level direct-attack ASR, running-time duration, prompt token counts, and transfer-attack results.The appendix points to Figures 10–11 for direct attacks, Table 7 for running time, Figure 12 for token counts, and Figure 4 for transfer attacks; it supplies no values in this passage.Source: K Additional Experiment Results · p. 22, Figure 10 · p. 23, Figure 11 · p. 24