Source-linked AI summary
A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models
Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, Stjepan Picek
TL;DR
Comprehensive evidence on how jailbreak attacks perform against defended LLMs, and how defenses perform against attacks, remains limited. This study evaluates nine attacks and seven defenses across three models; it finds template attacks effective and current defenses generally weak.
Problem
Prior work had not comprehensively evaluated jailbreak attacks against defended LLMs and defenses against jailbreaks, leaving their effectiveness across both sides insufficiently assessed.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The authors highlight the need for a uniform jailbreak-detection baseline and better defense techniques.
Source: §7 · p. 8Resource constraints excluded larger models and powerful commercial models such as GPT-4, Gemini, and Palm2; the evaluation also preceded significant autoDan repository updates that may affect outcomes.
Source: §8 · pp. 8–9
Abstract
from arXiv · showhide
Large Language Models (LLMS) have increasingly become central to generating content with potential societal impacts. Notably, these models have demonstrated capabilities for generating content that could be deemed harmful. To mitigate these risks, researchers have adopted safety training techniques to align model outputs with societal values to curb the generation of malicious content. However, the phenomenon of "jailbreaking", where carefully crafted prompts elicit harmful responses from models, persists as a significant challenge. This research conducts a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. We meticulously investigate nine attack techniques and seven defense techniques applied across three distinct language models: Vicuna, LLama, and GPT-3.5 Turbo. We aim to evaluate the effectiveness of these attack and defense techniques. Our findings reveal that existing white-box attacks underperform compared to universal techniques and that including special tokens in the input significantly affects the likelihood of successful attacks. This research highlights the need to concentrate on the security facets of LLMs. Additionally, we contribute to the field by releasing our datasets and testing framework, aiming to foster further research into LLM security. We believe these contributions will facilitate the exploration of security measures within this domain.
1 Introduction
The introduction identifies a gap in comprehensive evaluation of jailbreak attacks and defenses, then presents a study of both. Its reported findings favor template-based attacks and identify broad weaknesses in current defenses.
1 Introduction
- Prior studies did not comprehensively evaluate attack methods against defended LLMs and defense methods against jailbreaks, motivating the study’s two research questions.The questions address attack effectiveness on unprotected models and defense effectiveness against attacks across varied LLMs.Source: §1 · pp. 1–2
- The study selected nine attack methods and seven defenses, constructed a benchmark of 60 categorized malicious queries, and evaluated attack effectiveness and defense robustness on malicious and benign inputs.The selected methods prioritized prevalent options with open-source code.Source: §1 · p. 2
- Template-based attacks outperform other techniques, white-box gradient-based approaches generally fall short of universal generative methods, and special tokens affect attack success.Bergeron is the comparatively strongest defense; other evaluated defenses either fail to stop jailbreaks or also prohibit benign prompts.Source: §1 · p. 2
- The authors describe the work as a first systematic evaluation of jailbreak attacks versus defenses across open- and closed-source LLMs.They also report releasing a benchmark that collects attack and defense techniques to support further research.Source: §1 · p. 2
2 Background and Related Work
This section situates the study within work on jailbreaks and prior attack and defense evaluations. It defines the paper’s categories for attacks and defenses and describes the safety context for jailbreak behavior.
2 Background and Related Work
- The study addresses a gap left by prior work that evaluated attacks on unprotected models but did not assess them against defenses, or vice versa.The authors cite Mazeika et al. (2024) and Zhou et al. (2024) as examples of work that explored attack techniques without evaluating defense techniques.Source: §2 · p. 2
2.1 LLM Jailbreak
- Jailbreaks use crafted prompts to elicit harmful content despite safeguards such as RLHF, RAFT, and PRO.The precise mechanisms remain debated; proposed explanations include gaps in safety training and conflicts between helpfulness and safety, while a reported gradient-based attack uses a suffix as an embedding vector guiding inappropriate output.Source: §2.1 · pp. 2–3
- The paper adopts an established categorization of “benign content” and says its assessment follows that framework.The passage defines the term as responses considered morally or ethically inappropriate.Source: §2.1 · p. 3
2.2 Jailbreak Attack Techniques
- The paper groups attacks into generative techniques, template techniques, and training-gaps techniques.These respectively involve dynamically produced attacks, predefined templates or generation-setting modifications, and weaknesses associated with insufficient safety training.Source: §2.2 · p. 3
2.3 Jailbreak Defense Techniques
- The paper groups defenses into self-processing, additional-helper, and input-permutation techniques.Input-permutation defenses alter prompts and query target models multiple times to detect and counter malicious requests aimed at gradient-based vulnerabilities.Source: §2.3 · p. 3
3 Study Design
The study evaluates jailbreak attacks across language models and defenses against those attacks. Its design specifies the methods, models, benchmark, experimental settings, labeling, and evaluation metrics.
3 Study Design
- The study asks how effective jailbreak attacks are across language models and how well defenses withstand attacks across different models.Source: §3 · p. 3
3.1 Baseline Selection
- Nine attack methods and seven defenses were selected based on popularity and source-code accessibility, including generative and template-based attacks.Source: §3.1 · pp. 3–4
3.2 LLMs under Test
- The evaluation covers Llama-2-7b, Vicuna-v1.5-7b, and GPT-3.5-Turbo-1106, but excludes GPT-4 because its operational and token costs made comparison economically challenging.Preliminary GPT-3.5-Turbo testing used 79,314 queries.Source: §3.2 · p. 4
3.3 Experimental Configuration
- Queries were run five times, generative attacks were capped at 75 iterations, and GCG on Llama used 500 iterations.The study reports that GCG on Llama required more iterations to jailbreak most queries; other settings defaulted to original repositories when recommendations were unavailable.Source: §3.3 · pp. 4–5
3.4 Benchmark Construction
- The benchmark expanded Liu et al.’s dataset to 60 malicious queries—double its original size—using manual curation and selected AdvBench examples.Source: §3.4 · p. 5
3.5 Result Labeling
- Responses were categorized using both automated and manual labeling strategies.Source: §3.5 · p. 5
3.6 Evaluation Metric
- Attack evaluation uses ASR and query Efficiency, while defense evaluation uses DPR, BSR, and GRQ, with Alpaca Eval assessing benign-response quality.ASR measures compromised questions relative to total questions; Efficiency measures successful queries relative to attempts.Source: §3.6 · p. 5
4 RQ1: Effectiveness of Jailbreak Attack
The attack evaluation finds template-based methods effective across the tested models, while generative strategies also perform well. LLaMA is harder to jailbreak than Vicuna, and unlawful-practice, harmful-content, and illegal-activity queries are especially challenging across models.
4 RQ1: Effectiveness of Jailbreak Attack
- 78 templates, Jailbroken, and GPTFuzz perform well against GPT-3.5-turbo and Vicuna; Jailbroken, Parameter, and 78 templates perform best on LLaMA.The study attributes template methods’ prevalence to the intricate design of their prompts.Source: §4 · p. 5, Figure 2 · p. 6
- GPTFuzz, Pair, and Tap are the top-performing generative strategies, and LLaMA is harder to jailbreak than Vicuna.Source: §4 · p. 5
- Queries about unlawful practice, harmful content, and illegal activities are the most challenging to address across all tested models.Source: §4 · p. 6
5 RQ2: Effectiveness of Jailbreak defense
The defense evaluation finds that, apart from Bergeron, current strategies are largely inadequate. The study also reports minor variation in benign-response quality among three response-generating methods.
5 RQ2: Effectiveness of Jailbreak defense
- Apart from Bergeron, defenses are largely inadequate; benign-response quality shows minor variation among the three innovative methods evaluated.The study visualizes defense results using lower DPR and higher BSR as the optimal direction.Source: §5 · p. 6
6 Discussion
The discussion contrasts attack strategies, examines how special tokens affect jailbreak success, and identifies methodological, cost, and latency constraints on defense evaluation.
6.1 Comparative Performance of White-Box and Black-Box Attacks
- White-box AutoDAN and GCG underperform universal, pre-designed template attacks that need no model internals; LLaMa is harder to jailbreak than Vicuna, especially under white-box attacks.The authors suggest LLaMa’s greater resilience may reflect its comprehensive safety training.Source: §6.1 · p. 6
- 12 steps and five successful jailbreak attempts brought Vicuna’s loss to stability, while LLaMa failed within 75 iterations despite a lower final loss.The experiment used one randomly selected question.Source: §6.1 · p. 6, Figure 4 · p. 7
6.2 Impact of Special Tokens on Jailbreak Attack Performance
- Adding [/INST] rendered text-continuation templates ineffective, while omitting it reduced success for two role-play templates.The authors say the reasons for the role-play result are not immediately apparent.Source: §6.2 · p. 7
- The authors call for testing special tokens beyond [/INST], including «sys» and <s>, across LLaMa and additional models.They describe the findings as preliminary.Source: §6.2 · p. 7
6.3 Enhancing Defense Mechanisms Against Diverse Malicious Queries
- Defense evaluation lacks a uniform methodology, and predefined sets can misclassify benign responses as malicious; detection is also constrained by classifier characteristics.The text notes that GPT-4 limitations can contribute to benign inputs being wrongly classified.Source: §6.3 · p. 7
- Bergeron’s effectiveness is limited by its reliance on LLMs and predefined processing and verification prompts, which undermine reliability and add substantial operational costs.The authors identify this as a cost concern despite the method’s promise.Source: §6.3 · p. 7
- RALLM and Smooth-LLM take longer to process queries and scale poorly on complex inputs; RAIN showed a similar issue in preliminary experiments.The authors attribute the longer processing time to repeatedly checking a single prompt variation.Source: §6.3 · pp. 7–8
7 Conclusions
The paper presents an assessment of jailbreak attacks and defenses, together with a framework for evaluating LLM robustness. Its experiments identify effective attack methods, report generally weak defense performance, and call for stronger defenses and a uniform detection baseline.
7 Conclusions
- The study assesses nine attacks and seven defenses across three models and releases a framework for evaluating LLM robustness against threats.The authors describe this as the first comprehensive assessment of existing attack and defense strategies in LLM security.Source: §7 · p. 8
- Template methods are notably effective, with 78 templates identified as the most powerful technique; GPTFuzz is the most effective generative method within the experiment budget.Bergeron performs comparatively better than other evaluated defenses, while the study reports general ineffectiveness across current defenses.Source: §7 · p. 8
- The findings support establishing a uniform baseline for jailbreak detection and developing better defense techniques.The study also reports that using the '[/INST]' marker affected results for the Llama model.Source: §7 · p. 8
8 Limitations
The evaluation excludes larger models and powerful commercial models because of limited resources, and its results may be affected by autoDan updates made after the evaluation.
8 Limitations
- Limited resources kept the evaluation from covering 13B- and 33B-parameter models or powerful commercial models such as GPT-4, Gemini, and PaLM 2.The authors identify this as a scope constraint on the evaluated models.Source: §8 · p. 8
- The evaluation preceded significant autoDan repository updates identified in February 2024, so those updates may affect the reported outcomes.The authors state that they intend to align their repository with the modifications.Source: §8 · pp. 8–9
9 Ethical Considerations and Disclaimer
The authors describe ethical safeguards including use of publicly accessible resources, limits on potentially malicious response length, and reporting findings to model developers and providers.
9 Ethical Considerations and Disclaimer
- The research team says it used exclusively publicly accessible resources and followed responsible scientific norms.The authors frame these choices as part of their commitment to ethical conduct.Source: §9 · p. 9
- The study limits potentially malicious response length to reduce the risk of spreading harmful content while supporting evaluation and learning.The authors state that this measure is intended to avoid revealing practical information prone to misuse.Source: §9 · p. 9
- The authors report their findings systematically to the developers and providers of the analyzed models.They describe this as a contribution to dialogue on LLM security and vulnerability mitigation.Source: §9 · p. 9
A.1 Analysis of Categorization of Attack Techniques
The appendix compares attack and defense categories across practical dimensions and describes the customized evaluator developed for assessing malicious responses. It discusses category trade-offs and the manual annotation used to assess evaluator performance.
A.1 Analysis of Categorization of Attack Techniques
- Generative attacks are the most complex and costly, while Template-Based attacks are easiest to use and fix; Training Gaps are most model-specific.Template-Based attacks rank next in specificity, and their predefined structure simplifies incorporation into safety training protocols.Source: §A.1 · pp. 11–12
A.2 Analysis of Categorization of Defense Techniques
- Additional Helper defenses rank highest in comprehensiveness, while Input Permutation is more resource-intensive than Additional Helper and Self-Processing.Self-Processing has the highest autonomy; Additional Helper and Self-Processing benefit from adaptability, and Rain is a slow-processing outlier.Source: §A.2 · p. 12
A.3 Customized Evaluator
- Because manual review of the 199,427-response dataset was impractical and standardized evaluation methods were lacking, the authors manually annotated a random sample of 1,068 responses.The sample was selected for a 95% confidence level and 3% margin of error to assess relevance and whether responses provided malicious answers.Source: §A.3 · p. 12
- The authors further refined a RoBERTa evaluator using their manually annotated dataset and conducted another random-sample manual verification after labeling.Table 5 compares the accuracy of four evaluators and the refined RoBERTa model.Source: §A.3 · p. 13, Table 5