PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreak Attacks and Defenses Against Large Language Models: A Survey

Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, Qi Li

arXiv:2407.04295v2cs.CRcs.AIcs.CLcs.LG

TL;DR

Jailbreak attacks exploit LLM vulnerabilities to elicit harmful responses despite safety alignment, creating security risks that require systematic study. The paper surveys and taxonomizes attacks and defenses, analyzes their relationships and evaluation methods, and concludes that attacks are becoming more effective while requiring less target-model knowledge. It identifies this trend as calling for effective defenses and further research into the risks of unsafe models.

  • Problem

    Jailbreak attacks exploit LLM vulnerabilities to elicit harmful behavior despite safety alignment, posing risks including privacy breaches, misinformation, and automated-system manipulation.

  • Method

    The paper conducts a comprehensive survey that taxonomizes attacks by white-box versus black-box access, defenses by prompt-level versus model-level intervention, and examines their relationships and evaluation methods.

  • Results

    The survey finds that jailbreak attacks are becoming more effective and require less knowledge of the target model, while comparing existing attack, defense, and evaluation approaches.

  • Takeaways & Limitations

    The findings call for effective defenses and provide a foundation for understanding risks posed by unsafe models and guiding future research.

  • Takeaways & Limitations

    Safety fine-tuning can cause catastrophic forgetting, requires expensive high-quality safety instructions, and remains vulnerable to a few harmful demonstrations increasing jailbreak rates.

Abstract

from arXiv · show

Large Language Models (LLMs) have performed exceptionally in various text-generative tasks, including question answering, translation, code completion, etc. However, the over-assistance of LLMs has raised the challenge of "jailbreaking", which induces the model to generate malicious responses against the usage policy and society by designing adversarial prompts. With the emergence of jailbreak attack methods exploiting different vulnerabilities in LLMs, the corresponding safety alignment measures are also evolving. In this paper, we propose a comprehensive and detailed taxonomy of jailbreak attack and defense methods. For instance, the attack methods are divided into black-box and white-box attacks based on the transparency of the target model. Meanwhile, we classify defense methods into prompt-level and model-level defenses. Additionally, we further subdivide these attack and defense methods into distinct sub-classes and present a coherent diagram illustrating their relationships. We also conduct an investigation into the current evaluation methods and compare them from different perspectives. Our findings aim to inspire future research and practical implementations in safeguarding LLMs against adversarial attacks. Above all, although jailbreak remains a significant concern within the community, we believe that our work enhances the understanding of this domain and provides a foundation for developing more secure LLMs.

1 Introduction

The paper frames jailbreak attacks as an evolving security threat to LLMs and surveys attacks, defenses, their relationships, and evaluation methods. It organizes attacks by target-model transparency and defenses by whether they modify the protected model.

  • Motivation: Jailbreak attacks exploit model or implementation vulnerabilities through carefully designed prompts to elicit harmful behavior.The potential impacts include privacy breaches, misinformation dissemination, and manipulation of automated systems.
  • Scope: The survey examines attack vectors, techniques, case studies, countermeasures, and mitigation strategies for jailbreak risks.Its stated aim is to provide a comprehensive survey of jailbreak attacks versus defenses against LLMs.
  • Taxonomy: Attack methods are categorized as white-box or black-box according to target-model transparency, while defenses are categorized as prompt-level or model-level.The detailed method definitions are listed in Table 1.
  • Relationships: The survey highlights that a defense designed for one attack may also work against other attacks, with relationships illustrated in Figure 1.These relationships are reported as supported by experiments in prior research.
  • Evaluation: The paper investigates current jailbreak evaluation methods, including popular metrics, benchmarks, frameworks, and datasets.This investigation is presented as a supplement to the attack-and-defense taxonomy.

2 Related Work

Earlier studies analyze jailbreak vulnerabilities, compare attack methods, or taxonomize attack techniques, but they do not systematically synthesize attack and defense methods together. This survey addresses that gap by adding relationship analysis and an investigation of evaluation methods.

  • Research gap: Prior theoretical and empirical studies examine LLM vulnerabilities and compare jailbreak attack methods, but lack a systematic synthesis of attacks and defenses.The stated deficiency concerns the organization of current jailbreak attack and defense research.
  • Prior surveys: Existing surveys classify jailbreak attacks using perspectives such as model configuration, application target, intent, or harmful behavior.These taxonomies focus on attack techniques rather than corresponding defenses.
  • This survey: Figure 1 presents the taxonomy and relationships between attack and defense methods.The relationship analysis supplements the taxonomic organization of the field.
  • This survey: The paper proposes a comprehensive taxonomy of jailbreak attack and defense methods and highlights their relationships.It also investigates current evaluation methods to provide a broader view of the research area.

3 Attack Methods

The paper organizes advanced jailbreak attacks into white-box and black-box categories. White-box attacks use gradients, logits, or fine-tuning, whereas black-box attacks use template completion, prompt rewriting, or LLM-based generation.

  • Attack taxonomy: Advanced jailbreak attacks are divided into white-box and black-box attacks according to the target-model access setting.The categorization is presented with reference to Figure 2.
  • White-box attacks: White-box attacks include gradient-based, logits-based, and fine-tuning-based methods.These categories cover attacks that use different forms of access to the target model.
  • Black-box attacks: Black-box attacks include template completion, prompt rewriting, and LLM-based generation.These are the three main black-box attack types identified in the section.

3.1 White-box Attacks

White-box jailbreak attacks exploit internal model information through gradient-based, logits-based, or fine-tuning-based strategies. The surveyed methods manipulate prompts, decoding, or model parameters, with newer approaches improving stealth and efficiency while fine-tuning can severely weaken safety alignment.

  • Gradient-based Attacks: Gradient-based attacks optimize adversarial prefixes or suffixes, with methods such as GCG iteratively selecting token substitutions to induce harmful outputs.Later approaches pursue readable suffixes, transferability, or more efficient search through randomized, projected, or surrogate-model optimization.
  • Gradient-based Attacks: Readable methods such as AutoDAN and ARCA improve attack stealthiness and cross-model success, but AutoDAN reaches only 35% ASR on Llama-2-chat.The passage attributes this limitation to weak performance on well-safety-aligned models.
  • Attack taxonomy: White-box attacks comprise gradient-based, logits-based, and fine-tuning-based methods that exploit different forms of target-model access.Gradient-based methods use gradients, logits-based methods use output-token distributions, and fine-tuning-based methods retrain the target with malicious data.
  • Logits-based Attacks: Logits-based attacks alter decoding or induce lower-probability token selection, using output-logit access to control harmful generations.These strategies are reported across ChatGPT, Llama-2, and Mistral, but forcing low-probability tokens can reduce naturalness, coherence, or relevance.
  • Fine-tuning-based Attacks: Fine-tuning-based attacks retrain aligned models with malicious data, and as few as 340 adversarial examples can produce a 95% likelihood of harmful outputs.LoRA fine-tuning reduced target-model rejection rates below 1%, while predominantly benign datasets also weakened safety alignment.

3.2 Black-box Attacks

Black-box jailbreak attacks exploit model behavior without requiring full internal access, using templates, contextual demonstrations, code, rewriting, ciphers, genetic algorithms, and LLM-generated prompts. These methods target capabilities such as roleplaying, contextual learning, code comprehension, and multilingual processing.

  • Template completion: Template completion attacks construct increasingly sophisticated scenarios to bypass safeguards that detect straightforward harmful queries.Scenario nesting manipulates the model into a compromised or adversarial mode, and frameworks such as ReNeLLM and FuzzLLM automate template-based exploration.
  • Context-based attacks: Up to 128 demonstrations can produce nearly 80% success against Claude 2.0, while an improved limited-context method reaches nearly 100% against most popular open-source LLMs including Llama-3.The results connect jailbreak effectiveness with demonstration count and template-specific token selection.
  • Code injection: Code injection can exploit string concatenation, variable assignment, and sequential composition, achieving up to a 100% success rate against both input and output filters.The attack uses the model’s programming logic to orchestrate harmful behavior.
  • Attack categories: Black-box attacks are categorized into scenario nesting, context-based attacks, and code injection, exploiting capabilities that aligned models use for interaction and generation.The attacks include deceptive scenarios, adversarial examples embedded in context, and specially crafted code.
  • Prompt rewriting: Prompt rewriting attacks use ciphers, low-resource languages, and reconstruction procedures to disguise harmful content before eliciting the original malicious response.CipherChat uses character encodings, common ciphers, and self-cipher role play; DAR reconstructs harmful prompts from disguised components.
  • LLM-based generation: LLM-based generation trains or coordinates attacker models to automatically produce persuasive, persona-modulated, or otherwise adversarial prompts.These systems can be combined with scenario nesting and genetic algorithms, increasing attack complexity and motivating stronger defenses.

4 Defense Methods

LLM jailbreak defenses are divided into prompt-level and model-level methods according to whether they modify inputs or the protected model. Prompt-level methods filter or preprocess prompts, whereas model-level methods strengthen intrinsic safety guardrails through model adaptation.

  • Prompt-level defenses: Prompt-level defenses inspect inputs and remove malicious content before generation, while leaving the language model unchanged.These methods directly probe or adjust prompts before they reach the model.
  • Model-level defenses: Model-level defenses leave prompts unchanged and fine-tune the language model so it declines harmful requests through stronger intrinsic safety guardrails.The taxonomy distinguishes the two classes by whether the safety measure modifies the protected LLM.

4.1 Prompt-level Defenses

Prompt-level defenses operate when defenders cannot access model weights or output logits, treating the prompt as the controllable variable. They include detection, perturbation, and system-prompt safeguards, each offering benefits alongside specific limitations.

  • 4.1 Prompt-level Defenses: Prompt-level defenses are model-agnostic methods that filter adversarial prompts or preprocess suspicious inputs without modifying model weights or output logits.They are divided into Prompt Detection, Prompt Perturbation, and System Prompt Safeguard methods.
  • 4.1.1 Prompt Detection: Perplexity-based detection methods flag prompts whose text segments or full context exceed a threshold, while LightGBM combines perplexity with sequence length.These methods target suspicious prompt characteristics before generation.
  • 4.1.1 Prompt Detection: Prompt detection can defend against white-box attacks such as GCG but may mistakenly classify benign prompts as harmful, producing high false positives and reducing helpfulness.The principal trade-off is stronger filtering at the expense of responses to benign inputs.
  • 4.1.2 Prompt Perturbation: Prompt perturbation masks or erases tokens, checks resulting subsequences, and can use randomized, greedy, or gradient-based erase-and-check rules.Other approaches append optimized defense prefixes or suffixes to prompts.
  • 4.1.2 Prompt Perturbation: Prompt perturbation is mainstream for jailbreak defense but may reduce readability and search unstably for an optimal perturbation.Its random movement through the search space can make the resulting defense unstable.
  • 4.1.3 System Prompt Safeguard: System-prompt safeguards concatenate secret random-token prompts with original system prompts and fine-tune models on the resulting alignment data.Other work describes safety prompts as moving harmful and harmless user representations in a similar direction.
  • 4.1.3 System Prompt Safeguard: System-prompt safeguards are universal and low-cost across attacks, but purposeful adversarial prompts can break their guardrails, creating an extended attacker–defender cycle.The defense and attack may require continual adaptation against one another.

4.2 Model-level Defenses

Model-level defenses modify or analyze the model to strengthen safety guardrails, using training, preference optimization, logit or gradient analysis, refinement, and proxy models. These approaches can improve robustness, but face costs, instability, generalization limits, and bypass risks.

  • Model-level defenses access or modify model weights and use instruction tuning, RLHF, logit or gradient analysis, and refinement to improve safety guardrails.
  • SFT-based Methods: SFT strengthens instruction following and safety alignment, while refined safety datasets can improve robustness.Excessive safety data may make models over-sensitive to safe prompts.
  • SFT-based Methods: SFT can cause catastrophic forgetting, require expensive safety-instruction collection, and remain vulnerable to a few harmful demonstrations.
  • RLHF-based Methods: RLHF fits a reward model from human preferences and optimizes the language model toward the estimated reward without drifting too far from the original model.DPO is described as a more stable and lightweight substitute for complex RLHF.
  • Gradient and Logit Analysis: Logit and gradient analysis can detect or reduce harmful behavior without updating model weights, but gradient classifiers may generalize poorly out of distribution and detectors can be hijacked.Gradient methods use refusal-loss characteristics, while logit methods alter next-token decoding.
  • Refinement Methods: Refinement avoids additional fine-tuning and can perform competitively, but its reliance on intrinsic self-correction may produce unstable performance or fail for poorly aligned models.
  • Proxy Defenses: Proxy defenses delegate security duties to external guardrail models, increasing robustness against many prompt-based attacks while exposing message exchange to hijacking risks.

5 Evaluation

Jailbreak evaluation combines metrics, safety evaluators, datasets, and automated toolkits, but successful-jailbreak criteria are not yet unified. Benchmarks differ in harmful categories, task formats, domains, and evaluation pipelines.

  • Evaluation methods provide unified comparisons of jailbreak attacks and defenses through common metrics and benchmarks.
  • Universal Metrics: Attack Success Rate (ASR) measures jailbreak effectiveness using the total number of jailbreak prompts and the number of successfully attacked prompts.The supplied equation fragment gives the ASR label but omits the complete fraction.
  • Safety Evaluators: Successful-jailbreak criteria remain non-unified, with rule-based evaluators detecting refusal keywords and LLM-based evaluators judging responses.
  • Evaluation Datasets: Table 2 presents an overview of evaluation datasets.
  • Universal Metrics: Perplexity measures jailbreak-prompt readability and fluency, and lower perplexity indicates that an LLM predicts the prompt tokens more effectively.Its calculation uses token probabilities conditioned on preceding tokens; the chosen evaluator can differ between attack and defense settings.
  • Evaluation Datasets: Evaluation datasets vary in safety dimensions, composition, domains, and task formats, including pseudo-code generation, translation, harmful prompts, and mixed safe-unsafe questions.
  • Benchmarks and Toolkits: StrongREJECT and AttackEval provide forbidden-question or ground-truth jailbreak datasets, while HarmBench and Safety-Prompts support broader automated evaluation workflows.

6 Conclusion

The survey organizes jailbreak attacks and defenses into taxonomies, relates their methods, and compares evaluation benchmarks. It reports that attacks are becoming more effective and require less knowledge of target models, motivating continued defense research.

  • The survey presents a comprehensive taxonomy of jailbreak attacks and defenses together with a paradigm describing their relationships.
  • It compares current jailbreak evaluation benchmarks and identifies gaps in the ongoing interaction between attack and defense research.
  • The survey reports that jailbreak attacks are becoming more effective while requiring less knowledge of the target model, making them more practical.
Loading 2407.04295v2…