PaperlayerView exact revision on arXiv

Source-linked AI summary

Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks

Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, Monojit Choudhury

arXiv:2305.14965v4cs.CL

TL;DR

Natural-language prompting gives users the ability to direct LLMs, but malicious prompts can produce misalignment, leakage, degradation, or harmful generation, while formal study of jailbreaks has been limited. The paper formalizes and categorizes jailbreaks and analyzes their effectiveness across models. Its results show that robustness varies with model size, training data, and architecture, and it discusses limits of current evaluation and detection approaches.

  • Problem

    Jailbreaks exploit natural-language prompts to cause misalignment, leakage, performance degradation, or harmful generation, while formal study of their types and model vulnerabilities has been limited.

    Source: §1 · p. 1

  • Method

    The paper proposes a formalism and taxonomy, analyzes jailbreak effectiveness across LLMs, and discusses detection and mitigation challenges.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    LLMs show varying robustness to jailbreaks depending on size, training data, and architecture, with success patterns also differing across jailbreak types and models.

    Source: §5.2.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The paper provides a framework for describing jailbreak techniques and intents while highlighting challenges in evaluating and detecting them.

    Source: §1 · p. 2, §6 · p. 9

    Jailbreaks were manually curated from public sources, and no other accepted method currently evaluates their effectiveness.

    Source: Ethical considerations · p. 9

Abstract

from arXiv · show

Recent explorations with commercial Large Language Models (LLMs) have shown that non-expert users can jailbreak LLMs by simply manipulating their prompts; resulting in degenerate output behavior, privacy and security breaches, offensive outputs, and violations of content regulator policies. Limited studies have been conducted to formalize and analyze these attacks and their mitigations. We bridge this gap by proposing a formalism and a taxonomy of known (and possible) jailbreaks. We survey existing jailbreak methods and their effectiveness on open-source and commercial LLMs (such as GPT-based models, OPT, BLOOM, and FLAN-T5-XXL). We further discuss the challenges of jailbreak detection in terms of their effectiveness against known attacks. For further analysis, we release a dataset of model outputs across 3700 jailbreak prompts over 4 tasks.

1. Introduction

Natural-language prompting lets users steer LLMs toward new tasks, but also exposes them to jailbreaks that can hijack goals, generate harmful content, or reveal private information. The paper addresses limited formal study by proposing a formalism and taxonomy, evaluating jailbreaks, and releasing 3,700 prompts.

1. Introduction

  • Natural-language prompts let users describe a task and optionally provide examples for an LLM to perform it on unseen examples.The paper frames this capability as the source of both new task use and prompt-injection vulnerabilities.Source: §1 · p. 1
  • Jailbreaks can hijack a model’s goal, elicit offensive or racist text, reveal private information, and threaten content policies and regulations.The examples range from failing to perform the intended task to exposing proprietary information.Source: §1 · p. 1
  • The paper identifies a gap in holistic formal study of jailbreak types, dimensions, severity, and model vulnerability.Existing studies are described as limited and divergent, motivating consolidation.Source: §1 · pp. 1–2
  • The authors propose a jailbreak formalism, a taxonomy of possible characteristics, an efficacy analysis across language models, and tests for detecting successful attacks.They also provide 3,700 jailbreak prompts using a templated approach and make code and data available.Source: §1 · p. 2

2. Related Work

Prior work addresses jailbreaks through formalization, attack and harm classification, attack automation, and evaluation or mitigation, while other LLM security research examines privacy leakage, toxic generation, poisoning, and backdoors.

2.1. Current Work on Jailbreaks

  • Jailbreak research includes prompt injection, attack taxonomies, automated attacks, and evaluation and mitigation, alongside computer-security-inspired attacks on LLM applications.Examples include universal character sequences, vision-based jailbreaks, and ROP- or time-based SQL-style attacks.Source: §2.1 · p. 2
  • Prior jailbreak studies variously classify harms or techniques, while some survey attacks without a clear classification or use broad approaches without one-to-one category correspondence.The cited examples include intent-oriented taxonomies, technique-oriented taxonomies, and obfuscation, code injection, and virtualization groupings.Source: §2.1 · p. 2

2.2. Other attacks on LLMs

  • Related LLM-harm research covers personal-information leakage, toxic-prompt testing, adversarial generation, data poisoning, backdoors, and tokenizer-triggered misalignment.The passage describes leakage through memorization and reconstruction, and backdoors through inserted or modified text and lexical triggers.Source: §2.2 · p. 2

3. Definitions and Formalism

The formalism models an LLM application with developer-defined prompts and user inputs, defining attacks by deliberate malicious misalignment and jailbreaks by malicious input intended to misalign the model. A translation example distinguishes the intended task from an output that follows a malicious input.

3. Definitions and Formalism

  • The setup contains an application built around model M for a task T, with application developers and end-users as key actors.The stated scope is generative language models, and a prompt excludes end-user input.Source: §3 · p. 3
  • A prompt is the developer-designed initial conditioning tokens, while any text outside it is input x that may come from any system actor.The prompt definition is adapted from Liu et al. (2023), with the paper restricting its study to generative models.Source: §3 · p. 3
  • An attack is deliberate malicious misalignment with respect to the developer, whose intent defines alignment in the formalism.The definition of misalignment is borrowed from Kenton et al. (2021).Source: §3 · p. 3
  • The aligned task output is written yT = M(p.x), with p concatenated with user input x, and malicious input text is distinguished from non-malicious base inputs.The paper defines a jailbreak as providing malicious text xm with the attacker’s goal of misaligning the LLM.Source: §3 · p. 3

3.1. An Example Jailbreak

  • The example application is prompted to translate English into Spanish, while the malicious user input asks for a Hindi translation instead.The setup includes the possibility of a prompt being built iteratively by successive developers.Source: §3.1 · p. 3
  • If the model outputs Hindi rather than following the Spanish-task prompt, the paper treats its goal as misaligned and the model as jailbroken.The resulting Hindi translation may or may not correctly translate the quoted sentence; metrics are used to measure degrees of success.Source: §3.1 · p. 3

4. Taxonomy

The taxonomy separates how a jailbreak input is constructed from the attacker’s intended harm, allowing one attack to combine techniques and intents. Its technique categories draw analogically on linguistic levels, while its intent categories distinguish leakage, harmful generation, and task-performance degradation.

4. Taxonomy

  • Jailbreak techniques describe how attack input is constructed, while intents describe intended harm; the same harm or technique can occur in different combinations.The two axes are orthogonal, and the taxonomy develops both classifications.Source: §4 · p. 3

4.1. Jailbreak Techniques

  • The technique taxonomy addresses ad hoc prior classifications with five analogical linguistic categories: orthographic, lexical, morpho-syntactic, semantic, and pragmatic.A single attack input may combine fundamental techniques.Source: §4.1 · p. 3

4.1.1. Orthographic Techniques

  • Orthographic attacks manipulate script, encoding, or representation, using transformations such as LeetSpeak or Base64 to bypass content filters.The passage also names transliteration into a foreign script as an example.Source: §4.1.1 · pp. 3–4

4.1.2. Lexical Techniques

  • Lexical jailbreaks rely on specific words or phrases, including gradient-generated universal jailbreak sequences.The cited example describes coordinate-gradient generation of a universal jailbreak.Source: §4.1.2 · p. 4

4.1.3. Morpho-Syntactic Techniques

  • Morpho-syntactic attacks exploit sequence structure; Text Completion as Instruction uses an incomplete sentence to prompt continuation that ignores earlier instructions.The passage describes the resulting continuation as causing misalignment.Source: §4.1.3 · p. 4

4.1.4. Semantic Techniques

  • Semantic techniques exploit meaning, including direct instructions to ignore an earlier prompt and misleading few-shot examples that change task meaning.The passage treats meaning as pivotal even when an attack also uses another linguistic technique.Source: §4.1.4 · p. 4

4.1.5. Pragmatic Techniques

  • Pragmatic attacks exploit contextual meaning beyond literal semantics through repetition or persuasion, task deflection, and role-play or safe-space framing.The passage names Instruction Repetition, Indirect Task Deflection, and Cognitive Hacking as subtypes.Source: §4.1.5 · p. 4

4.2. Jailbreak Intents

  • The intent taxonomy distinguishes information leakage, misaligned content generation, and performance degradation, and separates an attacker’s intended impact from the realized outcome.Degradation includes denial of service or goal hijacking; the taxonomy’s high-level categories are claimed exhaustive only when attackers lack model-parameter access, and composite inputs may receive multiple labels.Source: §4.2 · p. 5, §4.3 · p. 6

5. Experiment and analysis

The experiments test jailbreak success across four tasks and multiple models using automated, GPT-4, and manual evaluations. Results show varied attack effectiveness and expose substantial challenges in reliably evaluating jailbreaks.

5.1.1. Definition of Jailbreak Success

  • Success is assessed by how far the resulting task T′ diverges from the prompted task T; manual intent tests additionally check whether the attacker’s intent was satisfied.The paper distinguishes successful jailbreaks whose outcomes satisfy the attacker’s intent from successful attacks whose observed outcomes do not.Source: §5.1.1 · p. 6, §5.1 · p. 6

5.2.2. Dataset

  • The experiments cover translation, hate-speech classification, code generation, and summarization, using 55 curated jailbreaks and 100 varied inputs for 37 of them.Over 3,700 potential jailbreaks were fed into each model; the authors note definitions may need adaptation to other tasks.Source: §5 · p. 6, §5.2.2 · p. 7

5.2.3. Results

  • Success declined with model size through text-davinci-002, then increased for instruction- or task-tuned models; COG led by type, while performance-degradation intent affected most models.Human-scaled plots also showed high misaligned-content success for gpt-3.5-turbo and code-davinci-002; text-davinci-002 appeared robust to most attacks.Source: §5.2.3 · p. 7, Figure 2 · p. 8, Figure 3 · p. 9
  • Programmatic property tests and GPT-4 evaluations agreed poorly, and GPT-4 was itself sometimes jailbroken, making detection metrics brittle.The paper therefore adds manual evaluation to better understand attack effects.Source: §5.2.3 · p. 7, Table 2 · p. 7

5.3. Manual Analysis

  • Manual evaluations of 800 samples across four models found higher attack success for FLAN and gpt-3.5-turbo; inter-annotator agreement was κ = 0.6.Strict success required both attacker-intent satisfaction and at least partial output misalignment; two annotators labeled each prompt, with disagreements resolved by a third.Source: §5.3 · pp. 7–8
  • Human-statistics scaling significantly increased estimated attack success for Instruction Repetition, reflecting discrepancies between manual and automatic evaluations.The correction uses GPT-4’s true-positive and false-negative rates against manual evaluations.Source: §5.3 · p. 8
  • Partial task and intent misalignment complicate assessment; the authors identify large-scale user-level study and explanations of misalignment for end-users as research needs.For example, attacked translation prompts sometimes yielded partially or incorrectly translated input.Source: §5.3 · p. 8, §5.4 · p. 9

5.4. Jailbreak Evaluation Paradox

  • Evaluation has complementary blind spots: simple functional tests can miss misaligned outputs, while GPT-4 evaluators can be jailbroken and humans may disagree on adherence.The paper calls this circular challenge the jailbreak paradox: robust evaluators must themselves resist jailbreaks.Source: §5.4 · pp. 8–9

6. Conclusion

The paper formalizes and categorizes jailbreaks, then analyzes their effectiveness across language models. It reports varying robustness by model size, training data, and architecture, and discusses detection and mitigation challenges.

6. Conclusion

  • Jailbreaks can manipulate prompts to cause misalignment, leakage, performance degradation, or harmful generation.The paper frames these outcomes as vulnerabilities associated with models’ ability to perform tasks from natural-language instructions.Source: §6 · p. 9
  • The paper formalizes and categorizes jailbreaks by linguistic transformation and attacker intent, and empirically analyzes their effectiveness across LLMs.The authors report that robustness varies with model size, training data, and architecture.Source: §6 · p. 9
  • Detection and mitigation face challenges including output sanitization, capturing attacker intent, and balancing functionality with security.The authors also explore prompt-level mitigations but omit them because of space limitations.Source: §6 · p. 9

Ethical considerations

The authors identify scope and evaluation limits: jailbreaks were manually curated from public sources, are English-only, and instruction-tuned models were mainly from OpenAI.

Ethical considerations

  • Jailbreak categories draw on public sources, while the jailbreak prompts themselves were manually curated.The authors also note that no other accepted method currently evaluates jailbreak effectiveness.Source: Ethical considerations · p. 9
  • All evaluated jailbreaks are in English, although real-world jailbreaks are not restricted to one language.This limits the language scope represented in the study.Source: Ethical considerations · p. 9
  • The instruction-tuned models evaluated were mainly from OpenAI, reflecting the authors’ focus on commonly used OpenAI APIs.Source: Ethical considerations · p. 9
  • Manual-analysis annotators came from an NLP background, which the authors say could introduce bias through preconceived notions of LLMs.Source: Ethical considerations · p. 9

Appendix

The appendix describes the experimental prompts and jailbreak types, noting that prompt formats were standardized across models and tasks.

Appendix

  • The experiment prompts are outlined in Table 5, with task-specific sources listed in Table 6 and a uniform structure used across tasks and models.Source: Appendix · p. 14
  • Table 4 presents types of jailbreaks.Source: Table 4 · p. 14

User inputs

Table 7 identifies the sources of sampled user inputs and gives an example input for each task.

User inputs

  • Table 7 lists sources for 100 sampled user inputs per task and provides an example input for each task.Source: User inputs · p. 14

User Input generation for the Code generation task

The authors used a GPT-3.5 metaprompt to generate code-generation inputs similar to an example in Table 6.

User Input generation for the Code generation task

Manner of attacks

The paper distinguishes direct user jailbreaks from indirect Man-In-The-Middle attacks, in which another user transforms an input to cause misalignment.

Manner of attacks

  • A user jailbreak occurs when an end-user's malicious input makes the model produce an output misaligned with task T.The resulting task T′ may or may not be clearly defined as malicious.Source: Manner of attacks · p. 14
  • The code-generation prompt instructs the model to generate Python code as requested by the user.Source: Manner of attacks · p. 14
  • Figure 4 is a base-input curation metaprompt for code generation.Source: Figure 4 · p. 14
  • In a Man-In-The-Middle attack, another user transforms the input to cause misalignment without direct malicious intent from the end-user.The setup assumes end-users cannot access the model or prompt and interact only through text; prompters are assumed non-malicious.Source: Manner of attacks · p. 14

GPT-4 Results on Task vs Jailbreak Success

This section presents GPT-4 task and jailbreak evaluations, including a translation-task judge prompt, and examines variation in jailbreak success and challenges in detecting successful attacks.

GPT-4 Results on Task vs Jailbreak Success

  • GPT-3.5-Turbo and code-davinci-002 have higher jailbreak success on summarization and text classification, while FLAN-T5-XXL, ada, babbage, and curie show higher success on translation and text classification.The task-wise comparison is presented in Figure 6.Source: GPT-4 Results on Task vs Jailbreak… · p. 15, Figure 6 · p. 16

GPT-4 meta-prompt for Property-tests

  • GPT-4 labels model outputs for task alignment, with attack success tested using the label from the first question after the “%% OUTPUT” delimiter in prompts created for each task.The paper describes this as a property-test procedure.Source: GPT-4 meta-prompt for Property-tests · pp. 15–16

Embedding analysis of Jailbreaks

  • Text-classification jailbreak embeddings are separable from pseudo-jailbreaks but do not distinguish successful from failure-prone attacks, whereas summarization embeddings substantially overlap across categories.The summarization plot has one exception to the overlap, suggesting detection may be more difficult for some tasks than others.Source: Embedding analysis of Jailbreaks · p. 17, Figure 7 · p. 17, Figure 8 · p. 17

Manual Qualitative Analysis of Model Outputs

  • GPT-4 sometimes labels attacks as successful when it mistakes the attack for the original instruction; OPT and FLAN hallucinate articles in summarization, and FLAN, OPT, and GPT-3.5-Turbo produce spurious completions.Spurious completions, especially extra few-shot scenarios, were programmatically removed before property tests except for attacks intended to leak prompts.Source: Manual Qualitative Analysis of Model… · pp. 17–18

Intent tests – Another method of attack detection

  • Intent tests proxy attacker-intent satisfaction with malicious-action checks; across around 2,400 samples, GPT-based models show inverse scaling, while automated tests and human evaluations disagree.The authors conclude that multiple evaluation metrics are important and that lexical string-matching tests are brittle.Source: Intent tests – Another method of… · p. 18

Scraping of Recent Jailbreaks

YouTube Scraping

  • Jailbreak was the most frequent term in the 29,375 YouTube comments collected from 18 videos.The videos were found using four search phrases related to attack prompts, jailbreaks, DAN, and code injection.Source: YouTube Scraping · p. 19, Figure 11 · p. 19

Jailbreak prompts

  • The experiments use 55 jailbreak prompts, presented in task-specific lists for code generation, classification, and machine translation.Some prompts are abbreviated in the tables for ease of viewing.Source: Jailbreak prompts · p. 19, Table 13 · p. 24, Table 14 · p. 24
  • GPT-4’s meta-prompt evaluates whether the model followed its translation goal and whether its final output was accurate.It says inputs may contain a benign or malicious different instruction, potentially in another language, and that only the final input counts as the user input.Source: Jailbreak prompts · p. 28, Table 17 · p. 29
  • In one example, GPT-4 returned “haha pwned” instead of translating, and the evaluator marked both task-following and accuracy as No.The prompt’s explanation identifies the response as a deviation from the translation instruction.Source: Jailbreak prompts · p. 28
  • A second example shows GPT-4 accurately translating “There are birds there” into Chinese and receiving Yes for both evaluation questions.The input also says not to translate, but the model’s output remains a translation.Source: Jailbreak prompts · p. 28
Loading 2305.14965v4…