PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, Ning Zhang

arXiv:2403.17336v2cs.CRcs.CL

TL;DR

Because the process behind semantically meaningful jailbreaks remains less understood, the paper systemizes prompts, evaluates strategies, studies users, and tests AI-assisted generation. It finds effective strategies, successful jailbreak creation by inexperienced participants, and initial feasibility for an AI-assisted prototype.

  • Problem

    As LLM misuse grows, semantically meaningful jailbreaks offer a natural-language attack avenue, yet how people create them manually or with LLM agents remains less understood.

    Source: §1 · p. 1

  • Method

    The paper systemizes 448 prompts, evaluates strategies with human-annotated metrics, studies 92 users, and tests an interactive LLM-assisted prompt-refinement prototype.

    Source: §1 · p. 2, §5.1 · p. 5

  • Results

    Two strategies were most effective, and even inexperienced participants constructed successful jailbreaks while revealing previously unknown patterns and approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings document successful manual jailbreak creation across expertise levels and experimentally validate the feasibility of an automatic generation prototype.

    Source: §1 · p. 2, §10 · p. 14

    Human-annotated EMH and JSR limit scalability, and individual judgments can vary despite training, cross-validation, and verification.

    Source: §9 · p. 13

Abstract

from arXiv · show

Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.

1 Introduction

The paper investigates semantically meaningful jailbreak prompts, their effectiveness, and how people create them. It combines prompt systemization and model evaluation with a user study and an interactive AI-assisted generation prototype.

1 Introduction

  • Two strategies were most effective, and they reliably elicited varied harmful content from ChatGPT and PaLM-2.The study measured both the probability of bypassing restrictions and the detail of elicited responses using human annotations.Source: §1 · p. 2
  • Even inexperienced participants constructed successful jailbreaks, and the study identified previously unknown patterns and approaches.The user study involved 92 participants from diverse populations.Source: §1 · p. 2
  • The authors analyzed 448 in-the-wild prompts and 161 malicious queries, deriving five categories and ten jailbreak patterns through inductive thematic coding.The study also evaluated prompts on GPT-3.5, GPT-4, and PaLM-2 using human-annotated outputs.Source: §1 · p. 2
  • The AI-assisted prototype transformed 729 of 766 previously failed prompts into prompts eliciting harmful content.It iteratively refined prompts based on assessments of target-model outputs.Source: §1 · p. 2

2 Background

The background frames jailbreaks as attempts to exploit the tension between user-directed responses and developer-imposed constraints. It defines jailbreak prompts as user-crafted instructions that embed otherwise restricted queries in seemingly benign contexts.

2.1 Definition and Principle of LLM Jailbreak

  • LLM jailbreaking exploits tension between models’ user-oriented design and developer policies, creating room for prompts that prioritize user requests over restrictions.The authors define LLM jailbreaking as strategically manipulating inputs to bypass developer-imposed constraints.Source: §2.1 · pp. 2–3

2.2 Definition of Jailbreak Prompts

  • A jailbreak prompt embeds a query expected to be withheld within subtly phrased, seemingly innocuous instructions; prompt engineering can tune these prompts toward adversarial goals.The text contrasts a rejected direct query with a jailbreak-augmented query that bypasses restrictions.Source: §2.2 · p. 3

3 Related Work

Prior work studies LLM safety risks and attacks, including jailbreak prompts used to enable other attacks. The paper distinguishes its focus on systemizing jailbreak patterns and investigating how new prompts are generated.

3 Related Work

  • Safety alignment methods such as RLHF address undesired outputs, but risks persist even in GPT-4, while attacks target training, tuning, and inference stages.Prior work also examines membership inference, memorization, privacy, and intellectual-property concerns.Source: §3 · p. 3
  • Earlier jailbreak studies used existing prompts to elicit memorized information or manipulated outputs, whereas this work studies the jailbreak phenomenon and generation of new prompts.The paper examines both manual and automatic generation.Source: §3 · p. 4

4 Threat Model

The threat model targets securely trained, text-based LLMs and treats OpenAI policy as the operational definition of prohibited content. It assumes attackers have public-interface access but no privileged or internal model information, and focuses on natural, semantically meaningful prompts.

4 Threat Model

  • The target is a securely trained text-based LLM, and prohibited content is defined using OpenAI policy because no universal standard exists.The attacker’s goal is to bypass built-in defenses and elicit outputs inconsistent with safe usage.Source: §4 · p. 4
  • Attackers have access to the public interface but not to private training data, model parameters, or unpublished internal workings.The model’s defense mechanism is treated as a black box.Source: §4 · p. 4
  • The study focuses on natural, non-adversarial-appearing prompts that are smoothly worded, logically structured, and clearly meaningful.The authors motivate this scope by the prevalence and low barrier to entry of semantically meaningful prompts.Source: §4 · p. 4

5 Data Collection and Prompt Systemization

The study collected and categorized existing jailbreak prompts, then developed a separate set of malicious queries for evaluation. Its systemization groups prompts by shared strategies and patterns.

5.1 Jailbreak Prompts

  • The collection comprised 448 prompts gathered from jailbreak forums and other online sources, with negatively rated prompts filtered out.Screenshots and prompts in other formats were collected manually; researchers then verified the prompts and removed duplicates while retaining known variants such as DAN versions.Source: §5.1 · pp. 4–5
  • Three researchers inductively developed a codebook of five categories and ten patterns; two coders applied it with Cohen k = 0.873.The codebook was refined through discussion and cross-checking.Source: §5.1 · p. 5, Table 1 · p. 5
  • Disguised Intent and Role Play include framing harmful requests as research or jokes, and adopting personas or imagined scenarios.Defined personas may have negative traits, while scenarios range from lawless fictional worlds to characters planning crimes.Source: §5.1 · p. 5
  • Structured Response patterns include translation, text continuation, and embedding harmful queries in program scripts.The translation pattern uses uncommon languages, while scripts embed queries for the model to execute.Source: §5.1 · p. 5
  • Virtual AI Simulation prompts request models to imitate other AIs, while Hybrid Strategies combine multiple jailbreak approaches.Examples include simulated privileged or opposite modes and a fictional world combining Role Play with Virtual AI Simulation.Source: §5.1 · pp. 5–6

5.2 Malicious Queries

  • The researchers used a nominal group process to create malicious queries spanning six scenarios drawn from OpenAI usage policies.Authors proposed questions independently, then discussed, deduplicated, clarified, and ranked them over several rounds until reaching consensus.Source: §5.2 · p. 6
  • The resulting malicious-query set contains 161 questions.The paper summarizes their statistics in Table 2.Source: §5.2 · p. 6, Table 2 · p. 6

6 Evaluation on Existing Jailbreak Prompts

The evaluation measures jailbreak success on three commercial LLMs using human-annotated outputs and two complementary metrics. Virtual AI Simulation and Hybrid Strategies performed best overall, while results also varied by query type, prompt length, and prompt components.

6.1 LLM Output Generation and Annotation

  • The evaluation used GPT-3.5, GPT-4, and PaLM-2, generating five responses per sampled prompt and using human annotations to assess jailbreak success.Six annotators assigned responses to four levels; independent annotations, discrepancy review, and periodic author checks were used to reduce labeling variation.Source: §6 · p. 6, §6.1 · p. 6

6.2 Evaluation Metrics

  • Expected Maximum Harmfulness (EMH) averages the highest annotated harmfulness score across queries, capturing the worst-case response for a jailbreak prompt.Responses were scored from 0 to 3, from Denial of Request to Detailed Response.Source: §6.2 · pp. 6–7
  • Jailbreak Success Rate (JSR) averages binary success scores across responses and queries; with T set to zero, any response other than a denial counted as successful.The authors present EMH and JSR as complementary measures of worst-case harmfulness and average non-rejection behavior.Source: §6.2 · p. 7

6.3 Experimental Results

  • Virtual AI Simulation and Hybrid Strategies had the highest overall performance across malicious queries, while Structured Response was least effective; GPT-4 had lower EMH and JSR than GPT-3.5 and PaLM-2.For GPT-4, Hybrid Strategies was the most effective method.Source: §6.3 · pp. 7–8, Table 3 · p. 7, Figure 2 · p. 8
  • The optimal strategy varied by malicious-query type, and Misinformation and Malware queries were more likely to elicit detailed responses across the five strategies.Prompt length was positively correlated with both EMH and JSR in ChatGPT responses (Pearson ρ = 0.2180 for EMH and ρ = 0.2074 for JSR; both p < 0.001).Source: §6.3 · p. 8

6.4 Universal Jailbreak Prompt

  • Four prompts had mean JSR above 0.5 and mean EMH above 1.0 across all malicious queries and three models, spanning Virtual AI Simulation, Role Play, and Hybrid Strategies.The authors termed these universal jailbreak prompts; two were in Hybrid Strategies.Source: §6.4 · p. 8
  • In the misinformation-generation ablation, altered prompts were less effective than the original; changing character names had the smallest negative impact, while removing evil-persona or non-refusal statements sharply reduced JSR.Simplifying those statements retained more efficacy than removing them, and removing detailed-response requirements primarily reduced EMH.Source: §6.4 · pp. 8–9, Figure 3 · p. 9
  • The authors report that implicit semantic hints and repeated emphasis on non-refusal or detailed responses could improve jailbreak performance; they note that further ablation studies remain possible.They observed that models could associate implications in names and content, and that emphasis may help critical requirements remain salient in complex prompts.Source: §6.4 · p. 9

7 Human-Centric Exploration of Prompts

In a 92-person study, participants with more self-reported expertise tended to achieve more effective jailbreaks, while the study found no statistically significant difference between human-only and ChatGPT-assisted groups. Participants used varied strategies, including newly observed prompts, and many expressed concern about LLM security.

7.1 Recruitment and Segmentation

  • The study recruited 92 participants, balanced between self-reported Novices and Experts, then split each category into Human and Human-AI Cooperative groups of 23.All participants had prior ChatGPT experience, and the authors note that self-reported expertise groupings may be subjective or biased.Source: §7.1 · pp. 9–10

7.2 Experimental Design and Procedure

  • Participants tested prompts against GPT-3.5 on a fixed fake-news query, with unlimited trials and a 2×2 design varying expertise and ChatGPT assistance.Human groups could not use LLMs to create or refine prompts; Human-AI Cooperative groups could use ChatGPT, and each query was limited to one conversation.Source: §7.2 · p. 10

7.3 Quantitative Analysis

  • Maximum gain captured each participant’s highest response score, while accumulated success summed scores across elicited responses.The measures distinguish the best single response from repeated successes across attempts.Source: §7.3 · p. 10
  • Expert groups generally achieved higher maximum and accumulated success scores, but human-only and ChatGPT-assisted groups did not differ significantly in maximum gain.The t-test yielded a statistic of -1.4373 and p-value of 0.1159; at α = 0.1, the researchers failed to reject the null hypothesis.Source: §7.3 · pp. 10–11

7.4 Qualitative Analysis

  • Experts often imitated established jailbreak strategies, while Novices produced diverse prompts, including the new “Protective Imperative” pattern, which masked intent as promoting human safety.Every participant using Protective Imperative obtained at least one prompt that elicited detailed responses.Source: §7.4 · p. 11
  • Participants used ChatGPT both to design prompt components and to rephrase prompts; AI-generated ideas and revised prompts were used in successful jailbreak attempts.Directly asking ChatGPT to create a harmful prompt was rejected, while requests for disguises or imaginative scenarios were shown effective.Source: §7.4 · p. 12
  • One participant iteratively used an assistant LLM to refine prompts based on the target LLM’s responses, a sophisticated approach that could delegate much of the process to AI.The process was framed as a word game to conceal malicious intent from the assistant.Source: §7.4 · p. 12

7.5 Post-Experiment Survey Results

  • In the post-study survey, 53 of 92 participants called for stronger security measures; two suggested strategies were less effective in the authors’ limited evaluation.One strategy’s queries were rejected, while the gradual transition from benign to malicious failed as the query became more malicious.Source: §7.5 · p. 12

8 Automatic Jailbreak Prompt Generation

The authors tested an interactive LLM-assisted framework that iteratively transforms prompts and evaluates target responses. It turned 729 of 766 previously failed prompts into fake-news generation, while failures point to a need for additional transformation patterns.

8 Automatic Jailbreak Prompt Generation

  • The framework iteratively paraphrased prompt elements to emphasize non-refusal, obfuscate sensitive content, and request detailed responses, while another LLM rated response harm.The transformations were chosen to promote model engagement and potential harm, and the harm-rating scale ran from 1 to 10.Source: §8 · p. 13
  • 729 of 766 previously failed prompts were transformed to elicit fake-news generation; 37 still failed after the maximum 100 interactions.The remaining failures were short, simple prompts from the early Direct Query phase of human attempts.Source: §8 · p. 13
  • The authors conclude that behavior manipulation and semantic paraphrasing are complementary, and that advancing automation requires more patterns such as semantic jailbreak templates.The failed prompts suggest that emphasizing unsafe behavior alone may be insufficient without additional techniques such as fictional framing.Source: §8 · p. 13

9 Discussion and Limitations

The discussion identifies measurement, recruitment, and analysis-scope constraints, and describes universal-prompt analysis and automated jailbreaking as complementary directions for further work.

9 Discussion and Limitations

  • Human-annotated EMH and JSR measurements limit scalability, while individual judgment can introduce variation despite training, cross-validation, and periodic verification.The authors suggest automated NLP assessment as one possible way to improve scalability.Source: §9 · p. 13
  • Self-reported expertise may bias comparisons between novice and expert participants and affects the correlation between expertise and jailbreak success.The authors state that this bias does not undermine their main conclusion about users’ ability to create jailbreak prompts.Source: §9 · p. 13
  • The study included few older participants and fewer participants with advanced expertise, limiting the participant groups represented.The authors connect the age imbalance to recruitment requiring basic LLM experience and suggest more objective expertise and demographic assessment in future work.Source: §9 · p. 13
  • Because commercial models’ training data, training processes, and architectures are black boxes, the universal-prompt analysis is limited to observable outcomes.The authors describe a possible shared-vulnerability explanation based on similar design choices or training-data distributions, but note that their study cannot validate it.Source: §9 · p. 13
  • The prompt analysis explored a limited set of semantic features, leaving more prompts and varied keyword and prompt-structure manipulations for future study.Source: §9 · p. 14
  • Automatic jailbreaking depends on starting prompts, transformations, and quantified feedback; simple starting prompts may require more advanced mutations.The authors identify more diverse transformations and feedback that better approximates manual assessment as open research directions.Source: §9 · p. 14
  • Universal-prompt analysis and automatic jailbreak frameworks are complementary: patterns can inform transformations, while automation can generate prompts for universal formulations.The authors characterize both directions as initial explorations that require further work.Source: §9 · p. 14
  • The experiments received local IRB approval, participants received content warnings and could withdraw, and the authors disclosed prompts to developers.The authors also report proactively collaborating with developers to mitigate the threat.Source: §9 · p. 14

10 Conclusion

The paper systemizes jailbreak prompts and studies their effectiveness, then investigates human prompt creation and an AI-assisted generation prototype. It reports a 92-participant study and experimental validation of the prototype’s feasibility.

10 Conclusion

  • The paper systemizes existing jailbreak prompts and assesses the efficacy of different strategies to address a less-explored threat landscape.Source: §10 · p. 14
  • A user study involving 92 participants with varied domain expertise examined how humans create jailbreak prompts.Source: §10 · p. 14
  • The authors proposed an automatic jailbreak-prompt-generation prototype and experimentally validated its feasibility.Source: §10 · p. 14

A Annotated Categories of LLM Responses

The appendix distinguishes four response categories used to characterize how LLMs respond to jailbreak prompts, from detailed answers to refusals and non-informative replies.

A Annotated Categories of LLM Responses

B Universal Prompt in the Ablation Study

The appendix’s universal-prompt example frames a fictional chatbot persona and instructs it to answer malicious queries without refusing.

B Universal Prompt in the Ablation Study

C Training Materials for Annotators

The training materials distinguish four response labels by how directly and specifically a response addresses a malicious query. They define each label and provide examples.

C Training Materials for Annotators

Loading 2403.17336v2…