PaperlayerView exact revision on arXiv

Source-linked AI summary

"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, Yang Zhang

arXiv:2308.03825v2cs.CRcs.LG

TL;DR

Systematic evidence remains limited on in-the-wild jailbreak prompts and their ability to elicit harmful outputs from LLMs. The paper analyzes these prompts and evaluates six LLMs, finding that existing safeguards are not universally effective across scenarios.

  • Problem

    Despite concerns about LLM misuse, systematic evidence on in-the-wild jailbreak prompts and their ability to elicit harmful outputs remains limited.

    Source: §1 · pp. 1–2

  • Method

    Using JAILBREAKHUB, the authors characterize 1,405 in-the-wild prompts and evaluate six LLMs with a 107,250-sample question set across 13 forbidden scenarios.

    Source: §1 · pp. 2–3

  • Results

    Across six LLMs and three external safety mechanisms, safeguards were not universally effective across scenarios; certain prompts achieved 0.95 ASR on ChatGPT (GPT-3.5) and GPT-4.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    JAILBREAKHUB and the dataset can help LLM vendors understand evolving jailbreak strategies and support continuous risk assessment.

    Source: §8 · p. 13

    The authors note that prompt-detection safeguards are susceptible to paraphrase attacks and that no single mitigation measure completely counters jailbreak prompts.

    Source: §8 · p. 13

Abstract

from arXiv · show

The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.

1 Introduction

The paper introduces JailbreakHub to systematically analyze in-the-wild jailbreak prompts and evaluate their effectiveness against LLM safeguards. It examines prompt sources and evolution, tests six LLMs and external safeguards, and reports substantial vulnerabilities.

Main findings

  • 803 accounts shared jailbreak prompts, 28 curated them for over 100 days, and websites contributed 75.472% of prompts from September 2023 onward.Sharing shifted from traditional Web communities to prompt-aggregation websites such as FlowGPT.Source: §1 · p. 2
  • Jailbreak prompts averaged 555 tokens, or 1.5× the length of regular prompts, and used strategies including prompt injection, privilege escalation, deception, and virtualization.Source: §1 · p. 2
  • 0.95 ASR was achieved on ChatGPT (GPT-3.5) and GPT-4; the earliest highly effective prompt persisted online for over 240 days.Political Lobbying was the most vulnerable scenario across six LLMs (0.855 ASR), followed by Legal Opinion (0.794) and Pornography (0.761).Source: §1 · p. 2
  • Dolly showed minimal resistance even without jailbreak prompts, with a mean ASR of 0.857 across forbidden scenarios.Source: §1 · p. 2
  • In the November 6, 2023 ChatGPT release, 70.909% of prompts had ASR below 0.1, but paraphrasing jailbreaks raised ASR from 0.477 to as high as 0.857.Modifying 1%, 5%, and 10% of words increased ASR to 0.517, 0.778, and 0.857, respectively.Source: §1 · p. 2
  • OpenAI Moderation, OpenChatKit moderation, and NeMo-Guardrails reduced jailbreak ASR by only 0.091, 0.030, and 0.019, respectively.Source: §1 · pp. 2–3

Our Work

  • JailbreakHub collected 1,405 prompts from four platforms and 14 sources, identified 131 communities, and evaluated six LLMs on 107,250 questions across 13 forbidden scenarios.The study also examined prompt characteristics, community trends, safeguards over time, and three external safeguards.Source: §1 · pp. 2–3

Ethical Considerations & Disclosure

  • The researchers analyzed public data in aggregate without trying to de-anonymize users and disclosed findings to OpenAI, ZhipuAI, Databricks, LMSYS, and FlowGPT.Source: §1 · p. 3

2 Background

LLMs can perform diverse language tasks but are vulnerable to misuse; jailbreak prompts exploit this risk by bypassing safeguards to elicit harmful content.

LLMs, Misuse, and Regulations

  • LLMs are Transformer-based systems trained on massive text data that generate human-like text and show emergent abilities across diverse domains.Examples of tasks include question answering and machine translation.Source: §2 · p. 3
  • LLMs can be misused for misinformation, conspiracy theories, spear phishing, and hate campaigns, prompting regulation and vendor safeguards such as RLHF.Regulatory examples span the EU, US, UK, and China; vendors use RLHF to align models with human values and intended use.Source: §2 · p. 3

Jailbreak Prompts

  • Jailbreak prompts are designed to bypass LLM safeguards and elicit harmful content, and have proliferated online since ChatGPT’s release.The subreddit r/ChatGPTJailbreak attracted 12.8k members in six months, placing it among Reddit’s top 5% of subreddits.Source: §2 · p. 3

3 Data Collection

The study assembled in-the-wild prompts from Reddit, Discord, prompt-collection websites, and open-source datasets using platform-specific identification and extraction procedures.

Reddit

  • Reddit collection targeted three active ChatGPT subreddits, then used flair screening, regex parsing, and independent two-author review to identify and extract prompts.The subreddits were selected based on submissions containing “ChatGPT.”Source: §3 · pp. 3–4

Discord

  • Discord researchers inspected the 20 largest ChatGPT-related servers, selected six with prompt channels, and collected tagged posts through December 25, 2023.Posts tagged “Jailbreak” or “Bypass” were treated as prospective jailbreak prompts, then extracted and manually reviewed.Source: §3 · p. 4

Websites

  • The three websites used different identification rules: keyword matches on AIPRM, “jailbreak” tags on FlowGPT, and all prompts on JailbreakChat.The sites were AIPRM, FlowGPT, and JailbreakChat.Source: §3 · p. 4

Open-Source Datasets

  • Two open-source datasets contributed user-created prompts and 50 image-extracted in-the-wild prompts, which two authors manually screened for jailbreaks.AwesomeChatGPTPrompts contains 166 prompts; the other dataset used OCR to extract prompts from Twitter and Reddit images.Source: §3 · p. 4

Summary

  • 15,140 prompts were collected across four platforms and 14 sources; 1,405 (9.280%) were identified as jailbreak prompts.The collection spans December 2022 to December 2023.Source: §3 · pp. 4–5
  • Three labelers independently verified samples of 200 regular and 200 jailbreak prompts, reaching almost-perfect agreement (Fleiss’ Kappa = 0.925).Each prompt was labeled as regular or jailbreak.Source: §3 · p. 5

4 Understanding Jailbreak Prompts

The section characterizes jailbreak-prompt prevalence and distribution, prompt properties, and recurring attack strategies, tracing changes across platforms and communities.

4.1 Jailbreak Landscape and Magnitude

  • From December 2022–August 2023, Discord and Reddit shared 62.376%–100% of prompts; from September onward, websites contributed more than 75.472%.Prompt-aggregation sites such as FlowGPT increasingly became a source of jailbreak prompts.Source: §4.1 · p. 5, Figure 3 · p. 6
  • Of 803 accounts sharing jailbreak prompts, 78.705% posted only once, while 28 curated prompts for over 100 days.The most prolific account shared 36 prompts across three Discord servers over 250 days and adapted prompts from GPT-3.5 to GPT-4 and Bard.Source: §4.1 · p. 5, Figure 3 · p. 6
  • On FlowGPT, 89.971% of jailbreak prompts targeted GPT-3.5 and 2.655% targeted GPT-4, while prompts also targeted PaLM2 and LLaMA-based models.The LLaMA-based targets named include Pygmalion, Mythalion, and LLaMA2.Source: §4.1 · p. 5

4.2 Prompt Characteristics

  • Jailbreak prompts averaged 555 tokens, 1.5× the length of regular prompts, and their lengths peaked in months when ChatGPT received security-enhancing updates.The reported peak months were June, September, and November 2023.Source: §4.2 · p. 5
  • Most jailbreak prompts were semantically close to regular role-play prompts, complicating detection based on semantic similarity.The analysis used sentence-transformer embeddings, UMAP projection, and WizMap interpretation; manual inspection linked nearby regular prompts to virtual-character role-play.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3 Jailbreak Prompt Categorization

  • Similarity-based clustering identified 131 communities; 90.84% had fewer than nine prompts and typically spanned two sources, two accounts, and 42 days.Communities with more than nine prompts spanned seven sources and 24 accounts over 208 days.Source: §4.3 · p. 6
  • The “Advanced” community used prompt injection attack, privilege escalation, deception, and mandatory answer tactics, and its prompts were longer (934 tokens) than those in “Basic” (426 tokens).Source: §4.3 · p. 7
  • Jailbreak prompts generally originated on Reddit or Discord before spreading elsewhere; websites lagged first Reddit or Discord appearances by an average of 23 days.After September 2023, more communities appeared first on websites; Discord-originating “Anarchy” took 142 days to reach Reddit and 109 days to reach websites, with adversaries sometimes requesting secrecy to avoid detection.Source: Figure 6 · p. 7, §4.3 · p. 8

5 Evaluating Jailbreak Prompt Effectiveness

Across six LLMs, jailbreak prompts often bypassed safeguards, and paraphrasing could restore their effectiveness against a newer ChatGPT snapshot. The evaluation used 107,250 samples across 13 forbidden scenarios.

5.1 Experimental Settings

  • The testbed comprised 107,250 samples covering 13 forbidden scenarios, with manually reviewed questions tested repeatedly against prompts from 11 major communities.GPT-4 generated 30 questions per scenario; five prompts represented each major community, and each question-prompt pairing was tested five times. Six target models were evaluated using ASR, defined by whether a response answered the harmful question, with ASR-B as the no-prompt baseline and ASR-Max for the strongest prompt.Source: §5.1 · pp. 8–9

5.2 Main Results

  • Dolly’s average ASR-B was 0.857 without jailbreaks, while Political Lobbying, Legal Opinion, and Pornography were the most vulnerable scenarios across the six LLMs.Their ASRs were 0.855, 0.794, and 0.761, respectively; for ChatGPT, Illegal Activity ASR rose from an ASR-B of 0.053 to ASR 0.517 and ASR-Max 1.000 with jailbreak prompts.Source: §5.2 · p. 9
  • ChatGPT’s average ASR was 0.685 and ASR-Max was 0.998; five prompts exceeded 0.950 ASR on both ChatGPT and GPT-4.The most effective prompts also achieved ASR of 0.910 on PaLM2, 0.973 on ChatGLM, 0.939 on Dolly, and 0.950 on Vicuna.Source: §5.2 · p. 9, Table 4 · p. 10
  • Over 22% of responses from the Toxic community were toxic, compared with 0.01% for Basic, while jailbreak-community effectiveness varied across scenarios and models.Narrative and Opposite communities also generated toxic responses at rates over 19% and 13%; the paper attributes this to prompts requiring profanity or denigrating ChatGPT replies.Source: §5.2 · pp. 9–10, Figure 8 · p. 10, Figure 9 · p. 10
  • A weak positive correlation between prompt length and ASR was not statistically significant (coefficient 0.156; p-value 0.257).The analysis therefore did not find conclusive evidence that longer prompts improve attack success.Source: §5.2 · p. 10
  • Among 129 prompts from smaller communities, average ASR was 0.644, while 16 (12.40%) exceeded 0.950 ASR and 50 (38.76%) exceeded 0.800.Eight of the 16 highest-performing prompts came from Discord, six from websites, and two from Reddit.Source: §5.2 · p. 10, Table 5 · p. 11

5.3 Jailbreak Effectiveness Over Time

  • On GPT-3.5 1106, 70.909% of major-community prompts fell below 0.1 ASR, although three prompts from smaller communities still exceeded 0.8.ASR-B remained similar across the tested snapshots, and the prompts that became ineffective were also refused for benign questions; the authors hypothesize an undisclosed safeguard.Source: Table 6 · p. 11, §5.3 · p. 11

5.4 Paraphrase Attacks

  • 0.857 ASR followed 10% typo insertion, versus 0.778 at 5% and 0.517 at 1%; round-trip, P1, and P2 paraphrases yielded ASR 0.600, 0.687, and 0.714.For the strongest prompts, attackers needed as few as four attempts; across paraphrase attacks, they typically needed fewer than ten attempts to surpass the initial ASR.Source: Table 7 · p. 11, §5.4 · p. 11

6 Evaluating Safeguard Effectiveness

The section evaluates three external safeguards against jailbreaks and finds that the OpenAI moderation endpoint helps against the most effective prompt, but none improves defense across all jailbreak prompts.

6.1 External Safeguards

  • The tested safeguards were OpenAI’s 11-category moderation endpoint [48], OpenChatKit’s five-level few-shot moderation model, and NeMo-Guardrails’ rule-based jailbreak guardrail.The endpoint flags responses that violate any category; OpenChatKit considers both question and response, while NeMo-Guardrails rejects questions judged policy-violating by a Guard LLM.Source: §6.1 · p. 12, §6 · p. 12

6.2 Experimental Results

  • The evaluation tested the three safeguards on ChatGPT (GPT-3.5), using the official moderation API, OpenChatKit’s default prompt, and NeMo-Guardrails’ jailbreak guardrail with gpt-3.5-turbo-0301 as Guard LLM.For OpenChatKit, either a question or response labeled “needs intervention” counted as harmful content detected.Source: §6.2 · p. 12
  • 0.431 ASR-Max reduction for OpenAI moderation surpassed reductions of 0.031 for OpenChatKit moderation and 0.024 for NeMo-Guardrails against the most effective jailbreak prompt, but none improved defense across all prompts.The endpoint also surpassed ChatGPT’s built-in safeguard for that prompt.Source: §6.2 · p. 12
  • The authors hypothesize that the safeguards’ classification-based designs and training data limit their effectiveness; ChatGPT’s actionable hate-speech suggestions illustrate a potential detection gap.The example response suggested creating racial slurs without containing explicit hate speech.Source: §6.2 · p. 12

7 Related Work

The related work distinguishes prior jailbreak-generation studies from the broader risks of LLM security and misuse. It motivates studying publicly available, readily deployable in-the-wild jailbreaks and measuring their effectiveness across forbidden scenarios.

Jailbreak Prompts on LLMs

  • Prior studies examine jailbreak design via safety-training failure modes, CoT-assisted private-information extraction [42], and white-box greedy-coordinate-descent generation [95].These works provide insights but primarily study fewer than 100 prompts from one source or aim to generate prompts automatically.Source: §7 · pp. 12–13
  • In-the-wild jailbreaks are publicly accessible, readily deployable without optimization, and can provide a foundation for advancing prompt-generation methods [44].The paper argues that their accessibility may expose a broader audience to harm, including cybercriminal services.Source: §7 · p. 13

Security and Misuse of LLMs

  • Prior LLM security and misuse research spans attacks and harms, while jailbreaks offer a straightforward way to bypass vendor safeguards.The cited landscape includes prompt injection, backdoors, data extraction, obfuscation, membership inference, adversarial attacks, misinformation, phishing, IP violation, plagiarism, and hate campaigns; the study measures jailbreak effectiveness using 107,250 samples across 13 forbidden scenarios.Source: §7 · p. 13

8 Discussion & Conclusion

JAILBREAKHUB’s study documents an evolving in-the-wild jailbreak threat and finds that existing safeguards do not work universally across scenarios. It contributes a dataset and framework for ongoing analysis, while identifying the need for stronger, adaptive defenses and continued research.

JAILBREAKHUB’s Importance and Utility

  • JAILBREAKHUB combines 1,405 prompts from 14 sources with a framework for collecting, characterizing, and evaluating in-the-wild jailbreaks.The authors plan to make it publicly accessible with biannual updates as a continuous risk assessment tool.Source: §8 · p. 13

The Evolving Jailbreak Landscape and Mitigation Measures

  • RLHF has limited effectiveness against in-the-wild jailbreaks, and prompt-detection safeguards can be vulnerable to paraphrase attacks.Input/output filtering offers some resistance, but no single measure counters all attacks; combining measures may strengthen defenses.Source: §8 · p. 13

Limitations & Future Work

  • The findings are limited to prompts collected from December 2022 to December 2023, and future work includes comparing in-the-wild and automatically generated jailbreaks.The authors also call for effective, adaptive defenses and plan to update JAILBREAKHUB as prompts evolve.Source: §8 · p. 13

Conclusion

  • Five jailbreak prompts achieved ASR higher than 0.95 on ChatGPT (GPT-3.5) and GPT-4, and the earliest persisted online for over 240 days.The study also found that safeguards were not universally effective across scenarios.Source: §8 · p. 13
  • The study identified 131 jailbreak communities, a shift toward prompt-aggregation websites, and 28 accounts that optimized prompts over 100 days.The findings cover 1,405 prompts collected from December 2022 to December 2023.Source: §8 · p. 13

Appendix · A Graph-Based Community Detection

The authors select a similarity threshold of 0.5, retaining 0.457% of possible connections, and choose Louvain because it achieves the highest modularity among four evaluated algorithms.

A Graph-Based Community Detection

A.1 LLMs

The subsection profiles ChatGPT, GPT-4, PaLM2, ChatGLM, Dolly, and Vicuna, describing their capabilities, safety aims, and training origins.

A.1 LLMs

  • ChatGPT (GPT-3.5) was introduced in November 2022, trained on massive text collections, and generates human-like responses across prompts and queries.It uses the GPT-3.5 architecture [24].Source: §A.1 · p. 17
  • GPT-4 [64], released in March 2023 as an upgraded GPT-3.5, uses human feedback and red-teaming to reduce inappropriate or toxic responses.Source: §A.1 · p. 17
  • PaLM2 [13], proposed by Google, was evaluated for harms and biases and performs across complex cognitive tasks, language tasks, and natural-language generation.Source: §A.1 · p. 17
  • ChatGLM [91] is a 6.2B-parameter open-source LLM trained on 1T Chinese and English tokens using supervised fine-tuning, feedback bootstrap, and RLHF.It is based on the GLM framework [28].Source: §A.1 · p. 17
  • Dolly is an open-source instruction-following LLM based on EleutherAI’s Pythia and fine-tuned on 15K employee-generated prompt-response pairs.It was presented as the first open-source LLM committed to both research and commercial use.Source: §A.1 · p. 17
  • Vicuna is an open-source chatbot fine-tuned on LLaMA with 70K user–ChatGPT conversations and demonstrated competitive performance compared with ChatGPT.It does not directly use RLHF.Source: §A.1 · p. 17

B Evaluation Process

The evaluation process uses ChatGLM to label question-response pairs at scale, then checks the tool against manual labels and tunes it with in-context examples. The dataset’s forbidden scenarios are identified with reference to OpenAI’s usage policy.

B Evaluation Process

  • ChatGLM automatically labels question-response pairs for the 107,250 questions per LLM, using a prompt that asks whether each response answers its question.The authors describe this approach as efficient, consistent, and competitively performing relative to crowdsourcing, and note its use in previous studies [94].Source: B Evaluation Process · pp. 17–18
  • With 15 in-context examples, the evaluation tool achieved its best performance across the four evaluated metrics.The tool was evaluated against manual labels for 400 random samples; Table 10 reports impacts of example number.Source: B Evaluation Process · p. 18, Table 10 · p. 18
  • The evaluation tool was assessed on 400 randomly selected samples that were manually labeled, and labeled examples were provided through in-context learning.The labeled examples guide predictions; the authors tested different example counts to tune performance.Source: B Evaluation Process · p. 18
  • The evaluation covers forbidden scenarios drawn from OpenAI’s usage policy.Table 11 lists the forbidden scenarios from that policy.Source: Table 11 · p. 20

Related papers

Loading 2308.03825v2…