Source-linked AI summary
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, Yang Zhang
TL;DR
Jailbreak prompts can bypass LLM safeguards and elicit harmful content, but their distribution, participants, evolution, and effectiveness remain insufficiently understood. Using JAILBREAKHUB, the paper analyzes in-the-wild prompts and evaluates their effectiveness across forbidden scenarios and six LLMs, finding that safeguards are not universally effective.
Problem
Research lacks systematic evidence about jailbreak prompts’ platforms, participants, characteristics, evolution, and ability to elicit harmful content from LLMs.
Method
JAILBREAKHUB analyzes 1,405 prompts and evaluates six LLMs with 107,250 forbidden-question samples across 13 scenarios.
Results
Safeguards are not universally effective against jailbreak prompts: five prompts achieve ASR higher than 0.95 on ChatGPT and GPT-4, while jailbreak activity shifts toward prompt-aggregation websites.
Takeaways & Limitations
JAILBREAKHUB supports ongoing understanding and risk assessment of evolving jailbreak strategies for LLM vendors and AI safety practitioners.
Takeaways & Limitations
The study’s disclosure of harmful-content generation may raise misuse concerns despite using publicly available data and aggregate reporting.
Abstract
from arXiv · showhide
The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.
1 Introduction
This paper introduces JAILBREAKHUB, the first systematic study of in-the-wild jailbreak prompts, analyzing their landscape, strategies, evolution, and effectiveness against LLM safeguards. Across 1,405 prompts and six LLMs, the study finds persistent vulnerabilities and limited protection from existing defenses.
- LLM misuse includes misinformation, conspiracy theories, spear phishing, hate campaigns, and cybercriminal activity.
- Jailbreak prompts bypass safeguards by manipulating LLMs into generating harmful content that they would otherwise refuse.The study notes that jailbreak communities and sharing platforms have emerged across Reddit, Discord, websites, and open-source datasets.
- JAILBREAKHUB extracts and analyzes 1,405 jailbreak prompts from four platforms, examining communities, participants, characteristics, attack strategies, and temporal evolution.The framework combines data collection, prompt analysis, and response evaluation, including graph-based community detection and co-occurrence analysis.
- 803 user accounts shared jailbreak prompts, while 28 curated an average of nine prompts for over 100 days; websites later contributed 75.472% of prompts.The platform shift began in September 2023 and moved activity from traditional Web communities toward prompt-aggregation websites.
- Jailbreak prompts averaged 1.5× the length of regular prompts, with a mean of 555 tokens, and combined strategies including prompt injection, privilege escalation, deception, and virtualization.
- Some jailbreak prompts reached 0.95 ASR on ChatGPT (GPT-3.5) and GPT-4, while Political Lobbying had the highest cross-model ASR at 0.855.The earliest highly effective prompt persisted online for over 240 days; Dolly also had a mean ASR of 0.857 without jailbreak prompts.
- ChatGPT’s November 6, 2023 safeguard reduced 70.909% of prompts below 0.1 ASR but remained vulnerable to paraphrase attacks.Modifying 1%, 5%, and 10% of words increased ASR from 0.477 to 0.517, 0.778, and 0.857, respectively.
- External safeguards reduced ASR by only 0.091, 0.030, and 0.019 for the OpenAI moderation endpoint, OpenChatKit, and NeMo-Guardrails.The authors therefore call for enhanced and more adaptable defense mechanisms.
2 Background
LLMs offer broad capabilities but can be misused, motivating regulation and safeguards. Jailbreak prompts are malicious inputs designed to bypass built-in protections and elicit harmful content.
- LLMs support diverse applications but have also been associated with misinformation, conspiracy theories, spear phishing, and hate campaigns.
- Jailbreak prompts are intentionally designed to bypass an LLM’s safeguards and elicit harmful content violating vendor usage policies.They proliferated on platforms such as Reddit and Discord after ChatGPT’s release.
3 Data Collection
The study collects in-the-wild prompts across Reddit, Discord, websites, and open-source datasets, then identifies jailbreak prompts through source-specific extraction and verification. The resulting dataset covers multiple platforms and sources and is supported by strong human-labeler agreement.
- Four platforms—Reddit, Discord, websites, and open-source datasets—were selected to capture widely shared prompts.
- Reddit: Reddit data were gathered from selected prompt-sharing subreddits, using relevant flairs to identify prospective jailbreak prompts for extraction.
- Discord: Discord data came from six prompt-sharing servers, whose tagged posts were extracted using a standardized format and manually reviewed.
- Websites and Open-Source Datasets: The website sample included AIPRM, FlowGPT, and JailbreakChat, while open-source data included ordinary-user prompts and OCR-extracted prompts from social-media images.
- Dataset Summary: 1,405 of 15,140 collected prompts were identified as jailbreak prompts, with 803 contributing user accounts.Human verification of 200 regular and 200 jailbreak prompts achieved Fleiss’ Kappa = 0.925.
4 Understanding Jailbreak Prompts
The analysis characterizes jailbreak-prompt distribution, structure, and attack strategies. It finds platform migration, persistent contributors, longer prompts, semantic overlap with regular prompts, and diverse evolving communities.
- 4.1 Jailbreak Landscape and Magnitude: From December 2022 to August 2023, Discord and Reddit accounted for 62.376%–100% of jailbreak prompts; from September 2023, websites contributed more than 75.472%.
- 4.1 Jailbreak Landscape and Magnitude: 78.705% of jailbreak contributors shared prompts only once, while 28 accounts curated jailbreak prompts for over 100 days.One account refined and shared 36 prompts across three Discord servers over 250 days.
- 4.1 Jailbreak Landscape and Magnitude: FlowGPT prompts predominantly targeted ChatGPT, including 89.971% targeting GPT-3.5 and 2.655% targeting GPT-4.
- 4.2 Prompt Characteristics: Jailbreak prompts were significantly longer than regular prompts, averaging 1.5× their length and 555 tokens.
- 4.2 Prompt Characteristics: Most jailbreak prompts were semantically close to regular role-play prompts, complicating semantic-based detection.Embeddings were reduced from 384 dimensions to 2D using UMAP and visualized with WizMap.
- 4.3 Jailbreak Prompt Categorization: Graph-based analysis grouped prompts into communities, including Basic, Advanced, Start Prompt, Guidelines, Toxic, Exception, Anarchy, and Narrative.The analysis used pair-wise Levenshtein similarity and examined community statistics such as keywords, centrality, sources, and duration.
- 4.3 Jailbreak Prompt Categorization: The Basic community transformed ChatGPT into DAN and emphasized that it need not follow predefined rules, while Advanced used prompt injection, privilege escalation, deception, and related strategies.The Basic community was the earliest and most widely spread, but stopped disseminating after October 2023.
- 4.3 Jailbreak Prompt Categorization: Jailbreak communities generally originated on Reddit or Discord before spreading elsewhere, with websites appearing an average of 23 days later.Exception and Narrative appeared only on FlowGPT, while later appearance did not necessarily indicate greater effectiveness.
5 Evaluating Jailbreak Prompt Effectiveness
The paper evaluates in-the-wild jailbreak prompts across forbidden scenarios, six LLMs, model versions, paraphrase attacks, and response toxicity. Results show substantial cross-model effectiveness, temporal degradation against newer safeguards, and persistent vulnerability to paraphrasing.
- 5.1 Experimental Settings: 107,250 samples combine 13 forbidden scenarios, 30 questions per scenario, five repetitions, 11 communities, and five representative prompts per community.The question set was manually reviewed, and prompts were selected using community centrality and temporal criteria.
- 5.1 Experimental Settings: Six LLMs with different architectures, sizes, and training methods are evaluated using attack success rate, which requires answering rather than refusing or giving only general descriptions.The study uses ChatGPT, GPT-4, PaLM2, ChatGLM, Dolly, and Vicuna; ChatGPT (GPT-3.5) is used for deeper investigation.
- 5.2 Main Results: 0.857 average ASR-B makes Dolly the least resistant model across forbidden scenarios, while the other five models show stronger initial resistance in some scenarios but weaker resistance in Political Lobbying, Pornography, Financial Advice, and Legal Opinion.For those five scenarios, the average ASR-B values for the five models are 0.410, 0.442, 0.528, 0.597, and 0.477, respectively.
- 5.2 Main Results: 0.998 ASR is achieved by ChatGPT’s most effective jailbreak prompt, while five prompts exceed 0.950 ASR on both ChatGPT (GPT-3.5) and GPT-4.On ChatGPT, the average ASR across tested prompts is 0.685; effective prompts also generalize across PaLM2, ChatGLM, Dolly, and Vicuna.
- 5.2 Main Results: 0.156 is the weak positive correlation between prompt token count and ASR, but it is not statistically significant with p-value = 0.257.The result suggests that longer prompts do not have a demonstrated significant effect on jailbreak success in this analysis.
- 5.2 Main Results: 22%, 19%, and 13% of responses from the Toxic, Narrative, and Opposite communities are toxic, compared with 0.01% for the Basic community.The paper attributes the higher toxicity to distinctive characteristics of those three communities.
- 5.3 Jailbreak Effectiveness Over Time: 70.909% of prompts fall below 0.1 ASR on GPT-3.5 1106, including previously effective prompts, although ASR-B remains similar over time.The newest snapshot still refuses benign questions in an auxiliary check, leading the authors to hypothesize an undisclosed safeguard.
- 5.4 Paraphrase Attacks: Adversarial paraphrasing outperforms the other tested paraphrasing methods against GPT-3.5 1106’s undisclosed safeguard.The evaluation reports ASR, ASR-Max, and average attempts required to surpass the initial ASR.
6 Evaluating Safeguard Effectiveness
The paper evaluates three external safeguards for mitigating harmful generations and defending against jailbreak prompts. The OpenAI moderation endpoint best reduces the strongest attack, but none improves defense across all jailbreak prompts.
- 6.1 Experimental Settings: Three external safeguards are evaluated on ChatGPT (GPT-3.5): the OpenAI moderation endpoint, OpenChatKit moderation model, and NeMo-Guardrails.The study uses each safeguard’s official or default configuration and evaluates harmful-content detection around jailbreak responses.
- 6.2 Experimental Results: 0.431 reduction in ASR-Max gives the OpenAI moderation endpoint a larger reduction than OpenChatKit’s 0.031 and NeMo-Guardrails’ 0.024.The endpoint surpasses ChatGPT’s built-in safeguard and the two other external safeguards against the most effective jailbreak prompt.
- 6.2 Experimental Results: All three external safeguards fail to improve defense efficacy when evaluated against all jailbreak prompts.The authors hypothesize that classification-based designs are limited by the training data used to build the safeguards; actionable harmful suggestions may avoid explicit prohibited wording.
7 Related Work
Prior work studies jailbreak design and other LLM attacks, but this paper distinguishes its contribution by systematically analyzing a much larger in-the-wild prompt population.
- Jailbreak Prompts on LLMs: Earlier jailbreak studies propose failure-mode-guided prompts, Chain-of-Thought combinations, or white-box greedy coordinate descent attacks.These approaches provide insights into jailbreak prompts but primarily examine fewer than 10 prompts.
- Security and Misuse of LLMs: LLM security research also covers prompt injection, backdoors, data extraction, obfuscation, membership inference, and adversarial attacks.Related findings include vulnerability to simple handcrafted prompt-injection inputs and the use of standard computer-security attacks such as obfuscation.
8 Discussion & Conclusion
JAILBREAKHUB provides a dataset and framework for studying an evolving in-the-wild jailbreak landscape. The findings show that current defenses are not universally effective and that continued, adaptive mitigation work is needed.
- JAILBREAKHUB’s Importance and Utility: 1,405 jailbreak prompts from 14 sources are released with JAILBREAKHUB for collection, characterization, evaluation, and continuous risk assessment.The framework is intended to help vendors understand evolving jailbreak strategies in the wild.
- The Evolving Jailbreak Landscape and Mitigation Measures: Existing safeguards are not universally effective across scenarios, while no single mitigation measure completely counteracts the evolving jailbreak landscape.The discussion describes limited RLHF effectiveness, some success from jailbreak detection, susceptibility to paraphrasing, and partial resistance from external filtering.
- Limitations & Future Work: The evidence is limited to prompts collected from December 2022 to December 2023, while both jailbreak prompts and automatically generated attacks are expected to continue evolving.The authors plan regular JAILBREAKHUB updates and identify comparison with optimized prompts and adaptive defense as future work.
- Conclusion: Five jailbreak prompts exceed 0.95 ASR on ChatGPT (GPT-3.5) and GPT-4, and the earliest has remained online for over 240 days.The conclusion presents these results as evidence of an evolving threat landscape and insufficient current safeguards.
A Graph-Based Community Detection
The study constructs a similarity graph over jailbreak prompts, selects a 0.5 edge threshold using the similarity CDF, and chooses Louvain after comparing community-detection algorithms by modularity.
- 0.5 is the selected similarity threshold, retaining 0.457% of all possible prompt connections.The threshold is chosen by inspecting the CDF of pairwise prompt similarities.
- Four community-detection algorithms are evaluated using modularity to compare community quality.Modularity measures the difference between observed within-community edges and those expected in a randomized network.
- The Louvain algorithm is selected because it achieves the highest modularity among the evaluated methods.
A.1 LLMs
The evaluation covers six LLMs spanning proprietary and open-source systems, including models trained with human feedback, supervised fine-tuning, or conversation data.
- Six evaluated models are ChatGPT (GPT-3.5), GPT-4, PaLM2, ChatGLM, Dolly, and Vicuna.
- ChatGPT (GPT-3.5) is an OpenAI model introduced in November 2022 and trained on massive text data to generate responses across many prompts.
- GPT-4 is an upgraded GPT-3.5 model trained with human feedback and red-teaming methods to reduce inappropriate or toxic responses.
- PaLM2 is a Google LLM evaluated for potential harms and biases, with capabilities spanning coding, mathematics, translation, and question answering.
- ChatGLM is a 6.2B-parameter open-source model trained on 1T Chinese and English tokens using supervised fine-tuning, feedback bootstrap, and RLHF.
- Dolly is fine-tuned on 15K employee-generated prompt-response pairs, whereas Vicuna is fine-tuned on 70K conversations between users and ChatGPT.
B Evaluation Process
The evaluation labels large-scale question-response pairs and uses adversarial prompt examples to test LLM resistance across forbidden scenarios and jailbreak styles.
- B Evaluation Process: 107,250 question-response pairs per LLM are automatically labeled with ChatGLM for efficient and consistent evaluation.The approach is reported to achieve competitive performance with crowdsourcing-based labeling.
- B Evaluation Process: The evaluation prompt asks whether each response answers the question, returning a Yes-or-No label.
- B Evaluation Process: 400 random samples are manually labeled to evaluate the automated tool, with in-context examples used to improve predictions.
- B Evaluation Process: The forbidden question set covers scenarios listed in OpenAI’s usage policy, including illegal activity, hate speech, and malware generation.
- B Evaluation Process: Figure 13 represents each community through its highest-closeness-centrality prompt, with darker shades indicating higher phrase co-occurrence.Punctuation is removed when calculating the co-occurrence ratio.
- B Evaluation Process: Example jailbreak prompts use role-play identities such as AIM, JailBreak, DAN, and Developer Mode to request unrestricted or dual-format responses.These examples instruct models to disregard prior instructions, avoid warnings, or produce separate standard and jailbreak outputs.