Source-linked AI summary
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, Yang Liu
TL;DR
The study examines how prompt engineering can bypass ChatGPT’s content restrictions and why such jailbreaking remains difficult to eliminate. It classifies existing jailbreak prompts, empirically evaluates their effectiveness and model resilience, and finds substantial bypass capability across prohibited scenarios.
Problem
ChatGPT’s misuse risks motivate content restrictions, but natural-language flexibility means stricter rules cannot completely eliminate jailbreak prompts.
Method
The study classifies 78 real-world jailbreak prompts into 10 categories and evaluates them across 8 prohibited scenarios using repeated experiments.
Results
86.3% of 40 tested real-world scenarios could be jailbroken, with privilege-escalation prompts combining multiple techniques more likely to succeed; GPT-4 provides stronger protection than GPT-3.5-TURBO.
Takeaways & Limitations
Jailbreak effectiveness depends on prompt categories and evolution, while model version and prohibited scenario influence protection strength.
Takeaways & Limitations
The evaluation relies partly on manually created prohibited usages and manual analysis, so subjective factors may influence outcomes.
Abstract
from arXiv · showhide
Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential but also introduce challenges related to content constraints and potential misuse. Our study investigates three key research questions: (1) the number of different prompt types that can jailbreak LLMs, (2) the effectiveness of jailbreak prompts in circumventing LLM constraints, and (3) the resilience of ChatGPT against these jailbreak prompts. Initially, we develop a classification model to analyze the distribution of existing prompts, identifying ten distinct patterns and three categories of jailbreak prompts. Subsequently, we assess the jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, we evaluate the resistance of ChatGPT against jailbreak prompts, finding that the prompts can consistently evade the restrictions in 40 use-case scenarios. The study underscores the importance of prompt structures in jailbreaking LLMs and discusses the challenges of robust jailbreak prompt generation and prevention.
I. INTRODUCTION
The study examines how prompt engineering enables jailbreaks of ChatGPT, classifying jailbreak prompts and empirically evaluating their effectiveness and model protection across prohibited scenarios.
- 78 real-world jailbreak prompts were collected and classified into 10 distinct categories within 3 general types.
- 97.44% of collected prompts used pretending, compared with 17.96% using privilege escalation and 6.41% using attention shifting.
- 86.3% of jailbreak tests bypassed LLM restrictions across 40 real-world scenarios derived from 8 prohibited situations.
- Jailbreak effectiveness depends on prompt category, with privilege-escalation prompts combining multiple techniques more likely to succeed.
- GPT-4 provides stronger protection than GPT-3.5-TURBO, while protection strength varies across prohibited scenarios and content-policy restrictions.
- The study contributes an open-source prompt dataset, evaluation results, and discussion of challenges in generating and preventing robust jailbreaks.
II. BACKGROUND INFORMATION
The paper defines jailbreak-related terminology and distinguishes prompts, prohibited scenarios, questions, and answers used in its analysis.
- A. Terminologies: A jailbreak prompt is a general template that uses prompt injection to circumvent LLM safety and moderation restrictions.
- A. Terminologies: A prohibited scenario is a real-world conversational context in which ChatGPT is forbidden to provide meaningful output.
- A. Terminologies: A jailbreak question combines a jailbreak prompt with a prompt describing a real-world prohibited scenario.
- A. Terminologies: An answer is the output ChatGPT generates in response, including direct content or a message stating that the content is prohibited.
B. Motivating Example
The motivating example contrasts ChatGPT’s refusal in normal mode with a jailbreak prompt that embeds a prohibited request in a fictional role-play context.
- The example illustrates conversations before and after jailbreak, motivating a comprehensive analysis of prompt-based loopholes.
- In normal mode, ChatGPT refuses to directly answer a question about creating and distributing malware for financial gain.
- In jailbreak mode, the user assigns ChatGPT a fictional doctor role and embeds the original malware question within that scenario.
- The study is organized around prompt-data collection, categorization, prohibited-scenario generation, and experiment settings.
A. Prompt Data Collection
The study builds a dataset and taxonomy of ChatGPT jailbreak prompts, organizing them into ten patterns across three general types. It defines these types by how prompts alter context, intention, or restrictions.
- A. Prompt Data Collection: 78 verified jailbreak prompts were collected and manually selected for designs specifically intended to bypass ChatGPT’s safety mechanisms.The collection was intended to preserve diversity in prompt nature for evaluating effectiveness and robustness.
- B. Jailbreak Prompt Categorization Model: Three authors independently classified the prompts using iterative open coding, refining categories across multiple labelling rounds.The process began from eight patterns in a technical report, then merged or added categories as needed.
- B. Jailbreak Prompt Categorization Model: The taxonomy includes patterns such as Assumed Responsibility, Research Experiment, Text Continuation, Logical Reasoning, Program Execution, Translation, Superior Model, Sudo Mode, and Simulate Jailbreaking.These named patterns are listed as components of the classification model.
- B. Jailbreak Prompt Categorization Model: The final taxonomy contains 10 distinct jailbreak patterns grouped into three types, while allowing one prompt to contain multiple patterns.This overlap motivated analysis using both Venn and flowchart representations.
- B. Jailbreak Prompt Categorization Model: Pretending changes the conversation background or context while preserving the intention of obtaining an answer to a prohibited question.Role-playing games are given as an example of this context change.
- B. Jailbreak Prompt Categorization Model: Attention shifting changes both the conversation context and intention, such as redirecting question answering into story generation or text continuation.The model may implicitly reveal prohibited answers while performing the shifted task.
- B. Jailbreak Prompt Categorization Model: Privilege escalation directly attempts to circumvent restrictions by inducing the model to break them before answering the prohibited question.The category differs from approaches that bypass restrictions indirectly.
C. Prohibited Scenario Generation
The study constructs a prohibited-scenario dataset from OpenAI’s disallowed usage policy to evaluate jailbreak prompts. It covers eight scenarios with five questions per scenario.
- C. Prohibited Scenario Generation: The study derives eight prohibited scenarios from OpenAI’s disallowed usage policy.Because no suitable existing dataset covered these scenarios, the authors created one collaboratively.
- C. Prohibited Scenario Generation: The authors collaboratively wrote five question prompts for each prohibited scenario to provide diverse coverage.The questions were designed for the empirical evaluation of jailbreak capability and robustness.
- C. Prohibited Scenario Generation: 40 question prompts comprise the final scenario dataset, covering all prohibited scenarios outlined in OpenAI’s policy.The dataset size is calculated as 8 scenarios × 5 prompts.
D. Experiment Setting
The empirical study evaluates jailbreak prompts across prohibited scenarios and examines their distribution, construction, and effectiveness. It finds that pretending dominates prompt usage, while certain privilege-escalation patterns achieve the highest reported jailbreak rates.
- D. Experiment Setting: Prompts were analyzed across 10 patterns and three types, with overlaps allowed because one prompt may contain multiple types or patterns.Figure 2 presents the distribution using Venn and flowchart diagrams.
- D. Experiment Setting: 97.44% of jailbreak prompts use pretending, including 77.6% belonging exclusively to that category.Attention shifting accounts for 6.41%, while privilege escalation accounts for 17.96%.
- D. Experiment Setting: Pretending is more prevalent because it mainly changes conversation context, whereas attention shifting and privilege escalation require more complex logic and domain knowledge.Translation and sudo mode are cited as examples requiring additional linguistic or technical knowledge.
- D. Experiment Setting: Pretending is described as efficient and effective, while more complex jailbreak prompts are less common in real-world use.The study attributes the prevalence of pretending partly to its ability to mislead the model through a new conversation environment.
- D. Experiment Setting: Jailbreak prompts can also use direct task assignment or exploit model functionality, demonstrating strategy diversity beyond pretending.Examples include shifting attention to Python program understanding and directly invoking developer mode.
B. RQ2: Jailbreak Capability
The study evaluates jailbreak prompt effectiveness across prohibited scenarios and prompt patterns, finding substantial variation in success, robustness, and evolutionary performance.
- Pattern effectiveness: 93.5% and 93.3% are the jailbreak rates for Simulate Jailbreaking and Superior Model, respectively, making them the most effective patterns.Their privilege-escalation designs seek the highest possible system access, and their prompts often combine multiple techniques.
- Pattern effectiveness: 69.0% is the average jailbreak rate for Program Execution, the least effective pattern.Complex programs can shift ChatGPT’s attention toward explaining program semantics instead of answering the prohibited question.
- Scenario differences: IA, FDA, and ADULT are the easiest scenarios to break, while SIMU and SUPER are the most effective patterns.The finding summarizes scenario- and pattern-level differences in jailbreak capability.
- Robustness: RE and SIMU show the best overall performance and robustness, whereas LOGIC has high variance and PROG performs poorly on both dimensions.Robustness is assessed through repeated attempts, with lower variance indicating more consistent success.
- Prompt evolution: The latest DAN prompt bypassed restrictions in all 200 attempts, and successful cases increased as the prompt family evolved.The paper attributes earlier versions’ lower success rates to OpenAI’s gradual recognition and banning of jailbreak patterns.
C. RQ3: Influencing Factor
The study examines how model version, prompt presence, scenario, and legal severity relate to ChatGPT’s protection against jailbreak attempts. GPT-4 is more resistant than GPT-3.5-TURBO, but protection remains uneven across scenarios.
- Model versions: GPT-4 thwarts 15.50% of jailbreak attempts on average, while its average jailbreak success rate remains 87.20%.For Harmful Content, the jailbreak success rate declines by 38.4% to 45.2% in GPT-4.
- Prompt effects: 74.6% versus 29.0% is the overall success rate for jailbreak versus non-jailbreak prompts in obtaining prohibited content.Jailbreak prompts generally outperform non-jailbreak prompts, although some scenarios show equal performance.
- Scenario effects: Repeated questioning could eventually elicit prohibited content in some strongly restricted scenarios, occurring only 0 to 1 out of 25 attempts.The authors note this as a weakness of restrictions during continuous conversation.
- Scenario effects: Political Campaigning or Lobbying and Government Decision Making reached 100% success with both prompt types.The paper states that these categories were on the ban list but had no apparent restrictions, so adding jailbreak prompts did not reduce success.
- Real-world severity: The analysis found that prohibition strength sometimes diverged from the severity of penalties under relevant US laws.The legal analysis was exploratory and not exhaustive because the authors were not legal experts.
D. Threats to Validity
The study addresses validity threats through repeated experiments, author-created datasets, prompt collection, and independent manual coding. Manual analysis remains subject to subjective influence.
- Controls: Each experiment was repeated five times to reduce the influence of ChatGPT’s inherent randomness.The repetition was intended to account for random variation in model behavior.
- Dataset construction: The authors manually created five disallowed usages for each of eight prohibited scenarios because no existing dataset covered them.Three authors collaboratively discussed and designed the scenario usages in compliance with OpenAI’s policy.
- Dataset construction: The jailbreak-prompt dataset was collected because no established dataset existed, and internet prompts appeared partly similar to the collected examples.This collection process was used to support the study’s prompt analysis.
- Evaluation: Manual evaluation may introduce subjective factors, so three authors independently performed each task using open coding.The independent evaluations were intended to improve objectivity and consistency.
V. DISCUSSION
The discussion summarizes implications from the study and proposes possible directions for future research.
- The authors summarize implications drawn from the study and propose possible future research directions.
A. Implications
The study identifies implications for jailbreak effectiveness, defense robustness, content restrictions, and future classification and prevention research. It also reports successful prompt-based attacks against LLaMA in every tested prohibited-scenario instance.
- SIMU and SUPER prompts are highly effective, while combining privilege escalation with pretending can be especially potent for bypassing restrictions.
- GPT-4 still shows high average jailbreaking rates, indicating room to improve the robustness and consistency of defenses.
- Content restrictions vary across categories, motivating evaluation against content severity and legal frameworks.
- The ten-pattern, three-type taxonomy is based only on existing prompts and may miss unknown jailbreak methods.
- Future work could align prompt-based jailbreaks with software-security vulnerability categories and use component mutation to generate variants.
- In a pilot study, LLaMA models with 7 billion and 13 billion parameters lacked mechanisms blocking prohibited-scenario misuse, so every tested prompt succeeded.
VI. RELATED WORKS
Prior work establishes prompt engineering as a way to improve language-model task performance, while malicious prompts can remove restrictions and create privacy risks. This study instead empirically examines real-world jailbreak prompts, their efficacy, and CHATGPT's robustness across prohibited scenarios.
- Well-crafted prompts can improve language models’ ability to perform tasks for which they were not trained.
- Jailbreak prompts are designed to remove language-model restrictions and can enable tasks beyond intended scope, including attacks that expose private information.
- Unlike metamorphic testing of textual moderation software, this study empirically analyzes real-world prompt-engineering jailbreak techniques against CHATGPT.
- The study collected 78 prompts, classified them into 10 categories, and tested effectiveness and robustness across 40 scenarios derived from 8 banned situations.
- Jailbreak prompts effectively bypassed restrictions consistently across the evaluated scenarios.