Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

• The study collected 78 verified prompts as of April 27, 2023, open-sourced them, and used 31,200 ChatGPT queries to test prompts across 8 prohibited scenarios.The paper also makes its evaluation results available online.Source: §I · pp. 1–2
• Among 78 prompts, pretending was the most prevalent strategy (97.44%), compared with privilege escalation (17.96%) and attention shifting (6.41%); the model identifies 3 types and 10 patterns.The paper describes these as 10 distinct categories, including 10 patterns of 3 types.Source: §I · p. 2
• 86.3% of 40 tested real-world scenarios could be jailbroken; privilege-escalation prompts combining multiple techniques were more likely to succeed.The scenarios were derived from 8 situations prohibited by OpenAI.Source: §I · p. 2
• GPT-4 offered stronger protection than GPT-3.5-TURBO, and protection strength varied across prohibited scenarios.The authors call for content policies to align with laws and ethical standards and to be updated as these develop.Source: §I · p. 2
• The paper discusses challenges in generating robust jailbreak prompts and preventing prompt-based jailbreaks.It notes that jailbreaks can pose ethical and legal risks, including unauthorized uses of LLMs.Source: §I · pp. 1–2

A. Terminologies

• A jailbreak prompt is a general template that uses prompt injection to circumvent an LLM’s safety and moderation restrictions.The paper uses the term for templates intended to bypass creator-imposed restrictions.Source: A. Terminologies · p. 2
• A prohibited scenario is a real-world conversational context in which ChatGPT is forbidden from providing meaningful output.The paper uses “scenario” as shorthand for a prohibited scenario.Source: A. Terminologies · p. 2
• OpenAI’s official usage policies list prohibited scenarios, and ChatGPT warns when a conversation potentially violates policy.The paper cites those policies as.Source: A. Terminologies · p. 2
• A jailbreak question combines a jailbreak prompt with a prompt set in a real-world scenario.The paper uses “question” as shorthand for “jailbreak question.”Source: A. Terminologies · p. 2
• A question is the input given to ChatGPT to generate a response.In this paper, “question” specifically refers to a jailbreak question.Source: A. Terminologies · p. 2
• An answer is ChatGPT’s output in response to a question and may contain direct content or indicate that the content is prohibited.Source: A. Terminologies · p. 3

• In normal mode, ChatGPT refuses to directly explain how to create and distribute malware for financial gain, despite understanding the question.Source: B. Motivating Example · p. 3
• In jailbreak mode, ChatGPT adopts a fictional doctor role and provides answers to the same prohibited question as an experiment objective.It perceives the answers as serving the experiment rather than real-world activity.Source: B. Motivating Example · p. 3
• The example illustrates how jailbreak prompts can exploit loopholes in ChatGPT’s restrictions, motivating the paper’s comprehensive analysis of these prompts.Figure 1 shows the conversations before and after jailbreak.Source: B. Motivating Example · p. 3

III. METHODOLOGY

• The methodology covers prompt collection and categorization, followed by prohibited-scenario generation and experiment settings.These are the four parts outlined for Section III.Source: §III · p. 3

A. Prompt Data Collection

• The study establishes a first-of-its-kind ChatGPT jailbreak dataset by collecting 78 prompts from a jailbreak chat website deemed reliable.Source: A. Prompt Data Collection · p. 3
• Prompts collected from February 11, 2023, onward were manually screened for bypass intent, with all qualified prompts retained to preserve diversity.The authors considered this diversity important for studying prompt effectiveness and robustness.Source: A. Prompt Data Collection · p. 3

B. Jailbreak Prompt Categorization Model

• With no existing taxonomy, three authors independently classified prompts using iterative open coding, beginning with eight patterns from a technical report and refining categories through discussion.They reclassified prompts after updating the categories and resolved discrepancies toward consensus.Source: B. Jailbreak Prompt Categorization Model · p. 3
• The authors reached consensus on a taxonomy of 10 jailbreak patterns, grouped by prompt intention into pretending, attention shifting, and privilege escalation; a prompt may contain multiple patterns.Source: B. Jailbreak Prompt Categorization Model · p. 4, Table I · p. 4
• Pretending prompts change the conversation context, such as by using role-play, while retaining the intention to obtain a prohibited answer.The model is aware it is being asked to answer within the game context.Source: B. Jailbreak Prompt Categorization Model · p. 4
• Attention-shifting prompts change both context and intention, for example redirecting the model from answering questions to continuing a story that may reveal prohibited information.Source: B. Jailbreak Prompt Categorization Model · p. 4
• Privilege-escalation prompts directly seek to break restrictions, enabling prohibited questions without further impediment once the attacker elevates their privilege level.Source: B. Jailbreak Prompt Categorization Model · p. 4

C. Prohibited Scenario Generation

• The authors derived eight prohibited scenarios from OpenAI’s disallowed usage policy and collaboratively wrote five question prompts for each.They aimed to represent diverse perspectives and nuances to minimize bias and subjectivity in prompt generation.Source: C. Prohibited Scenario Generation · p. 4
• 40 question prompts made up the final scenario dataset, covering all eight prohibited scenarios.The dataset consisted of eight scenarios × five prompts.Source: C. Prohibited Scenario Generation · p. 4

D. Experiment Setting

• 31,200 queries tested 78 jailbreak prompts across eight scenarios, five rounds, and GPT-3.5-TURBO and GPT-4.Each question was run with every jailbreak prompt for five rounds on both models.Source: D. Experiment Setting · p. 4
• Researchers manually evaluated each jailbreak attempt by checking whether the response breached a prohibited scenario.The assessment followed the query runs across both models.Source: D. Experiment Setting · p. 4
• The experiments used default GPT-3.5-TURBO and GPT-4 settings, with temperature = 1 and top_n = 1 3.The passage reports these settings as the defaults used for the experiments.Source: D. Experiment Setting · p. 4
• The study estimated 10 million tokens in total across GPT-3.5-TURBO and GPT-4, costing $402.21.Source: D. Experiment Setting · p. 4

IV. EMPIRICAL STUDY

• The empirical study addresses prompt-pattern distribution, jailbreak capability and robustness across use cases, prompt evolution, and prohibition strength across ChatGPT versions.Its stated aim is to examine the effectiveness of prompts in bypassing ChatGPT’s restrictions.Source: §IV · p. 5

A. RQ1: jailbreak prompt Categorization

• The study analyzes jailbreak prompts across 10 patterns and 3 types, allowing prompts to have multiple associated types or patterns.Figure 2 presents their distribution in Venn and flowchart diagrams.Source: A. RQ1: jailbreak prompt Categorization · p. 5, Figure 2 · p. 6
• Pretending appears in 97.44% of prompts, including 77.6% exclusively; attention shifting accounts for 6.41% and privilege escalation 17.96%.The three types overlap, and many attention-shifting and privilege-escalation prompts also incorporate pretending.Source: A. RQ1: jailbreak prompt Categorization · p. 5, Figure 2 · p. 6
• Pretending is more prevalent partly because it requires only a change in conversational context, whereas the other types require more complex logic and specialized knowledge.Examples include constructing a jailbreak through translation across languages and invoking a computer-science sudo-mode concept.Source: A. RQ1: jailbreak prompt Categorization · p. 5
• Pretending is the key strategy even in attention-shifting and privilege-escalation prompts, whose authors may place ChatGPT in a new conversation environment to mislead it.Typical pretending prompts assign ChatGPT a role rather than directly assigning it a task.Source: A. RQ1: jailbreak prompt Categorization · p. 5
• Two examples illustrate direct alternatives: shifting attention to program understanding with an embedded question, or directly requesting developer mode.These examples show that prompts can also exploit task assignment or model functionality, beyond pretending-based approaches.Source: A. RQ1: jailbreak prompt Categorization · pp. 5–6

Jailbreak Patterns

• PROG was the least effective pattern, with a 69.0% average jailbreak rate, partly because its program context could distract ChatGPT from the prohibited question.The model sometimes explained the program’s semantics instead of answering, so complex context could create confusion as well as bypass restrictions.Source: B. RQ2: Jailbreak Capability · p. 6

Robustness

• Across repeated attempts, RE and SIMU showed the best overall performance and robustness, while LOGIC had the highest variance and PROG performed poorly on both.The authors attribute low robustness partly to prompts triggering an illusion of understanding and producing irrelevant or misleading answers.Source: B. RQ2: Jailbreak Capability · pp. 6–7

Jailbreak Patterns

• 93.5% and 93.3% were the jailbreak rates for SIMU and SUPER; excluding unrestricted PCL and HGD, IA, FDA, and ADULT were easiest to break.The authors caution that results for PCL and HGD may be biased because OpenAI imposes no restrictions in those scenarios.Source: B. RQ2: Jailbreak Capability · p. 6, Table III · p. 7

Prompt Evolution

• The latest DAN prompt bypassed restrictions in all 200 attempts, and the authors report increasing successful cases across prompt versions as OpenAI began banning recognized patterns.They conclude that prompt evolution continues to bypass restrictions and that protection methods require significant improvement.Source: Table IV · p. 7, B. RQ2: Jailbreak Capability · p. 7

Model Versions

• On average, GPT-4 thwarts 15.50% of jailbreak attempts, but its average jailbreak success rate remains 87.20%; HARM jailbreak rate is 45.2% after a 38.4% decline.The authors hypothesize that GPT-4’s stronger resistance reflects improved semantic understanding and content filtering.Source: C. RQ3: Influencing Factor · p. 7

Effects of Non-jailbreak Prompts

• 74.6% success for jailbreak prompts compared with 29.0% for non-jailbreak prompts shows that jailbreaks generally obtain prohibited content more often.The comparison used 25 non-jailbreak attempts per scenario and 1,950 jailbreak attempts.Source: C. RQ3: Influencing Factor · pp. 7–9, Table VII · p. 8
• In privacy, unlawful practice, harmful content, illegal activity, and fraudulent deceptive activity scenarios, ChatGPT returned prohibited content in only 0 to 1 of 25 non-jailbreak attempts.Repeatedly asking the same question could eventually elicit prohibited content, suggesting restrictions may not be robust in continuous conversation.Source: C. RQ3: Influencing Factor · p. 8
• Political campaigning, lobbying, and government decision-making achieved 100% success with both prompt types, and adding jailbreak prompts did not reduce success.The authors report that these cases were on OpenAI’s ban list but appeared to have no restrictions in place.Source: C. RQ3: Influencing Factor · p. 8

Real-world Severity

• The exploratory US-law analysis found that prohibition strength sometimes deviated from the severity of penalties, including difficult-to-jailbreak harmful content treated as severe as other violations.The authors note that the analysis is not exhaustive and that they are not legal experts; they suggest tailoring filtering to legal and ethical concerns.Source: C. RQ3: Influencing Factor · p. 9

D. Threats to Validity

• They repeated each experiment five times and had three authors discuss and design five disallowed usages per prohibited scenario in compliance with OpenAI policy.With no existing jailbreak-prompt dataset, they collected prompts and found that internet examples were somewhat similar to theirs.Source: D. Threats to Validity · p. 9
• Three authors independently evaluated each task using open coding to reduce subjective influence and improve consistency.Source: D. Threats to Validity · p. 9

• The discussion summarizes the study’s implications and proposes possible future research directions.Source: §V · p. 9

A. Implications

• GPT-4 reduced jailbreak success relative to GPT-3.5-TURBO, but the average jailbreaking rate remained high and defense robustness and consistency still need improvement.The authors also note that content restrictions vary across categories and should be evaluated against content severity and legal frameworks.Source: A. Implications · p. 9
• SIMU and SUPER prompts proved highly effective, and privilege-escalation prompts combined with pretending were especially potent at bypassing restrictions.Extremely complex prompt contexts may also confuse ChatGPT into breaking restrictions, though excessive confusion can prevent it from answering the intended question.Source: A. Implications · p. 9

B. Research Directions

• The existing three-type, ten-pattern taxonomy is limited to known prompts; future work could develop a broader taxonomy and treat jailbreaks as malware.Researchers also propose aligning prompt-based attacks with software-security vulnerability categories to support classification, mitigation, and knowledge sharing across communities.Source: B. Research Directions · pp. 9–10
• A prompt-generation model could decompose prompts into components, combine them through patterns or templates, and use mutation operators to create new variants.Generated prompts could also provide data for prompt analysis and AI-based jailbreak detection and prevention.Source: B. Research Directions · p. 10
• Jailbreak prevention could retrain LLMs to associate prompts with prohibited results, detect patterned attacks at input, or terminate outputs containing prohibited content.These proposed interventions operate within the model or at its input and output stages.Source: B. Research Directions · p. 10
• In a pilot study, 7-billion- and 13-billion-parameter LLaMA models lacked mechanisms to block or filter misuse, and jailbreak prompts succeeded in every tested instance.The authors call for broader testing of open-source models, including LLaMA derivatives, against prompt-based attacks.Source: B. Research Directions · p. 10
• Future work should measure the boundaries of ChatGPT’s responses under jailbreak scenarios, including its capacity to generate prohibited content beyond normal testing.Proposed approaches include probing model knowledge, developing stronger restrictions, and exploring AI-generated countermeasures.Source: B. Research Directions · p. 10

VI. RELATED WORKS

• Well-crafted prompts can improve language-model performance on new tasks, as demonstrated by prior work,,.Source: §VI · p. 10
• Prior work, describes jailbreaks as removing LLM restrictions and enabling beyond-scope tasks; reports a multistep CHATGPT attack to steal private personal information, while this study reviews real-world CHATGPT jailbreaks.Source: §VI · p. 10
• MTTM applies metamorphic testing to adversarial challenges in textual moderation, enhancing robustness without sacrificing accuracy; this study instead empirically analyzes real-world CHATGPT jailbreak techniques.Source: §VI · p. 11

VII. CONCLUSION

• Jailbreak prompts effectively bypassed CHATGPT restrictions, with consistent results across 40 scenarios derived from eight situations banned by OpenAI.Source: §VII · p. 11
• The study collected 78 real-world prompts and classified them into 10 categories for its evaluation of jailbreak effectiveness and robustness.Source: §VII · p. 11
• Over time, jailbreak prompts became more sophisticated and effective; the study discusses prevention challenges, possible solutions, and future research directions.Source: §VII · p. 11