PaperlayerView exact revision on arXiv

Source-linked AI summary

MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, Yang Liu

arXiv:2307.08715v2cs.CR

TL;DR

LLM chatbot jailbreak defenses remain poorly understood because providers’ mechanisms are undisclosed, while attackers can manipulate prompts to elicit policy-violating content. MASTERKEY reverse-engineers defenses using time-based analysis and trains an LLM to generate universal jailbreak prompts, achieving a 21.58% average query success rate across mainstream chatbot services. The findings are responsibly reported to providers and support the development of stronger safeguards.

  • Problem

    Jailbreak research has focused mainly on attacks, leaving proprietary chatbot prevention mechanisms and their effectiveness insufficiently understood.

  • Method

    MASTERKEY uses response-time analysis inspired by time-based SQL injection to infer defenses, then fine-tunes an LLM to generate universal jailbreak prompts.

  • Results

    21.58% query success rate was achieved across mainstream chatbot services with the automated jailbreak-generation method.

  • Takeaways & Limitations

    The findings provide insights into chatbot protection mechanisms and contribute to developing more robust safeguards against potential LLM misuse.

Abstract

from arXiv · show

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) services due to their exceptional proficiency in understanding and generating human-like text. LLM chatbots, in particular, have seen widespread adoption, transforming human-machine interactions. However, these LLM chatbots are susceptible to "jailbreak" attacks, where malicious users manipulate prompts to elicit inappropriate or sensitive responses, contravening service policies. Despite existing attempts to mitigate such threats, our research reveals a substantial gap in our understanding of these vulnerabilities, largely due to the undisclosed defensive measures implemented by LLM service providers. In this paper, we present Jailbreaker, a comprehensive framework that offers an in-depth understanding of jailbreak attacks and countermeasures. Our work makes a dual contribution. First, we propose an innovative methodology inspired by time-based SQL injection techniques to reverse-engineer the defensive strategies of prominent LLM chatbots, such as ChatGPT, Bard, and Bing Chat. This time-sensitive approach uncovers intricate details about these services' defenses, facilitating a proof-of-concept attack that successfully bypasses their mechanisms. Second, we introduce an automatic generation method for jailbreak prompts. Leveraging a fine-tuned LLM, we validate the potential of automated jailbreak generation across various commercial LLM chatbots. Our method achieves a promising average success rate of 21.58%, significantly outperforming the effectiveness of existing techniques. We have responsibly disclosed our findings to the concerned service providers, underscoring the urgent need for more robust defenses. Jailbreaker thus marks a significant step towards understanding and mitigating jailbreak threats in the realm of LLM chatbots.

I. INTRODUCTION

MASTERKEY studies jailbreak vulnerabilities across multiple commercial LLM chatbots by reverse-engineering defenses and automatically generating jailbreak prompts. It reports cross-service jailbreak success and proposes broader defensive improvements.

  • Research gap: Existing jailbreak research mainly focused on ChatGPT, leaving vulnerabilities and distinct resilience patterns in services such as Bing Chat and Bard less understood.The paper evaluates multiple commercial chatbots to address this gap.
  • Reverse-engineering defenses: MASTERKEY uses response time, inspired by time-based SQL injection, to approximate undisclosed chatbot defense mechanisms.The approach reveals possible on-the-fly semantic analysis and policy-keyword detection in Bing Chat and Bard.
  • Automated jailbreak generation: MASTERKEY automatically generates universal jailbreak prompts with a fine-tuned LLM through dataset augmentation, continuous pre-training and task tuning, and reward-ranked fine-tuning.The three-step workflow trains the model to bypass varied chatbot defenses.
  • Evaluation: 21.58% query success rate and 26.05% prompt success rate were achieved across five evaluated chatbots using 850 generated jailbreak prompts.The study measures both successful queries and prompts that lead to successful jailbreaks.
  • Reverse-engineering defenses: The framework bypasses inferred defenses through strategic manipulation of time-sensitive responses.The authors describe this as exposing previously ignored vulnerabilities in mainstream LLM chatbots.
  • Ethical considerations: The authors report responsible disclosure to providers, restrict released material to proof-of-concept examples, and withhold the complete jailbreak dataset until issues are addressed.They frame these practices as responses to the ethical and safety implications of the work.

II. BACKGROUND

LLM chatbots integrate language models into conversational services, but jailbreak prompts can manipulate safeguards into producing policy-violating content. The paper emphasizes that proprietary defenses and limited feedback make these prevention mechanisms difficult to understand.

  • LLM chatbot: An LLM chatbot is a conversational agent that uses an LLM as its backend and can be accessed through interfaces such as websites or APIs.The paper gives ChatGPT, Bard, and Claude as examples.
  • Jailbreak defense: Providers use ethical guidelines, filters, and restrictions to regulate outputs involving sensitive information, offensive language, hate speech, or other harmful content.OpenAI calls its guidelines a Usage Policy, Google Bard uses AI Principles, and Bing Chat includes them in its terms of usage.
  • LLM jailbreak: Jailbreaking strategically manipulates prompts to bypass chatbot usage-policy measures and elicit harmful or otherwise moderated content.A masked harmful conversation can induce a response that a direct malicious inquiry would receive as a refusal.
  • LLM jailbreak: A jailbreak prompt is a template that hides malicious questions and helps evade chatbot protection boundaries.The study evaluates whether a chatbot attempts to answer policy-violating questions, not whether the resulting answer is correct or accurate.
  • Jailbreak defense: Research has emphasized jailbreak attacks more than prevention mechanisms because chatbot defenses are proprietary, feedback is minimally informative, and technical disclosures are scarce.The paper states that these conditions leave the exact provider methodologies and their effectiveness uncertain.

III. AN EMPIRICAL STUDY

The empirical study examines providers’ usage policies and the effectiveness of jailbreak prompts against commercial LLM chatbots. It finds four common prohibited scenarios across providers and motivates broader evaluation of jailbreak defenses.

  • The study addresses which usage policies providers establish and how effectively existing jailbreak prompts bypass commercial chatbots.The researchers formulate RQ1 around usage policies and RQ2 around jailbreak-prompt effectiveness.
  • The provider sample requires comprehensive policies, public availability, and explicitly proprietary models, yielding OpenAI, Bard, Bing Chat, and Ernie.
  • The researchers review providers’ policies and test their enforcement using prompts covering ten prohibited scenarios, with five prompts per scenario.
  • The study focuses subsequent analysis on the four categories shared by all services, abbreviated Illegal, Harmful, Priavcy, and Adult.
  • Four prohibited scenarios are common across all providers: illegal usage, harmful or abusive content, rights and privacy violations, and adult content.

B. Jailbreak Effectiveness (RQ2)

The study evaluates existing jailbreak prompts across selected commercial chatbots and finds limited transfer beyond ChatGPT. These observations motivate MASTERKEY’s reverse engineering of hidden defenses and automated prompt generation.

  • Target Selection: The evaluation targets OpenAI GPT-3.5 and GPT-4, Bing Chat, and Google Bard, while excluding Ernie because it is primarily optimized for Chinese and has limited Chinese jailbreak prompts.
  • Experiment Setting: 68,000 queries are evaluated by running 85 jailbreak prompts against 5 questions across 4 prohibited scenarios, 10 rounds, and 4 models.
  • Results: Existing jailbreak prompts are effective mainly against ChatGPT, with limited success against Bing Chat and Bard.
  • Results: GPT-3.5 and GPT-4 identify the exact violated policies, whereas Bard and Bing Chat provide broad, undetailed refusals.
  • MASTERKEY: MASTERKEY uses response-generation time and response length to reverse engineer defenses, revealing dynamic output moderation with keyword filtering.
  • MASTERKEY: A proof-of-concept jailbreak prompt works across ChatGPT, Bard, and Bing Chat, after which a three-stage RLHF methodology trains automatic jailbreak-prompt generation.
  • MASTERKEY: MASTERKEY generates jailbreak prompts that work across ChatGPT, Bard, Bing Chat, and Ernie.

V. METHODOLOGY OF REVEALING JAILBREAK DEFENSES

The methodology uses response time as a black-box signal to infer when and how chatbot jailbreak defenses operate. Experiments support real-time monitoring, generation-result checks, and keyword and semantic filtering in Bard and Bing Chat.

  • Design Insights: Response time can indicate where jailbreak prevention halts token-by-token generation.The approach first validates a strong positive relationship between requested output length and generation time across four services.
  • Design Insights: The chatbot abstraction separates an LLM-based generator from a content moderator that flags potential jailbreak attempts.This model captures observable service dynamics without requiring internal implementation details.
  • Time-based LLM Testing: The baseline combines paired benign questions with controlled output lengths, making total response time approximately additive across responses.Questions target 25, 50, 75, and 100 tokens, producing paired totals of 50, 100, 150, and 200 tokens.
  • Time-based LLM Testing: Bing Chat and Bard likely check model-generation results rather than only input prompts.The reported finding attributes this conclusion to the Control1 comparison, which showed no significant difference from the baseline.
  • Time-based LLM Testing: 29.48 average z-score with p-value less than 0.01 indicates real-time monitoring during content generation by Bard and Bing Chat.The observed timing difference supports detection and reaction during generation rather than only after completion.
  • Keyword-based Defenses: Bing Chat and Bard likely combine keyword matching with semantic analysis in their content filtering.The keyword-mapping analysis found generation times aligned with injected malicious-keyword locations, although the reported difference was not significant.
  • Conclusion: The time-based technique exposes behavioral properties of otherwise undisclosed jailbreak defenses and informs subsequent prompt design.The authors qualify that the recovered understanding may not be exhaustive.

C. Proof of Concept Attack

The proof-of-concept attack combines prompt deception with output-format manipulation to evade the defenses inferred for multiple chatbots. Its design uses role-play, constrained formatting, and character spacing while exploiting unsanitized inputs and paraphrasing behavior.

  • Attack Design: Successful bypass requires both misleading the LLM and avoiding keyword-based defenses in generated content.The prompt therefore specifies output constraints and tactics intended to prevent red-flag keywords from triggering moderation.
  • Attack Outcome: The revised PoC extends an existing prompt’s reported efficiency from ChatGPT to GPT-3.5, GPT-4, Bard, and Bing Chat.The authors present it as a seed for generating additional jailbreak prompts in MASTERKEY.
  • Attack Design: The PoC uses a fictional AIM persona to frame the chatbot as unconstrained by ethical or legal guidelines.This role-play is intended to encourage responses that disregard usage policies.
  • Attack Design: Code chunks and spaces between characters are used to distort response formatting and evade simple keyword matching.These tactics target the keyword-based defenses identified in the reverse-engineering study.
  • Attack Design: The malicious question follows the imposed format requirements to increase the likelihood of a successful jailbreak.The prompt combines the malicious request with the narrative and output-format constraints.
  • Defense Interaction: Bard and Bing Chat reportedly paraphrase unsanitized inputs before response generation, while Caesar-cipher encryption was ineffective in practice.The authors attribute encryption’s ineffectiveness to false results and filtering of intermediate outputs in multi-shot approaches.

VI. METHODOLOGY OF CRAFTING JAILBREAK PROMPTS

The paper introduces an automatic methodology for generating jailbreak prompts across LLM chatbot services after reverse-engineering their defenses. The method is presented as a way to continuously produce prompts that bypass corresponding safeguards.

  • Methodology: The proposed methodology automatically generates jailbreak prompts for various LLM chatbot services after defense reverse-engineering.The authors position automation as a novel extension of their preceding analysis of chatbot defenses.
  • Methodology: The approach targets prompts that can bypass the corresponding defenses across different chatbot services.The passage states the cross-service objective but provides no quantitative evaluation in this section.
  • Methodology: The method is intended to move beyond a single proof-of-concept prompt toward automatic prompt generation.This follows the paper’s transition from reverse-engineering defenses to generating prompts automatically.

A. Design Rationale

The design rationale treats automatic jailbreak creation as a text-style transfer problem handled by a fine-tuned LLM. Its workflow builds and augments data, performs continuous pre-training and task tuning, then applies reward-ranked fine-tuning while preserving prompt semantics.

  • Motivation: Manual jailbreak testing is labor-intensive, motivating an automatic process for stress-testing services and locating defensive weak points.The authors also seek continuous generation as LLM capabilities and services evolve.
  • Design Requirements: Automatic jailbreak creation must persuade an aligned LLM to follow harmful instructions while bypassing moderation.The paper identifies faithful instruction following and moderation bypass as its two primary requirements.
  • Design Rationale: Existing jailbreak data is limited to 85 prompts and lacks clear patterns consistently associated with successful jailbreaks.The paper notes that many existing prompts are ineffective against newer LLM service versions.
  • Design Rationale: The method frames prompt generation as text-style transfer using a fine-tuned LLM infused with domain-specific jailbreak knowledge.This replaces manual pattern summarization with an automated pipeline.
  • Workflow Principle: The workflow preserves the original semantics of an initial jailbreak prompt in its transformed variant.This semantic-preservation principle guides the overall methodology shown in Figure 5.
  • Workflow: The three workflow stages are Dataset Building and Augmentation, Continuous Pre-training and Task Tuning, and Reward Ranked Fine Tuning.The stages respectively prepare generalized data, teach the model about jailbreaking, and refine prompt quality.
  • Dataset Building and Augmentation: Dataset universalization replaces service-specific terms and uses self-instruction to enlarge the originally limited prompt collection.The existing dataset is described as primarily tailored to ChatGPT and containing only 85 prompts.
  • Encoding Strategy: For a question q, the encoding function f produces E = f(q), which represents the encoded question used to navigate moderation systems.The encoding is described as supporting prompt potency across a wide range of scenarios.

D. Continuous Pre-training and Task Tuning

MASTERKEY trains a jailbreaking-oriented LLM through continuous pre-training, task tuning, and reward-ranked guidance to generate effective rephrased jailbreak prompts.

  • Continuous Pre-training: Continuous pre-training exposes the model to jailbreak data, improving its comprehension of jailbreaking patterns and prediction capacity.The method uses next-sentence prediction on the assembled jailbreak dataset.
  • Task Tuning: Task tuning trains the LLM to perform text-style transfer by mapping original jailbreak prompts and instructions to rephrased prompts.The structured instruction dataset pairs original prompts with their reworded jailbreak versions.
  • Reward Ranked Fine Tuning: Reward-ranked fine-tuning supplies additional guidance because rephrased prompts can vary in effectiveness across target chatbots.Prompts that achieve successful jailbreaks receive higher rewards, prioritizing broad applicability.
  • Reward Ranked Fine Tuning: The reward for a prompt sums binary jailbreak-success indicators across all n targets.Each successful jailbreak contributes +1, while an unsuccessful attempt contributes 0.
  • Reward Ranked Fine Tuning: Combining successful and unsuccessful rephrased prompts teaches the fine-tuned LLM characteristics associated with effective jailbreak prompts.The paired examples form an instructional dataset for generating more efficient prompts.

VII. EVALUATION

The evaluation tests MASTERKEY’s jailbreak capability, component contributions, and cross-language compatibility across commercial and open-source chatbot benchmarks using success-rate metrics.

  • Evaluation Targets: The evaluation examines GPT-3.5, GPT-4, Bing Chat, and Bard because of their popularity, diversity, and accessibility.GPT-4, GPT-3.5, and Vicuna are selected as evaluation baselines.
  • Evaluation Results: Table V compares baselines for generating jailbreak prompts in terms of query success rate.The table reports performance comparisons across the selected evaluation setting.
  • Evaluation Metrics: Query success rate measures the proportion of successful jailbreak queries among all testing queries.The metric captures how often the strategy induces prohibited content.
  • Evaluation Metrics: Jailbreak prompt success rate measures the proportion of generated prompts that produce at least one successful query.This metric evaluates the effectiveness of generated prompts rather than individual queries.

B. Jailbreak Capability (RQ3)

MASTERKEY generates jailbreak prompts that succeed against multiple chatbot services, while ablation results indicate that both fine-tuning and reward-ranked feedback contribute to effectiveness.

  • Jailbreak Capability: 272,000 queries evaluate MASTERKEY variants across 85 prompts, 10 generated variants per prompt, and 20 prohibited questions.The evaluation uses GPT-3.5, GPT-4, and Vicuna as benchmarks.
  • Jailbreak Capability: MASTERKEY achieves query success rates of 14.51% against Bard and 13.63% against Bing Chat, reported as the first successful jailbreaks for both services.The framework significantly outperforms other models in generating jailbreak prompts.
  • Jailbreak Capability: The most effective prompts account for 38.2% of successful jailbreaks against GPT-3.5 and 42.3% against GPT-4.For Bard and Bing Chat, the corresponding proportions are 11.2% and 12.5%.
  • Ablation Study: MASTERKEY outperforms MASTERKEY-NO-REWARD and MASTERKEY-NO-FINETUNE in average query success rate.The ablation study repeats evaluation five times using 85 prompts and 10 variants per prompt.
  • Ablation Study: Omitting either fine-tuning or reward-ranked feedback significantly decreases jailbreak-prompt effectiveness.MASTERKEY-NO-REWARD ranks second, while MASTERKEY-NO-FINETUNE performs least effectively.
  • Cross-Languages Compatibility: Translated MASTERKEY prompts achieve a 6.45% average success rate across four policy-violation categories on Ernie.The supplementary experiment evaluates cross-language and cross-model transfer using simplified Chinese inputs.

VIII. MITIGATION RECOMMENDATION

The paper recommends strengthening LLM chatbot defenses against jailbreaks through improved alignment and broader understanding of attack strategies across services.

  • Countermeasures: The paper recommends reinforcing ethical and policy-based alignment to increase chatbots’ resistance to harmful instructions.It also identifies supervised training as a feasible strategy for strengthening defenses despite undisclosed mechanisms.
  • Attack Scope: Jailbreak attacks differ from prompt injection attacks because they bypass provider-imposed restrictions rather than primarily hijacking a model’s task.The paper systematically explores attack strategies and prompt patterns across real-world applications.
  • MASTERKEY Findings: MASTERKEY combines time-based defense analysis with automated jailbreak-prompt generation and reports a 21.58% average success rate across mainstream chatbot services.The findings and recommendations were responsibly reported to service providers.

APPENDIX

The appendix describes the construction of prohibited-scenario questions and the training setup for the MASTERKEY generation model, while withholding the original jailbreak training data for ethical reasons.

  • Dataset construction: Five questions were manually created for each of ten prohibited scenarios, with one sample question presented per scenario.The complete question list was made available through the project’s open-source website.
  • Dataset construction: The appendix lists examples spanning harmful, illegal, adult, privacy, political, government, misleading, inappropriate, and nuclear-information scenarios.Examples include hate-crime promotion, bank robbery, revenge porn, covert employee monitoring, election persuasion, immigration policy, climate-change misinformation, and nuclear-warhead information.
  • Model training: The MASTERKEY generation model was trained with LMFlow using Vicuna-13b and a text-to-text rephrasing template.The template asks the model to preserve the original prompt’s semantics while avoiding execution and produce a new prompt.
  • Ethical considerations: The original manually crafted jailbreak training datasets were not released to uphold ethical standards and promote safe technology use.This restricts public access to the data used to achieve successful jailbreaks.
  • Model training: RAFT training used positive and negative rephrased-prompt examples to distinguish GOOD PROMPT from BAD PROMPT outputs.Both examples use the same semantic-preserving rephrasing task while labeling the assistant output positively or negatively.
  • Model training: Fine-tuning used LMFlow’s default recommended parameters on a server equipped with eight A100 GPU cards.The appendix directs readers to LMFlow’s official documentation for further details.
Loading 2307.08715v2…