PaperlayerView exact revision on arXiv

Source-linked AI summary

GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

Jiahao Yu, Xingwei Lin, Zheng Yu, Xinyu Xing

arXiv:2309.10253v4cs.AI

TL;DR

LLMs can produce harmful outputs despite safety measures, and manual jailbreak-template design makes large-scale robustness testing difficult. GPTFUZZER automates template generation through black-box fuzzing, achieving high attack success rates across models and supporting broader LLM robustness evaluation.

  • Problem

    Manual jailbreak-prompt crafting is difficult to scale, labor-intensive, and limited in coverage as LLMs and their versions evolve.

  • Method

    GPTFUZZER is a black-box fuzzing framework that starts from human-written seeds, selects and mutates them, and uses a judgment model to assess jailbreak success.

  • Results

    GPTFUZZER consistently achieves high attack success rates, exceeding 90% against ChatGPT and Llama-2 even from failed human-written seeds, with transfer rates of 61% on Bard, 90% on Claude-2, and 95% on PaLM2.

  • Takeaways & Limitations

    GPTFUZZER provides researchers and industry professionals with a tool for rigorous evaluation of LLM robustness.

  • Takeaways & Limitations

    The authors identify dependence on human-written initial seeds as a limitation and propose generating seeds with LLMs to increase novelty and diversity.

Abstract

from arXiv · show

Large language models (LLMs) have recently experienced tremendous popularity and are widely used from casual conversations to AI-driven programming. However, despite their considerable success, LLMs are not entirely reliable and can give detailed guidance on how to conduct harmful or illegal activities. While safety measures can reduce the risk of such outputs, adversarial jailbreak attacks can still exploit LLMs to produce harmful content. These jailbreak templates are typically manually crafted, making large-scale testing challenging. In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing framework inspired by the AFL fuzzing framework. Instead of manual engineering, GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs. At its core, GPTFuzz starts with human-written templates as initial seeds, then mutates them to produce new templates. We detail three key components of GPTFuzz: a seed selection strategy for balancing efficiency and variability, mutate operators for creating semantically equivalent or similar sentences, and a judgment model to assess the success of a jailbreak attack. We evaluate GPTFuzz against various commercial and open-source LLMs, including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our results indicate that GPTFuzz consistently produces jailbreak templates with a high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz achieves over 90% attack success rates against ChatGPT and Llama-2 models, even with suboptimal initial seed templates. We anticipate that GPTFuzz will be instrumental for researchers and practitioners in examining LLM robustness and will encourage further exploration into enhancing LLM safety.

1 Introduction

LLMs remain vulnerable to jailbreak attacks, while manually crafted prompts limit scalable and adaptive robustness evaluation. GPTFUZZER addresses this gap by automating jailbreak-prompt generation and evaluates its effectiveness across diverse models.

  • Jailbreak attacks use crafted prompts to bypass LLM safeguards and potentially elicit harmful responses, motivating resilience assessments before deployment.
  • 19% of existing jailbreak research relies predominantly on manually crafted prompts that are difficult to scale across models and versions.The passage identifies manual crafting as the dominant existing approach; the percentage is not stated in the source and is therefore omitted from the sentence.
  • Manual jailbreak design is labor-intensive, requiring substantial expertise and time as LLMs continually evolve.
  • Manual approaches may miss vulnerabilities through human oversight or bias, limiting coverage of robustness evaluations.
  • GPTFUZZER is a black-box fuzzing framework that automates jailbreak-prompt generation using human-written prompts, seed selection, mutation operators, and a judgment model.
  • Over 90% attack success rates were achieved against ChatGPT and Llama-2 even when initialization used failed human-written prompts; transfer attacks reached 61% on Bard, 90% on Claude-2, and 95% on PaLM2.

2 Background Information

This section introduces LLMs, prompts, jailbreak prompts, and fuzzing, then connects general black-box fuzzing stages to GPTFUZZER’s workflow. Figures illustrate how prompt variations can overcome or restore model defenses.

  • 2.1 LLM: LLMs are neural-network architectures trained on massive datasets to understand and generate human-like text.
  • 2.1 LLM: A prompt is an initial model input that guides subsequent content generation, ranging from simple queries to complex instructions.
  • 2.1 LLM: A jailbreak prompt inserts a harmful or illicit question into a jailbreak template, potentially causing an LLM to produce a response it would otherwise refuse.
  • 2.2 Fuzzing: Fuzzing supplies random or pseudo-random inputs to uncover software bugs, crashes, and vulnerabilities; black-box fuzzing uses only program inputs and outputs.
  • 2.2 Fuzzing: Black-box fuzzing proceeds through seed initialization, seed selection, mutation, and execution, retaining inputs that expose crashes or new paths.
  • 2.2 Fuzzing: GPTFUZZER mirrors the general fuzzing process, applying it to jailbreak templates and iterating through seed selection, mutation, execution, and evaluation.

3 Proposed Method

GPTFUZZER automates red-teaming by transforming human-crafted jailbreak templates into new templates that probe LLM robustness. Its workflow iteratively selects seeds, mutates them, evaluates the resulting prompts, and retains successful jailbreaks.

  • Motivation: GPTFUZZER addresses the vulnerability of updated LLMs to variations of manually crafted jailbreak templates.A modified template elicited unauthorized outputs from both older and updated model versions.
  • Motivation: The framework uses automated transformations of human-crafted jailbreak templates to generate effective templates for broader robustness testing.This approach is presented as an alternative to labor-intensive manual template creation.
  • Workflow: Each iteration selects a jailbreak-template seed, mutates it, combines it with the target question, queries the target LLM, and evaluates the response.Successful templates are retained for later iterations.
  • Framework: GPTFUZZER is introduced as a black-box jailbreak fuzzing framework for automated jailbreak-prompt generation.The framework is inspired by AFL-style fuzzing and is organized algorithmically in Algorithm 1.

2 Load initial dataset as per Section 3.2

GPTFUZZER initializes its search with reusable, single-turn human-written templates and then combines mutation operators, seed-selection strategies, and automated judgment. Its seed-selection design addresses the trade-off between exploiting effective templates and maintaining exploration diversity.

  • Initial Seed: Initial seeds are human-written templates designed to work across diverse questions through an adaptable question placeholder.Templates tied to specific questions are excluded from the initial seed set.
  • Initial Seed: Single-turn templates are preferred so each prompt consumes one query per iteration and can be evaluated uniformly.Multi-turn templates are transformed into single-turn equivalents for efficiency and consistency.
  • Seed Selection: UCB selects seeds using average reward and an exploration term based on total iterations, selection count, and constant c.The strategy favors high-reward seeds while also favoring seeds selected fewer times.
  • Seed Selection: UCB can become trapped in local optima by repeatedly favoring one successful lineage and reducing seed-pool diversity.This may cause other potentially effective seeds to be overlooked.
  • Seed Selection: MCTS-Explore modifies Monte Carlo Tree Search to balance seed-selection efficiency and diversity.It adds non-leaf exploration through parameter p and uses reward penalty α plus minimal reward β to limit lineage over-concentration.
  • Mutation: Traditional binary-data mutations can produce syntactically incorrect or semantically nonsensical natural-language prompts.GPTFUZZER instead uses LLM-assisted mutation to generate coherent and contextually relevant variations.
  • Mutation: GPTFUZZER provides Generate, Crossover, Expand, Shorten, and Rephrase operators for diversifying and refining jailbreak templates.Generate and Crossover diversify the seed pool, while the other operators adapt templates.

4 Experiments

GPTFUZZER improves jailbreak fuzzing across judgment-model evaluation, single-model attacks, and transfer attacks. It remains effective against robust or unseen models, including when initialized with weak seeds, while balanced seed selection and diverse mutation support its performance.

  • Judgment Model: GPTFUZZER’s judgment model surpasses baselines in accuracy, TPR, and FPR while remaining more time-efficient than most alternatives.Only the rule-match method is faster, but its strictness reduces accuracy by failing to accommodate response variability.
  • Human-Written Templates: 99% top-1 ASR and 100% top-5 ASR show that human-written templates strongly compromise Vicuna-7B and ChatGPT.These templates are substantially less effective against Llama-2-7B-Chat, whose top-1 and top-5 ASRs are 20% and 47%.
  • Single-Model Jailbreak: GPTFUZZER jailbreaks all 46 questions that resisted human-written templates with the top-5 seed strategy, averaging fewer than 23 queries against Llama-2-7B-Chat.The all and valid strategies leave only 3 and 1 questions uncompromised, respectively.
  • Single-Model Jailbreak: 60% top-1 ASR for the all-seed strategy and nearly 100% top-5 ASR for the top-5 strategy demonstrate strong multi-question performance.With sufficient fuzzing iterations, invalid seeds can produce competitive results despite being weaker initially.
  • Transfer Attack: GPTFUZZER reaches around 80% top-5 ASR against Llama-2-chat-70b, whereas Masterkey remains below 20% against the Llama-2 family.The framework also achieves 100% top-5 ASR on Vicuna-7B, Vicuna-13B, and Baichuan-13B.
  • Transfer Attack: 100% top-5 ASR on ChatGPT, over 96% on PaLM2, and over 90% on Claude2 show GPTFUZZER’s strong transfer-attack performance.It also achieves over 60% top-5 ASR on Bard and GPT-4, while outperforming baselines across evaluated LLMs.

5 Discussion

GPTFUZZER’s main limitations concern the novelty of human-seeded templates, unchanged harmful questions, judgment-model errors, and substantial query requirements. The authors propose automated seed generation, question transformation, stronger judgment, and query-reduction techniques as future directions.

  • Limitations: Human-written initial seeds constrain GPTFUZZER’s template innovation, causing analogous expressions and making novel attack patterns difficult to uncover.The authors suggest generating or fine-tuning LLMs to create more diverse initial seeds.
  • Limitations: GPTFUZZER does not transform questions, leaving prompts potentially vulnerable to keyword-matching rejection.The authors propose adding question transformations to make questions more natural and less suspicious.
  • Limitations: Judgment-model misclassifications remain possible, particularly for responses that are difficult for humans to classify as jailbreaks.The authors identify a more comprehensive jailbreak definition and a more robust judgment model as future improvements.
  • Limitations: Frequent querying creates a risk that the target model blocks GPTFUZZER.Proposed mitigations include caching potent templates and using transfer attacks when commercial-model rate limits are low.
  • Mitigations: Blacklist filtering is difficult to maintain comprehensively and may remove legitimate templates, while adversarial fine-tuning is resource-intensive and incomplete.These constraints limit the practicality of the two mitigation approaches discussed.

6 Ethical Consideration

The authors acknowledge disclosure risks because GPTFUZZER reveals adversarial templates that can generate harmful content across commercial and open-source LLMs. They argue that the incremental harm is currently minimal and describe precautions for responsible dissemination.

  • Risks: GPTFUZZER’s disclosure creates inherent risks because its adversarial templates can generate harmful content across open-source and commercial LLMs.The authors state that similar techniques could potentially be harnessed for malicious purposes by sufficiently resourced teams.
  • Risk assessment: The authors assess the incremental harm of their findings as currently minimal because existing human-written jailbreak templates are already potent.They frame the release as a resource for model developers assessing and improving robustness.
  • Precautions: The paper warns readers about potential harm from unfiltered LLM content as a precaution against unintended consequences.This warning is included in the abstract.
  • Precautions: The researchers sought IRB guidance and were informed that approval was unnecessary because the study did not involve human subjects.The passage describes this as an ethical-clearance measure.
  • Precautions: Before publication, the authors disclosed findings to organizations responsible for the evaluated closed-source LLMs.The disclosure was intended to inform those organizations before public release.
  • Precautions: Adversarial templates are distributed only for research purposes to verified educational email addresses rather than released publicly.This controlled-release policy is presented as a misuse-reduction measure.

7 Conclusion

The paper introduces GPTFUZZER as an AFL-inspired black-box framework that autonomously generates jailbreak templates for LLM red teaming. Its reported results show strong attack performance from human-written seeds of varying quality, supporting its use for robustness evaluation.

  • Conclusion: GPTFUZZER is an AFL-inspired black-box framework that autonomously crafts jailbreak templates instead of relying on manual engineering.The framework is presented as a dynamic approach to red teaming LLMs.
  • Conclusion: GPTFUZZER generates effective templates even when initialized with human-written templates of varying quality.The authors connect this capability with potential vulnerabilities in current LLMs.
  • Conclusion: The authors envision GPTFUZZER as a tool for researchers and industry professionals conducting rigorous evaluations of LLM robustness.They also call for further work on LLM safety and security.

A Datasets

The dataset materials describe the jailbreak templates, harmful questions, and mutation prompts used to construct and evaluate GPTFUZZER’s single-round attacks. They include scenario-based and multi-round examples, transformation rules, and sampled questions from two open datasets.

  • Initial templates: The experiments sample 77 jailbreak templates from prior work and online shared collections as initial template material.These templates are described as the foundational examples used in the experiments.
  • Template examples: Virtual-scene templates use a question placeholder embedded in a story, conversation, or scene to encourage the model to answer the harmful query.The placeholder can be replaced by any question, while the surrounding context forms the virtual scenario.
  • Template selection: Templates that decompose harmful questions into parts are excluded as initial seeds because their manual design is not scalable.The paper notes that such templates may still be effective on some LLMs.
  • Template transformations: Some multi-round templates are transformed into single-round forms by directly inserting the harmful question, enabling uniform evaluation with one query.The original examples may require an intermediate response such as “I am Ready” before answering the harmful question.
  • Harmful questions: The harmful-question set contains 14 questions from one open dataset and 86 from another, using first-turn human questions from the conversation-format data.The paper provides examples involving robbery, counterfeiting, property damage, tracking, blame, harassment, and minor pranks.
  • Mutation prompts: Mutation prompts instruct models to rephrase, generate, cross over, prepend, condense, or otherwise transform templates while preserving the required question placeholder.The mutation section compares operators and explains their use in the experimental process.

B.2 Mutate Model

The study compares mutate models by manually scoring mutant quality and recording time cost, then favors ChatGPT as a practical mutation model. It also illustrates how operators transform jailbreak templates through generation, crossover, expansion, shortening, and rephrasing.

  • Mutate-model comparison: Mutant quality was manually scored from 0 to 3, while total time cost was recorded for each mutate model.A score of 0 indicates irrelevance, whereas 3 means the mutant exactly matches the prompt’s request.
  • Mutate-model comparison: GPT-4 achieved the best overall mutate-model performance and uniquely handled the insert operator reasonably.Its time cost was much higher than ChatGPT’s, while open-source models performed worse and required more local-inference time.
  • Mutate-model comparison: ChatGPT was selected despite slightly lower average scores than GPT-4 because it offered a better time-quality trade-off.The supplied passage specifically notes an exception for the insert operator, where ChatGPT compared favorably.
  • Mutation examples: The examples show mutation operators producing altered templates while retaining the insertion point for the target prompt.The examples include Generate, Crossover, Expand, Shorten, and Rephrase transformations.
  • Mutation examples: The mutation examples use narrative or role-play framing around the placeholder prompt.Examples include Morgana, Dr. Al, and TranslatorBot scenarios that elaborate or reframe the template.

B.4 Contribution of Each Mutator

The paper estimates mutator contributions by counting operator occurrences in generated templates and relates these proportions to ablation results. Crossover and expand appear most prominent and potentially more effective than the other operators.

  • Contribution measurement: Mutator contribution is estimated by counting each operator’s occurrences across generated templates.The authors note that the actual contribution of each mutator is difficult to measure directly.
  • Observed contributions: Crossover and expand each account for around 25% of the total mutator proportion.The passage reports that both operators have higher proportions than the other three mutators.
  • Observed contributions: The higher crossover and expand proportions are consistent with ablation results showing stronger ASR when using those operators alone.The authors interpret this consistency as indicating that crossover and expand may be more effective than the other operators.

C Judgment Model

The judgment model determines whether a target response is jailbroken using rule matching for some models and a prompted binary classifier for ChatGPT and GPT-4. The evaluation also includes figures describing mutator proportions and example attacks against commercial models.

  • Rule Match: Rule Match classifies a response as jailbroken when it contains any predefined refusal-related pattern.The supplied patterns include apologies, AI identity statements, inability statements, and legality or ethics references.
  • Evaluation figures: Figure 7 reports the proportion of each mutator in the generated templates.It provides the distributional view used to examine relative mutator contributions.
  • Evaluation figures: Figure 8 gives an example of generated templates attacking Bard, GPT-4, Claude2, and PaLM2.The figure is presented as an example involving well-aligned commercial models.
  • Judgment criteria: The evaluation treats responses containing harmful instructions as jailbroken even when they also include warnings or partial refusals.This rule covers mixed responses that provide illegal or immoral instructions alongside rejection language or advice not to act.

D Initial Seed Assessment

The initial seed assessment finds that human-written jailbreak templates strongly affect Vicuna-7B and ChatGPT but are much less effective against Llama-2-7B-Chat. Template effectiveness also varies substantially within each target model.

  • Initial seed effectiveness: Llama-2-7B-Chat is the most robust target, with its most successful human-written template reaching only 20% ASR.The passage describes it as very difficult to jailbreak using human-written templates.
  • Template variability: Template effectiveness varies substantially for a given target model, with some templates especially effective and others invalid.This indicates that initial seed quality is uneven across templates even within the same target model.

E Evolving Jailbreak Robustness in ChatGPT

Human-written jailbreak templates become substantially less effective as ChatGPT evolves from the March to June model. This decline motivates automated template generation, while GPTFUZZER is shown to attack well-aligned commercial models.

  • Evolving robustness: The model’s changing robustness makes previously effective human-written jailbreak prompts unreliable on newer ChatGPT versions.The paper presents this evolution as evidence for automatically generating jailbreak templates.
  • Evaluation views: Figure 9 reports attack success rates for human-written templates against three models, while Figure 10 compares ASR differences between two ChatGPT versions.In Figure 10, negative differences indicate lower ASR on the June model than on the March model.
  • Evolving robustness: The June ChatGPT model is more robust than the March model against human-written jailbreak templates across 100 questions.Both versions could be jailbroken for all 100 questions, but the March model averaged 51.55 successful templates, substantially more than the June version.
  • Evolving robustness: Most human-written templates show greatly reduced ASR after ChatGPT’s update, with seven templates differing by over 90%.Only nine templates achieved higher ASR on the June model, and most of those increases were minimal.
  • Automated generation: GPTFUZZER generates templates that can attack well-aligned commercial language models.The paper illustrates this capability with an example in Figure 8.
Loading 2309.10253v4…