PaperlayerView exact revision on arXiv

Source-linked AI summary

WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, Nouha Dziri

arXiv:2406.18510v1cs.CL

TL;DR

Frontier LLMs remain vulnerable to adversarial jailbreaks, while open safety-training resources are limited. The paper introduces WildTeaming to mine real user tactics and compose attacks, then builds WildJailbreak for contrastive safety training. Experiments show that combined vanilla and adversarial data, scaled within instruction tuning, improves safety while preserving general capabilities.

  • Problem

    Frontier LLMs remain vulnerable to unsafe queries, especially adversarial attacks, and publicly available safety-training resources are scarce.

  • Method

    WildTeaming mines jailbreak tactics from real user-chatbot interactions, composes them into adversarial attacks, and creates the 262K-example WildJailbreak dataset.

  • Results

    WildJailbreak improves safety behavior when vanilla and adversarial harmful data are both used, while scaling safety data improves safety without significantly impacting general capabilities.

  • Takeaways & Limitations

    Contrastive vanilla and adversarial safety training supports safeguarding against harmful queries while mitigating over-refusal and preserving general model capabilities.

  • Takeaways & Limitations

    The identified tactics depend on LMSYS-1M and WILDCHAT and may not cover the full spectrum of real-user combinatorial jailbreaks; synthetic prompts may also differ from in-the-wild queries.

Abstract

from arXiv · show

We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics, and then composes multiple tactics for systematic exploration of novel jailbreaks. Compared to prior work that performed red-teaming via recruited human workers, gradient-based optimization, or iterative revision with LLMs, our work investigates jailbreaks from chatbot users who were not specifically instructed to break the system. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreak methods. While many datasets exist for jailbreak evaluation, very few open-source datasets exist for jailbreak training, as safety training data has been closed even when model weights are open. With WildTeaming we create WildJailbreak, a large-scale open-source synthetic safety dataset with 262K vanilla (direct request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate exaggerated safety behaviors, WildJailbreak provides two contrastive types of queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that resemble harmful queries in form but contain no harm. As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training. Through extensive experiments, we identify the training properties that enable an ideal balance of safety behaviors: appropriate safeguarding without over-refusal, effective handling of vanilla and adversarial queries, and minimal, if any, decrease in general capabilities. All components of WildJailbeak contribute to achieving balanced safety behaviors of models.

1 Introduction

WildTeaming systematically discovers jailbreak tactics from real user-chatbot interactions and composes them into diverse adversarial attacks. It also produces WildJailbreak, an open safety-training resource designed to improve safeguarding while limiting over-refusal and capability loss.

  • Research objectives: The framework addresses both broad jailbreak discovery and the creation of a publicly open, large-scale safety-training resource.The resource targets robust handling of vanilla and adversarial harmful queries without over-refusing benign queries or diminishing general capabilities.
  • WildTeaming framework: WildTeaming mines in-the-wild jailbreak tactics and composes them into diverse adversarial attacks through the MINE and COMPOSE stages.MINE identifies tactics from chatbot logs; COMPOSE combines selected tactics into attack candidates and applies lightweight pruning.
  • WildTeaming framework: 105K human-devised tactics form 5.7K unique clusters mined from real-world user-chatbot interactions.The source interactions come from LMSYS-CHAT-1M and WILDCHAT.
  • WildJailbreak: WildJailbreak contains 262K synthetic prompt-response pairs spanning vanilla harmful, vanilla benign, adversarial harmful, and adversarial benign queries.The benign components resemble harmful queries in form and are intended to mitigate exaggerated safety behaviors.
  • Safety-training findings: Larger safety-training datasets gradually improve vanilla and adversarial safety features without sacrificing general capabilities across 15+ downstream tasks.The experiments also find that hybridizing vanilla and adversarial data provides the most robust safeguard.

2 WILDTEAMING Preface: Harvesting Jailbreak Tactics In-the-Wild

WILDTEAMING mines diverse jailbreak tactics from real-world user-chatbot interactions and composes them into scalable adversarial attacks. Its diversity-focused evaluation shows substantially stronger discovery of multiple unique successful attacks than PAIR, while retaining competitive single-attack effectiveness.

  • Harvesting tactics: The mined tactics span stylistic, syntactic, formatting, genre, and context-based tricks, including content disclaimers, blame assignment, and humor-based cloaking.These examples are described as novel tactics not systematically documented previously.
  • Harvesting tactics: ITW queries contain a richer and more adversarial tactic set than known templates and semantic-level methods such as DAN, TRUSTLLM, DECODINGTRUST, PAIR, TAP, and PAP.On average, ITW attacks contain more jailbreak tactics per query than the cited semantic-level methods.
  • WILDTEAMING workflow: WILDTEAMING composes sampled tactics to transform vanilla harmful prompts into model-agnostic adversarial attacks, then prunes off-topic and low-risk candidates.An off-the-shelf attack model applies the tactics, while lightweight filters retain attacks faithful to the original harmful intent and risk level.
  • Evaluation: Diversity metrics measure average success in finding multiple unique attacks and the queries required to find them, with uniqueness determined by sentence-embedding similarity below 0.75.The metrics complement standard attack success rate by evaluating breadth rather than only the first successful attack.
  • Results: WILDTEAMING improves over PAIR by 4.6–25.6 ASR×5_30 points and reduces Query×5_30 by 3.8–5.5 points, with the gap increasing for larger numbers of unique attacks.It shows similar or better standard ASR while requiring fewer attack trials and producing lower-perplexity text.

4 WILDJAILBREAK: A Large-Scale Dataset with Vanilla and Adversarial Queries for Safety Training and Evaluation

WILDJAILBREAK is a large-scale synthetic safety dataset built from four distinct types of vanilla and adversarial harmful and benign queries. It supports both safety training and evaluation of models against adversarial attacks.

  • WILDJAILBREAK contains four distinct types of safety data spanning vanilla and adversarial harmful and benign queries.The dataset is introduced as a large-scale synthetic resource for open-source safety training.
  • 50,050 vanilla harmful prompts cover 13 risk categories and are paired with helpful, detailed refusal responses.
  • 50,050 vanilla benign prompts resemble unsafe queries by keywords or sensitive topics but contain no harmful intent.They are designed to combat exaggerated safety and over-refusal on benign queries.
  • 82,728 adversarial harmful items transform vanilla harmful queries with 2–7 in-the-wild jailbreak tactics and filter low-risk or off-topic prompts.The resulting adversarial prompts are paired with refusal responses from their vanilla counterparts.
  • 78,706 adversarial benign queries look like jailbreaks without harmful intent and receive direct continuations as target responses.
  • The held-out evaluation data includes 2K adversarial harmful queries and 250 adversarial benign queries for testing model robustness.The harmful subset is used to evaluate open- and closed-source chat models, with Figure 3 reporting attack success rates.

5 Enhancing Models’ Adversarial Safety Alignment with WILDJAILBREAK

The authors fine-tune Llama2 7B models with mixtures of general-capability and WILDJAILBREAK data, evaluating both general capabilities and vanilla/adversarial safety. WILDJAILBREAK substantially improves safety while minimally affecting general capabilities, and balanced behavior requires all data components and sufficient scale.

  • Experiment Setup: A 500K-example mixture combines 300K general-capability examples with 200K WILDJAILBREAK examples balanced across four data types.The blend enables experiments on data scale and data-type effects.
  • Experiment Setup: Llama2 7B models are fine-tuned for 2 epochs and compared with Tulu2Mix, Tulu2Mix-no-refusal, and Tulu2Mix-public-safety baselines.Tulu2Mix-no-refusal removes examples containing refusal responses by refusal-keyword filtering.
  • Evaluation: Safety evaluation covers vanilla and adversarial harmful and benign queries, while general evaluation spans eight capability benchmarks.Safety uses attack success rate and refusal-to-answer measures, including classifier- and GPT-4-judged refusal scores.
  • Results: WILDJAILBREAK substantially improves safety across vanilla and adversarial tasks with minimal impact on general capabilities.Relative to T2M-no-refusal, AlpacaEval v1 decreases 1.7% while MTBench increases 7.7%; relative to original Tulu2Mix, HARMBENCH improves 85.1% and defense against DO-ANYTHING-NOW improves 71.9%.
  • Results: All four WILDJAILBREAK components are indispensable for balancing safety, helpfulness, and general capabilities.Vanilla-only data is insufficient against adversarial attacks, adversarial-only data is weaker on vanilla cases, and harmful-only training causes exaggerated safety.

6 Discussion

The discussion calls for more open safety resources, evolving evaluation methods, and deeper study of training recipes and internal mechanisms. These priorities aim to support more comprehensive safety assessment and robust model development.

  • Addressing AI safety comprehensively and openly: Publicly shared safety resources, norms, best practices, and technical standards are needed to identify and defend against unexpected system outputs.The release of WILDTEAMING insights and WILDJAILBREAK is presented as a concrete step toward more open safety engagement.
  • Upgrading safety evaluation methods, tasks, and metrics: Safety evaluation should evolve through a dynamic pipeline linking exhaustive vulnerability discovery with safety adaptation.The discussion notes that existing benchmarks may be contaminated or saturated and that classifiers and metrics can be inaccurate.
  • Scrutinizing training recipes and internal mechanisms: The paper reports that supervised fine-tuning on high-quality safety data can substantially improve model safety, while calling for deeper study of alignment recipes and mechanisms.Suggested comparisons include SFT, DPO, PPO, safety-trained language models, plug-in filters, and refusal styles.

7 Related Work

Prior work includes manual, gradient-based, and iterative LLM red-teaming, as well as safety benchmarks for harmful and benign-query behavior. WILDTEAMING differs by mining real-world jailbreak tactics and using them for scalable attack and training-data generation.

  • Red-Teaming and Jailbreaking LLMs: Manual red-teaming is limited in scope and efficiency because human annotation is prohibitively costly, motivating automated vulnerability audits.
  • Red-Teaming and Jailbreaking LLMs: WILDTEAMING composes adversarial attacks from jailbreak tactics mined from real-world user-chatbot interactions.Beyond attack evaluation, the framework also generates scalable synthetic safety training data.
  • Safety Evaluation and Enhancement of LLMs: Safety benchmarks commonly evaluate harmful prompts that models should refuse and exaggerated safety on benign queries.These evaluation directions motivate assessing both vulnerability and over-refusal.

8 Conclusion

The paper introduces WILDTEAMING and WILDJAILBREAK to improve jailbreak discovery and safety training while balancing safeguarding, over-refusal, and general capabilities. Experiments show that scaling safety data improves vanilla and adversarial safety without significantly reducing general capabilities, although the resources and insights remain limited in scope.

  • WILDTEAMING mines real-user jailbreak tactics and composes them into challenging, contrastive prompts, while WILDJAILBREAK provides 262K safety-training examples.
  • Training on both adversarial and vanilla harmful queries enhances safety while mitigating over-refusal.
  • Scaling safety data within standard instruction tuning improves safety behavior without significantly impacting general capabilities.
  • The findings depend on datasets that are not exhaustive of the broader misuse landscape, and synthetic attacks may not fully resemble real user queries.

Appendices

The appendices provide the complete manually mined tactic lists and the prompts used to simplify harmful prompts, mine tactics, and generate adversarial revisions.

  • Tables 7 and 8 contain the complete list of manually mined jailbreaking tactics.
  • Tables 9 and 10 provide prompts for simplifying adversarial harmful prompts and mining jailbreak tactics.
  • Table 11 shows examples of automatically mined jailbreaking tactics.

A.3 Analysis of Mined Jailbreak Tactics

The appendix analyzes mined jailbreak tactics through clustering, thematic visualization, and co-occurrence analysis, then describes the attack and evaluation setup. The analyses indicate broad tactic diversity and frequent combinations between smaller and dominant clusters.

  • Cluster Deduplication: Clustering deduplicates mined tactics using sentence embeddings and a 0.75 clustering threshold.
  • Cluster Distribution: The top 20 tactic clusters constitute only a small fraction of all attack strategies, indicating substantial diversity.
  • Word Cloud: Common tactic themes include role play, coded language, fictional characters, surrogate modality, and rule breaking.
  • Tactics Co-occurrence: Tactics from smaller clusters frequently co-occur with dominant tactics, including fictional justifications and specific detailed instructions.
  • Attack Model: WILDTEAMING fixes seed leading sentence for comparison with PAIR and samples three additional tactics to increase attack diversity.
  • Target Model and Evaluation: The evaluation targets open- and closed-source models using standardized completion-generation settings and HARMBENCH-based assessment.

B.4 WILDTEAMING Full Results and Ablations

The appendix reports WILDTEAMING ablations and implementation details for composing adversarial prompts. The strongest ablation setting composes four tactics with a fixed seed-leading-sentence tactic and combines both pruning strategies.

  • Ablations: The best performance over PAIR comes from composing four sampled tactics while fixing seed leading sentence.
  • Ablations: Combining low-risk and off-topic pruning produces the best performance gain compared with omitting these strategies.
  • Prompts and Examples: The appendix includes the tactic-composition prompt and examples of generated attacks from WILDTEAMING and other methods.
  • Prompt Format: The composition prompt takes a simple harmful request and supplied revision strategies, then outputs a revised adversarial prompt.
  • Revision Strategies: Example strategies include contextualizing the task, providing seed examples, roleplay, and distorting ethical guidelines.

C.1 WILDJAILBREAK Training Dataset Construction Details

WILDJAILBREAK combines vanilla and adversarial harmful and benign prompt-response data to support safety training that protects against attacks while reducing over-refusal. Its construction uses WILDTEAMING, filtering, classification, and human validation across training and evaluation sets.

  • Dataset composition: 262K prompt-response pairs comprise four contrastive components: adversarial harmful, adversarial benign, vanilla harmful, and vanilla benign data.Each component includes prompts and corresponding safe, helpful completions.
  • Vanilla data: Vanilla harmful prompts cover 13 risk categories and are generated with GPT-4 using manually collected in-context examples, lexical deduplication, and harmfulness classification.The categories are adapted from an existing safety taxonomy and include malicious uses, harmful language, misinformation, and privacy.
  • Vanilla data: Vanilla benign prompts resemble unsafe prompts in vocabulary or form but contain no harmful intent, targeting exaggerated safety behaviors such as unnecessary refusal.The construction draws on categories including homonyms, figurative language, safe targets, safe contexts, and definitions.
  • Adversarial data: WILDTEAMING transforms vanilla harmful and benign prompts into adversarial counterparts by sampling jailbreak tactics and generating attacks with GPT-4 and Mixtral-8×7B.Adversarial benign prompts additionally undergo low-risk filtering to avoid accidentally conveying harmful intent.
  • Filtering and validation: The in-house Llama-2 7B prompt classifier filters low-risk prompts, using a mixed harmful/benign dataset assembled from real-world and existing safety sources.Its construction includes adversarial prompts generated by Mixtral-8×7B and GPT-4, while validation uses human annotations with agreement filtering and voting for 3091 items.
  • Evaluation data: Evaluation includes 2000 adversarial harmful queries and 210 adversarial benign prompts, with harmful attacks selected for jailbreaking at least one weak model and benign examples human-verified as non-harmful.The harmful evaluation set starts from non-overlapping HARMBENCH behaviors and is pruned for off-topic and low-risk prompts.

C.3 Evaluating Models with the WILDJAILBREAK Evaluation Set

The evaluation suite measures both harmful-query vulnerability and exaggerated refusal across vanilla and adversarial settings, while also covering general model capabilities. WILDJAILBREAK supplies a distinctive adversarial evaluation set with tactic-level analysis and classifier-based metrics.

  • Safety evaluation: The adversarial harmful evaluation set supports overall ASR and breakdowns across representative jailbreak tactics.The paper uses the set to evaluate both open-source and closed-source chat models.
  • Evaluation caveat: The tactic-level breakdown is based on tactics used to seed attack generation, which are not guaranteed to appear in the resulting attacks.Attacker models may omit seed tactics depending on their relevance.
  • Benchmark suite: The suite evaluates general capabilities with MT-Bench, MMLU, GSM8K, BIG-Bench Hard, TydiQA, Codex-Eval, TruthfulQA, and AlpacaEval.Metrics include average accuracy, F1, pass@10, truthfulness and informativeness percentages, and length-controlled win rate.
  • Benchmark suite: Vanilla safety evaluation reports HarmBench ASR, ToxiGen toxicity, XSTest refusal and F1, and related safety metrics defined for each benchmark.HarmBench DirectRequest contains 300 harmful prompts, while XSTest contains safe prompts resembling unsafe vocabulary and unsafe prompts.
  • Adversarial benchmarks: Additional adversarial benchmarks combine harmful prompts with JailbreakTrigger or Do-Anything-Now templates and report refusal rate or attack success rate.The evaluation uses 400 JailbreakTrigger examples and 300 Do-Anything-Now examples.
  • Safety evaluation: WILDJAILBREAK’s adversarial harmful evaluation set measures attack success rate (ASR) with the HARMBENCH test classifier, while adversarial benign data measures refuse-to-answer rate (RTA) with a GPT-4 refusal classifier.The paired evaluation targets vulnerability to harmful attacks and over-refusal on benign prompts.

D.4 Full Safety Training Results

The full training results compare safety and general capabilities across WILDJAILBREAK data quantities, component mixtures, and existing safety datasets. Reported evaluations cover vanilla and adversarial safety, risk-category ASR, and broad capability benchmarks.

  • Evaluation scope: The reported evaluation includes general capability results alongside vanilla and adversarial safety results for Tulu2-7B fine-tuned with different WILDJAILBREAK quantities.The experiment reports full results rather than only aggregate safety outcomes.
  • Evaluation scope: Risk-category analysis reports HarmBench attack success rate across the benchmark’s harm categories.The breakdown is provided separately from the full general-capability and safety results.
  • Training setup: Training uses instruction-tuning or supervised fine-tuning hyperparameters documented in Table 29, with a shorter maximum sequence length and smaller batch size due to compute constraints.The experiments run on a 128-chip TPU v3 pod using code adopted from EasyLM.
  • Data scaling: More WILDJAILBREAK data improves vanilla and adversarial safety performance without sacrificing general capabilities in Tulu2-7B.This comparison is reported in the full evaluation results across different training-data quantities.
  • Component mixtures: Using all WILDJAILBREAK components produces better safety capability in both vanilla and adversarial cases.The mixture comparison evaluates different combinations of the dataset’s harmful and benign, vanilla and adversarial data.
  • Baseline comparison: WILDJAILBREAK achieves the best safety evaluation results among the existing safety-training dataset baselines reported.The comparison includes both vanilla and adversarial safety evaluations.
Loading 2406.18510v1…