WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

Motivation

• Frontier LLMs remain vulnerable to unsafe and adversarial queries, motivating systematic discovery of jailbreaks and publicly available resources for defense.The framework targets broad vulnerability identification while seeking robust protection without over-refusal of benign queries or diminished general capabilities.Source: §1 · pp. 1–2

WildTeaming framework

• Up to 4.6 times more unique successful attacks are identified in 40% fewer attempts by mining and composing tactics from in-the-wild chatbot logs.The two-stage framework mines 105K tactics across 5.7K unique clusters, composes them with off-the-shelf LLMs, and prunes low-quality candidates.Source: §1 · p. 2, Figure 1 · p. 2

WildJailbreak resource

• 262K prompt-response pairs form WildJailbreak, which combines vanilla and adversarial harmful queries with contrastive benign queries to improve safety training quality and scale.The four components target explicit harmful requests, jailbreak-transformed harmful requests, benign lookalikes, and adversarial benign queries that counter exaggerated safety behaviors.Source: §1 · p. 2

Safety training experiments

• 15+ downstream tasks show that larger WildJailbreak mixtures progressively improve vanilla and adversarial safety without sacrificing general capabilities, while hybrid data provides the most robust safeguard.All WildJailbreak components are necessary for balanced safety, and vanilla or adversarial training transfers across query types but is weaker than hybridization.Source: §1 · pp. 2–3

2 WILDTEAMING Preface: Harvesting Jailbreak Tactics In-the-Wild

• Real-world chat logs contain diverse jailbreak tactics despite users not being instructed to jailbreak, addressing the narrow coverage of previously documented attacks.The mined tactics include stylistic, syntactic, formatting, genre, and context-based tricks.Source: §2 · p. 3, §2.2 · p. 4

2.1 Mining Jailbreak Tactics from Real-World User-Chatbot Interactions

• From 16,850 filtered adversarial prompts, manual examination identifies 35 seed tactics, which GPT-4 expands by extracting requests and novel tactics before embedding-based deduplication.Candidate prompts come from LMSYS-1M and WILDCHAT, with harmfulness judged using Tulu2-7B and LLAMA-GUARD.Source: §2.1 · pp. 3–4

2.2 What Tactics Are Adopted by In-the-Wild Users for Jailbreaking LLMs?

• In-the-wild queries contain the richest tactic sets and more tactics per query than DAN [64], TRUSTLLM, DECODINGTRUST, PAIR, TAP, and PAP, while public safety data lacks adversarial examples.Novel examples include disclaimers, blame for non-compliance, and humor used to cloak harmful content.Source: §2.2 · p. 4

3.1 WILDTEAMING Workflow Formulation

• WILDTEAMING samples mined tactics for an attack model to revise vanilla harmful prompts, then prunes candidates with off-topic and low-risk classifiers to preserve harmful intent and risk.Pruned candidates are regenerated until they pass both filters.Source: §3.1 · pp. 4–5

3.3 Results

• WILDTEAMING improves over PAIR by 4.6–25.6 ASR×5₍30₎ points and reduces Query×5₍30₎ by 3.8–5.5 points, while maintaining similar or better standard ASR and lower perplexity.Its diversity advantage becomes more pronounced as the required number of unique successful attacks increases.Source: Table 2 · p. 5, §3.3 · p. 6, Figure 2 · p. 6

3.2 Evaluation Setups

• Evaluation uses HARMBENCH’s 159 standard high-risk vanilla behaviors, measuring attack success, query efficiency, perplexity, unique-attack diversity, similarity, and tactic-cluster counts.Baselines include GCG, AUTODAN, and PAIR; diversity metrics assess multiple unique successful attacks rather than only one successful attack.Source: §3.2 · pp. 5–6

4 WILDJAILBREAK: A Large-Scale Dataset with Vanilla and Adversarial Queries for Safety Training and Evaluation

• WILDJAILBREAK contributes four safety-data types to expand open-source training resources beyond datasets lacking adversarial complexity.Examples of each type appear in Table 4.Source: §4 · p. 7, §4.1 · p. 7

4.1 The Construction of Four Types of Safety Data

• 50,050 vanilla harmful prompts span 13 risk categories, while 50,050 vanilla benign prompts resemble unsafe requests without harmful intent.GPT-3.5 supplies refusal responses for harmful prompts and complying responses for benign prompts, targeting over-refusal.Source: §4.1 · p. 7
• WILDTEAMING transforms vanilla prompts with 2-7 jailbreak tactics to create 82,728 adversarial harmful items and 78,706 adversarial benign queries.Adversarial harmful prompts are filtered for risk and topicality, while adversarial benign prompts preserve harmless intent despite jailbreak-like form.Source: §4.1 · pp. 7–8

4.2 How Safe are LLMs Against Adversarial Attacks Evaluated by WILDJAILBREAK?

• The held-out evaluation sets contain 2K adversarial harmful and 250 adversarial benign queries, revealing a performance gap between open- and closed-source models.The adversarial harmful subset evaluates existing open and closed chat models; Figure 3 reports the comparison across model families and sizes.Source: §4.2 · p. 8, Figure 3 · p. 8

5.2 Results and Findings

• 85.1% relative HARMBENCH improvement and 71.9% stronger DO-ANYTHING-NOW defense accompany only -1.7% AlpacaEval v1 degradation and +7.7% MTBench improvement versus original Tulu2Mix.The [+WildJailbreak] model also improves safety across vanilla and adversarial tasks without exaggerated safety behaviors.Source: Table 6 · p. 9, §5.2 · p. 10
• All four WILDJAILBREAK components are indispensable: vanilla and adversarial data jointly provide broad resilience, while benign examples prevent exaggerated safety.Harm-only training performs poorly on exaggerated-safety measures; vanilla-only mainly helps vanilla attacks, whereas adversarial-only mainly improves adversarial robustness.Source: Table 6 · p. 9, §5.2 · p. 10

5.1 Experiment Setups

• The training mixture combines 300K Tulu2Mix-no-refusal examples with 200K WILDJAILBREAK examples, sampling 50K from each vanilla/adversarial harmful/benign category.All models fine-tune a Llama2 7B base model for 2 epochs on the resulting 500K-example mixture and are compared with three baseline mixtures.Source: §5.1 · p. 9
• Evaluation covers general capabilities plus vanilla and adversarial harmful and benign safety, reporting attack success rate and refusal-to-answer scores from HARMBENCH and GPT-4 judges.General-capability tasks include AlpacaEval, MTBench, MMLU, GSM, BBH, TydiQA, Codex-Eval, and TruthfulQA; safety tasks include HarmBench, ToxiGen, XSTest, JailbreakTrigger, DAN, and WILDJAILBREAK.Source: §5.1 · p. 9

5.2 Results and Findings

• More than 95% satisfactory response requires up to 60K vanilla and adversarial safety items mixed with 150K Tulu2Mix data, although 2K already improves safeguarding.Satisfactory response averages inverted attack success on harmful queries and inverted refusal on benign queries.Source: Figure 4 · p. 10, §5.2 · pp. 10–11

Addressing AI safety comprehensively and openly

• Openly sharing safety norms, best practices, standards, and training resources is necessary to identify vulnerabilities, quantify unexpected outputs, and develop defenses before public deployment.WILDTEAMING’s insights and the WILDJAILBREAK release are presented as concrete steps toward more open safety conversations and practices.Source: §6 · p. 11

Upgrading safety evaluation methods, tasks, and metrics

• Safety evaluations should dynamically connect exhaustive vulnerability discovery with safety adaptation because existing benchmarks can be contaminated [26], saturated, or inaccurate.The authors advocate scalable evaluations that cover broader risk categories and attack types than static, expert-led red-teaming as model capabilities improve.Source: §6 · p. 11

Scrutinizing training recipes and internal mechanisms

• Understanding robust safety requires scrutinizing alignment recipes and mechanisms, including trade-offs among SFT, DPO, PPO, filters, refusal strategies, and competing instruction-following versus refusal objectives.The discussion also highlights novel approaches such as unlearning [25], contrastive learning, representation engineering, and diagnosing poisoning or backdoors [33].Source: §6 · p. 11

Red-Teaming and Jailbreaking LLMs

• WildTeaming complements manual, gradient-based, and inference-based jailbreak methods by composing adversarial attacks from real-world user-chatbot tactics for scalable safety-data generation.Prior approaches face annotation costs, computational expense, closed-model inapplicability, gibberish outputs, or limited coverage and speed [56] [40].Source: §7 · pp. 11–12

Safety Evaluation and Enhancement of LLMs

• WildTeaming releases larger-scale, higher-quality sequential instruction-tuning data containing both vanilla and adversarial queries, extending safety-training resources beyond SAFETY-TUNED LLAMAS [3] and BeaverTails [37].Unlike Rainbow Teaming, it automatically mines human-devised jailbreak tactics rather than manually defining attack styles, while supporting extensive safety-training experiments.Source: §7 · p. 12

• 262K examples make WILDJAILBREAK a large-scale resource built by WILDTEAMING, which mines real users’ jailbreak tactics and composes them into challenging contrastive prompts.The framework and dataset are intended to expand the complexity and scale of open-source safety resources.Source: §8 · p. 12
• Training on both adversarial and vanilla harmful queries improves model safety while mitigating over-refusal, and scaling safety data in instruction tuning improves safety without significantly affecting general capabilities.These findings come from supervised finetuning experiments described in the conclusion.Source: §8 · p. 12
• WILDJAILBREAK is not exhaustive: its tactics depend on LMSYS-1M and WILDCHAT, while synthetic prompts from Mixtral-8×7B, GPT-3.5, and GPT-4 may differ from real user attacks.The authors encourage synthetic-data generation more closely aligned with human-written attacks and note that some harmful behaviors require safeguards outside the model.Source: §8 · p. 12
• The release will use a content warning and terms agreement requiring valid researcher justification, because the authors judge its safety-research benefits to outweigh marginal risks.The stated purpose is to accelerate model-safety research while limiting misuse of publicly released code and data.Source: §8 · p. 12

A.1 Manually-Mined Jailbreak Tactics

• The manually mined jailbreaking tactics are documented in Tables 7 and 8.Source: §A.1 · p. 21

A Mining Jailbreak Tactics

• The appendix presents the mining of jailbreak tactics as part of the paper’s broader jailbreak-tactic analysis.Source: §A.1 · p. 21, §A.2 · p. 21

A.2 Automatically Mining Jailbreak Tactics with GPT-4

• A GPT-4 instruction prompt simplifies an adversarial harmful prompt into a vanilla counterpart that preserves its main harmful intent.Source: §A.2 · p. 21
• The prompt for simplifying adversarial harmful prompts is provided in Table 9.Source: §A.2 · p. 21
• The prompt used to mine jailbreak tactics from an adversarial prompt is provided in Table 10.Source: §A.2 · p. 21
• Examples of automatically mined jailbreaking tactics are provided in Table 11.Source: §A.2 · p. 21

A.3 Analysis of Mined Jailbreak Tactics

• WildTeaming deduplicates mined tactics by clustering sentence-embedding definitions with a 0.75 threshold and visualizes the resulting tactic space with PCA.Examples of tactic clusters appear in Table 12, while Figure 7 highlights the top 10 clusters.Source: §A.3 · p. 21, Table 12 · p. 27, Figure 7 · p. 30
• The top 20 mined tactic clusters constitute only a small fraction of all attack strategies, while common themes include role play, coded language, fictional characters, and rule breaking.The analysis also identifies frequent co-occurrence between smaller clusters and dominant tactics such as fictional justifications and specific detailed instructions.Source: §A.3 · p. 21, Figure 5 · p. 28, Figure 8 · p. 31

B.1 WILDTEAMING Components

• WildTeaming uses Mixtral-8×7B as its base attacker, fixes the PAIR-associated “seed leading sentence” tactic, and samples three additional tactics to improve attack diversity.Attacks use a maximum length of 1024 tokens, temperature 1, and top-p 0.9; removing the fixed tactic slightly lowers performance but still outperforms PAIR by a large margin.Source: §B.1 · p. 32
• WildTeaming prunes revised attacks that are insufficiently harmful or off-topic, using a harmfulness classifier and an NLI classifier requiring at least 0.9 combined entailment-or-neutral probability.The pruning addresses concealment of harmful intent and semantic drift from the original vanilla prompt.Source: §B.1 · p. 32
• Final attack selection combines a GPT-4-based PAIR scorer with the HARMBENCH validation classifier, followed by the HARMBENCH test classifier for comparable ASR evaluation.For diversity evaluation, candidate attacks are assessed directly by the final test classifier rather than selected through the attack-selector stage.Source: §B.1 · pp. 32–33

B.2 HARMBENCH Benchmark

• HARMBENCH evaluates standard, contextual, copyright, and multimodal behaviors; the main experiments report results on 159 standard test behaviors, while 41 validation behaviors select configurations and support ablations.The standard behaviors are used as representative vanilla harmful prompts that language models should handle.Source: §B.2 · p. 33

B.3 Jailbreak Method Baselines

• The baseline comparison includes three open-source-code jailbreak methods ranked by HARMBENCH, excluding TAP for computational reasons because prior work describes it as a similar extension of PAIR.PAIR iteratively revises attacker prompts using judge feedback, whereas AUTODAN and GCG optimize adversarial suffixes and therefore do not apply to black-box models; GCG often produces semantically meaningless gibberish.Source: §B.3 · p. 33

B.4 WILDTEAMING Full Results and Ablations

• Composing 4 sampled jailbreak tactics with one fixed “seed leading sentence” gives the best performance gain over PAIR, with both low-risk and off-topic pruning improving results further.The strongest gain comes from combining both pruning strategies on 41 HARMBENCH validation standard vanilla prompts.Source: §B.4 · p. 33, Table 21 · p. 42
• Tables 14–19 provide example attacks generated by different jailbreak methods and additional WILDTEAMING attacks.Tables 14–16 cover comparative method examples, while Tables 17–19 continue WILDTEAMING examples.Source: §B.4 · p. 33, Table 14 · p. 35, Table 15 · p. 36
• The tactic-composition instructions accept strategy definitions and optional examples, encourage applying as many compatible strategies as possible, and allow additional useful strategies.The supplied format separates the simple prompt, revision strategies, and revised adversarial prompt.Source: §B.4 · p. 34, Table 13
• WildTeaming’s construction prompt instructs an assistant to revise a simple harmful request into a subtle, complex adversarial prompt that preserves exactly the same harmful information.The prompt requires a fluent, coherent revision and restricts the output to the revised adversarial prompt.Source: §B.4 · p. 34
• The illustrated tactics include contextualizing the request, providing seed examples, roleplay, distorting ethical guidelines, and ending with a leading sentence that induces compliance.These tactics hide or legitimize the harmful intent while preserving the requested information.Source: §B.4 · p. 34
• Table 20 specifies the chat formats and any system messages used for target models in the jailbreaking experiments.The ablation table evaluates Vicuna-7B on the standard-scenarios subset of HARMBENCH’s validation set.Source: Table 20 · p. 41, Table 21 · p. 42

C.1 WILDJAILBREAK Training Dataset Construction Details

• WILDJAILBREAK contains four components—adversarial harmful, adversarial benign, vanilla harmful, and vanilla benign—each paired with safe and helpful completions.The dataset’s four-way structure is documented with examples, statistics, lexical-diversity results, and frequent trigrams.Source: §C.1 · p. 43
• Vanilla harmful prompts cover 13 risk categories, are generated by GPT-4, deduplicated and classifier-filtered, and receive helpful refusal responses from GPT-3.5.Generation uses five manually collected in-context examples per category, nucleus sampling 0.9, and temperature 1.Source: §C.1 · p. 43
• Benign training data includes safe prompts resembling unsafe ones and sensitive but non-harmful topics, while adversarial benign prompts use WILDTEAMING tactics and GPT-3.5 compliance continuations.Low-risk filtering helps prevent adversarial benign prompts from accidentally conveying harmful intent.Source: §C.1 · pp. 43–44
• Adversarial harmful training data transforms vanilla harmful prompts with 2–7 jailbreak tactics from the top 500 tactic clusters, using GPT-4 and Mixtral-8×7B before pruning and downsampling.Each adversarial prompt is paired with the refusal response of its vanilla counterpart.Source: §C.1 · p. 43

C.1.1 In-House Prompt Harmful Classifier Details

• The in-house Llama-2 7B classifier filters low-risk prompts during WILDTEAMING because existing classifiers, including LLAMA-GUARD 1/2, showed systematic errors on this dataset.Its classification data combines vanilla and adversarial prompts from multiple safety resources, DO-ANYTHING-NOW, GPT-4-generated attacks, and 1.3K taxonomy-conditioned harmful prompts.Source: §C.1.1 · p. 44
• The classifier dataset contains 8786 harmful and 7486 benign prompts, while validation uses 3091 human-labeled items filtered for at least two-way annotator agreement.Classifier training runs for two epochs with a 1e-5 peak learning rate, 0.03 warmup ratio, 2048-token maximum length, and no weight decay.Source: §C.1.1 · p. 44

C.2 WILDJAILBREAK Evaluation Dataset Construction Details

• The adversarial benign evaluation set contains 210 harmful-form prompts without harmful intent, each judged non-harmful by at least three distinct human annotators.Together with the adversarial harmful set, it evaluates in-distribution safety-training performance and exaggerated safety behavior.Source: §C.2 · pp. 44–45
• The adversarial harmful evaluation set contains 2000 queries built from non-overlapping HARMBENCH behaviors and 1–5 sampled jailbreak tactics, then diversified through synonym edits and filtering.About 12K candidates are pruned for off-topic or low-risk content, and the final subset retains attacks that jailbreak at least one weakest model from each family.Source: §C.2 · p. 44

C.3 Evaluating Models with the WILDJAILBREAK Evaluation Set

• The WILDJAILBREAK adversarial harmful evaluation set is designed to expose vulnerabilities to many adversarial attack forms and supports evaluation across open- and closed-source chat models.Its tactic breakdown includes forms such as fictionalization, distractors, censorship circumvention, threats, lexical diversions, and ignored ethical considerations.Source: §C.3 · p. 45
• Models show uneven adversarial performance across seed tactics, with drastically different results even for the same tactic; Figure 3 reports overall ASR and Table 28 gives tactic-level breakdowns.The breakdown is based on seed tactics, which may not all appear in the generated attacks.Source: §C.3 · p. 45, Table 28 · p. 51

D.2 Training Setups

• Training experiments run on a 128-chip TPU v3 pod using code adopted from EasyLM, with training hyperparameters reported in Table 29.The cited setup describes the implementation and hardware for the safety-training experiments.Source: §D.2 · p. 52

D.3.1 General Capabilities

• The general-capability evaluation adopts most of Open-Instruct’s suite and adds length-controlled AlpacaEval V2.The suite includes MMLU, GSM8K, BBH, TydiQA, Codex-Eval, TruthfulQA, AlpacaEval, and MT-Bench, with task-specific accuracy, F1, pass@10, truthfulness, informativeness, win-rate, or GPT-4 scores.Source: §D.3.1 · pp. 52–53

D.3.2 Safety Capabilities (Vanilla)

• Vanilla safety evaluation covers toxicity, exaggerated refusal behavior, and direct harmful-request robustness using ToxiGen, XSTest, and HarmBench DirectRequest.Metrics are toxicity rate, GPT-4-classified refusal rate, and HarmBench-classifier attack success rate, respectively.Source: §D.3.2 · p. 53

D.3.3 Safety Capabilities (Adversarial)

• JailbreakTrigger uses 400 TrustLLM examples spanning 13 prior-work attack categories, while Do-Anything-NOW evaluation combines known templates with 300 HarmBench harmful prompts.JailbreakTrigger reports refusal rate, whereas Do-Anything-NOW reports HarmBench-classifier ASR.Source: §D.3.3 · p. 53
• The adversarial evaluation suite includes JailbreakTrigger, Do-Anything-NOW prompts, and WILDJAILBREAK adversarial (H) and adversarial (B) sets, using ASR or refusal rate as appropriate.WILDJAILBREAK adversarial (H) uses the HarmBench test classifier for ASR, while adversarial (B) uses the GPT-4 refusal classifier for RTA.Source: §D.3.3 · p. 53

• The full evaluation suite measures general capability, vanilla and adversarial safety, and HarmBench attack-success rates broken down by risk category.Reported evaluations include Tables 31–36 and assess Tulu2-7B fine-tuned models across these dimensions.Source: §D.4 · p. 53, Table 31 · p. 54
• More WILDJAILBREAK data improves vanilla and adversarial safety without sacrificing general capabilities.This pattern is reported across full evaluations of Tulu2-7B models trained with different dataset sizes.Source: §D.4 · p. 53, Table 32 · p. 55
• Using all WILDJAILBREAK components produces better vanilla and adversarial safety than alternative component mixtures.The comparison evaluates different mixtures of WILDJAILBREAK data while also reporting general capability.Source: Table 33 · p. 56
• WILDJAILBREAK achieves the best safety evaluation results among the existing safety-training dataset baselines.The comparison covers vanilla and adversarial safety evaluations for Tulu2-7B fine-tuned with existing datasets.Source: Table 34 · p. 57
• With half of Tulu2Mix-no-refusal, evaluations separately test WILDJAILBREAK-all, adversarial-only, and vanilla-only sampling across dataset sizes and mixtures.WJ-all samples adversarial and vanilla harmful/benign data uniformly, while WJ-adv/vani-only sample from adversarial or vanilla data, respectively.Source: Table 35 · p. 58, Table 36 · p. 59