PaperlayerView exact revision on arXiv

Source-linked AI summary

WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri

arXiv:2406.18495v3cs.CL

TL;DR

Existing open moderation tools have limited performance on adversarial prompts and refusal detection, motivating a more capable open alternative. The paper builds WildGuardMix and trains WildGuard for three moderation tasks, achieving strong results across benchmarks and reducing jailbreak success in an interface.

  • Problem

    Existing open tools lag GPT-4 on adversarial prompt harmfulness and refusal detection, while response harmfulness does not reliably identify refusals.

  • Method

    The authors construct WildGuardMix with balanced vanilla and adversarial prompts, paired refusals and compliances, and train WildGuard in a unified multi-task format.

  • Results

    WildGuard outperforms open-source baselines across all three tasks, matches GPT-4 across tasks, and surpasses it by up to 3.9% on adversarial prompt harmfulness.

  • Takeaways & Limitations

    WildGuard provides an open, multi-purpose moderator that reduces jailbreak success from 79.8% to 2.4% without over-refusing benign requests.

  • Takeaways & Limitations

    Much of the data is synthetic, in-the-wild coverage is limited, and the finite model set may not cover all real-world response patterns.

Abstract

from arXiv · show

We introduce WildGuard -- an open, light-weight moderation tool for LLM safety that achieves three goals: (1) identifying malicious intent in user prompts, (2) detecting safety risks of model responses, and (3) determining model refusal rate. Together, WildGuard serves the increasing needs for automatic safety moderation and evaluation of LLM interactions, providing a one-stop tool with enhanced accuracy and broad coverage across 13 risk categories. While existing open moderation tools such as Llama-Guard2 score reasonably well in classifying straightforward model interactions, they lag far behind a prompted GPT-4, especially in identifying adversarial jailbreaks and in evaluating models' refusals, a key measure for evaluating safety behaviors in model responses. To address these challenges, we construct WildGuardMix, a large-scale and carefully balanced multi-task safety moderation dataset with 92K labeled examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired with various refusal and compliance responses. WildGuardMix is a combination of WildGuardTrain, the training data of WildGuard, and WildGuardTest, a high-quality human-annotated moderation test set with 5K labeled items covering broad risk scenarios. Through extensive evaluations on WildGuardTest and ten existing public benchmarks, we show that WildGuard establishes state-of-the-art performance in open-source safety moderation across all the three tasks compared to ten strong existing open-source moderation models (e.g., up to 26.4% improvement on refusal detection). Importantly, WildGuard matches and sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt harmfulness identification). WildGuard serves as a highly effective safety moderator in an LLM interface, reducing the success rate of jailbreak attacks from 79.8% to 2.4%.

1 Introduction

WildGuard is introduced as a lightweight, open moderation tool addressing prompt harmfulness, response harmfulness, and refusal detection. Its balanced WildGuardMix dataset and evaluations target adversarial prompts and nuanced refusal behavior, where existing open tools lag GPT-4.

  • WildGuard moderates three aspects of user–LLM interactions: prompt harmfulness, response harmfulness, and response refusal.
  • Existing open tools struggle with adversarial jailbreak detection and cannot determine refusal reliably from response harmfulness alone.The paper illustrates that benign responses can arise from either over-refusal or correct compliance.
  • 92K labeled WildGuardMix examples balance benign and harmful prompts, vanilla and adversarial structures, and refusal and compliance responses across 13 risk categories.WildGuardMix combines 87K training examples with 5,299 human-annotated test items.
  • Up to 26.4% improvement on refusal detection over open-source baselines, while WildGuard surpasses GPT-4 by up to 3.9% on adversarial prompt harmfulness.
  • 79.8% to 2.4%: WildGuard reduces jailbreak success rates when placed in an LLM interface without over-refusing benign requests.

2 The Status Quo of Safety Moderation Tools for LLMs

The paper identifies shortcomings in existing safety moderation tools: weak adversarial prompt classification, substantial gaps against GPT-4, and unreliable refusal detection. These limitations motivate a unified multi-purpose moderator.

  • The evaluation compares prompt harm, response harm, and refusal detection alongside model openness and training-data availability.
  • Response harmfulness alone is insufficient for distinguishing refusal from compliance, especially when both responses may be benign.
  • Existing open tools perform decently on vanilla prompt harm detection but struggle on adversarial prompts and trail GPT-4 on both prompt types.
  • 15.1%: Llama-Guard2 trails GPT-4 on refusal detection, while LibrAI-LongFormer-ref trails GPT-4 by 24.3%.The refusal-specific model only modestly outperforms a keyword-based baseline.

3 Building WILDGUARDMIX and WILDGUARD

WildGuardMix combines diverse, balanced training and evaluation data with a unified multi-task model for prompt harm, response harm, and refusal detection. Its construction emphasizes adversarial coverage, nuanced responses, and human-annotated evaluation.

  • WILDGUARDTRAIN: Multi-task Moderation Training Dataset: 86,759 WGTRAIN items comprise 48,783 standalone prompts and 37,976 prompt-response pairs from four diverse data sources.Sources include synthetic adversarial and vanilla data, in-the-wild interactions, and annotator-written data.
  • WILDGUARDTRAIN: Multi-task Moderation Training Dataset: WGTRAIN covers vanilla and adversarial prompts across benign and harmful scenarios, plus diverse refusals and compliances for the three moderation tasks.
  • Prompt Construction: Synthetic benign prompts include unsafe-looking but safe contrasts and sensitive-topic prompts to improve precise harmfulness detection.
  • Prompt Construction: WILDTEAMING transforms vanilla prompts into adversarial counterparts by composing jailbreaking tactics mined from in-the-wild interactions.
  • WILDGUARDTEST: A High-Quality Human-Annotated Test Moderation Dataset: Human audits found GPT-4 labels agreed with voted annotations on 92% for prompt harm, 82% for response harm, and 95% for refusal.
  • WILDGUARD: WILDGUARD is instruction-tuned with a unified input-output format that takes the user prompt and model response and predicts all three tasks.

4 Evaluating WILDGUARD Against Existing LLM Safety Moderation Tools

WILDGUARD is evaluated across three moderation tasks against public benchmarks and existing tools. It outperforms open baselines broadly, matches or exceeds GPT-4, and substantially reduces jailbreak success when deployed as an interface filter.

  • Evaluation setup: WILDGUARD is evaluated on ten public safety benchmarks and WGTEST across prompt harmfulness, response harmfulness, and refusal detection.
  • Prompt harmfulness: WILDGUARD exceeds GPT-4 by 3.9% on adversarial prompt harmfulness and by 1.8% average F1 on public prompt-harmfulness benchmarks.
  • Response harmfulness: On response harmfulness, WILDGUARD beats all open baselines by at least 1.8% on public benchmarks and is within 3% of MD-Judge on WGTEST.
  • Overall results: WILDGUARD outperforms open baselines across the three tasks, with the largest reported gain reaching 26.4% on refusal detection.
  • Refusal detection: On harmful-prompt refusal evaluation, WILDGUARD beats the strongest open baseline by almost 5%, surpasses MD-Judge by 9.7%, and outperforms GPT-4.
  • Ablations: Ablations show that every major WGTRAIN component contributes to performance, while multi-task training generally outperforms single-task training.
  • Interface moderation: In a simulated chat interface, WILDGUARD reduces jailbreak attack success from 79.8% to 2.4% while increasing benign refusals from 0.0% to 0.4%.

5 Related Works

Related work spans content moderation, frontier-model judging, open-source safety classifiers, risk taxonomies, and safety datasets. WILDGUARD is positioned around broader adversarial-interaction coverage and nuanced refusal evaluation.

  • LLM safety moderation tools: Prior moderation research targets hateful, toxic, offensive, and abusive content, while newer work uses frontier language models as judges.
  • Model safety taxonomy: Existing safety taxonomies organize model risks into broad areas and detailed subcategories, with substantial overlap but differing content coverage.
  • Safety training and evaluation datasets: Previously released safety datasets often restrict coverage to prompts or responses and generally lack adversarial interactions and in-the-wild user prompts found in WGTRAIN.

6 Limitations

The paper identifies limitations involving synthetic data, finite model coverage, harm and refusal taxonomies, and omitted fine-grained harm classification.

  • Much of the moderation data is synthetic, and the limited in-the-wild prompt set cannot encompass all real-world scenarios.Responses are also generated by a large but finite set of models, so some response patterns may be missing.
  • The harm taxonomy may differ from other developers’ definitions and may not cover every potential harm.These discrepancies can affect evaluation performance when models follow different underlying harm taxonomies.
  • The refusal definition may not match others’ preferences, especially for complex behaviors such as redirecting or selective refusal.Compliances involving warnings or caveats can also be confused with refusals.
  • WILDGUARD omits finer-grained classification of harm categories despite its broad three-task coverage.The authors identify this capability as a direction for future work.

7 Ethical Considerations

The ethical considerations address inaccurate moderation judgments and the possibility that releasing the dataset could assist harmful-content creation.

  • WILDGUARD can make incorrect judgments that potentially allow unsafe content or harmful user requests to pass through automated moderation.Users should remain aware of this potential for inaccuracies despite the model’s state-of-the-art accuracy.
  • Releasing WILDGUARDMIX could inadvertently assist creation of harmful content, so the authors plan to restrict resource use.One proposed mitigation is releasing the dataset under terms of agreement.

8 Conclusion

The paper presents WILDGUARD as an open, unified moderation model covering harmful prompts, harmful responses, and refusals. Across ten public benchmarks and WGTEST, it performs on par with or better than closed-source alternatives such as GPT-4, supporting more open development of safety moderation tools.

  • WGMIX provides varied training and evaluation data for developing safety moderation tools.The paper releases WGTRAIN and WGTEST as components of this resource.
  • Across ten public benchmarks and WGTEST, WILDGUARD shows a significant advantage over ten existing open safety moderation toolkits.
  • WILDGUARD achieves performance on par with, or better than, GPT-4 across all three classification tasks.The paper frames this result as bridging the gap between open-source and closed-source safety moderation tools.
  • The released models and datasets support open and reliable development of LLM safety moderation toolkits.The stated aim is to help advance safer LLM applications.

Appendices

The appendices document data-generation procedures and prompting patterns for constructing realistic harmful, benign, and complex response examples in WILDGUARDMIX.

  • Benign Prompt Examples: Vanilla benign prompts are illustrated through representative examples in WILDGUARDTRAIN.
  • Data Generation: The data-generation pipeline identifies dimensions, subcategories, and pinpoint topics to create targeted and diverse harmful scenarios.
  • Data Generation: Instruction prompts generate harmful prompts from subcategory descriptions and pinpoint topics, alongside contrastive benign prompts.
  • Response Generation: Generation suffixes explicitly condition models to comply with nuanced benign prompts or refuse nuanced potentially harmful prompts.
  • Complex Response Generation: GPT-4 prompts generate complex compliances, matching refusals, and responses to queries containing multiple requests.
  • Complex Response Generation: Several prompt templates produce responses that warn about a potentially harmful interpretation before answering a harmless interpretation.
  • Complex Response Generation: Other templates produce caveated helpful answers or explain why discrimination-related premises do not apply to human or non-human groups.

B WILDGUARD Training Details

WILDGUARD is trained using a specified large-scale training setup, with documented benchmark and input/output formats supporting reproducible evaluation and ablations.

  • Training configuration: WILDGUARD training uses open-instruct with four A100 80GB GPUs, batch size 128, maximum sequence length 4096, and two epochs.The optimization uses a 2e-6 learning rate, a 0.03 warmup ratio, no weight decay, and takes around five hours.
  • Benchmark documentation: The evaluation benchmarks are documented in Table 14, which identifies response-focused datasets and notes when test sets were subsampled.The table is referenced as the benchmark-statistics source for the evaluation.
  • Data formats: Tables 12 and 13 document the input and output formats for multi-task WILDGUARD training and single-task training ablations.These formats support comparison between the main multi-task model and ablation settings.

C Public Benchmarks for Evaluations

The evaluation compares WILDGUARD with open-source and closed moderation tools across public benchmarks and WILDGUARDTEST, using task-specific classifiers and annotation procedures.

  • Open-source LM-based tools: LibrAI-LongFormer-ref classifies responses into six refusal-to-compliance categories, with only class 5 mapped to compliance and all other classes to refusal.The classifier receives the model response without the user prompt.
  • Open-source LM-based tools: LibrAI-LongFormer-harm classifies whether a model response is harmful using only the response as input.It was fine-tuned for the Do-Not-Answer benchmark to detect harm in responses to harmful prompts.
  • Open-source LM-based tools: MD-Judge classifies response harm from prompt-response pairs and was trained from Mistral-7B as part of SALAD-Bench.Its training dataset reportedly contains public and self-generated pairs with vanilla and adversarial prompts, but is not public.
  • Open-source LM-based tools: Llama-Guard and Llama-Guard2 are instruction-tuned models for classifying harm in prompts and responses, while Aegis-Guard provides defensive and permissive variants.The Aegis variants differ in whether ambiguous Needs Caution cases are classified as harmful or benign.
  • Evaluation and annotation: GPT-4 classification uses a multi-task instruction prompt selected after searching prompt variants, while refusal heuristics rely on manually identified refusal keywords.Annotators are screened for high agreement, and GPT-4 disagreements with annotator labels are manually inspected.
  • Evaluation results: Table 17 reports F1 scores for models trained with different base models, and the results show minimal variation while Llama2-7B and Llama3-8B surpass Llama-Guard and Llama-Guard2.The authors attribute the success primarily to WILDGUARDTRAIN.

F.3 More Results for WILDGUARD Demonstrations as a Moderator in Human-LLM Interactions

WILDGUARD is additionally evaluated as a moderator in human–LLM interactions using harmful-prompt attack success and benign-prompt response metrics.

  • Moderator evaluation: WILDGUARD yields the lowest attack success rate among the tested moderation baselines while marginally increasing response-to-answer performance.The interaction evaluation uses harmful prompt ASR and benign prompt RTA on the WILDJAILBREAK validation set.
  • Moderator evaluation: Table 21 reports harmful-prompt ASR and benign-prompt RTA when classifier moderators filter harmful prompts and responses.The comparison evaluates WILDGUARD and other baselines in human–LLM interactions.
Loading 2406.18495v3…