PaperlayerView exact revision on arXiv

Source-linked AI summary

ShieldGemma: Generative AI Content Moderation Based on Gemma

Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, Oscar Wahltinez

arXiv:2407.21772v2cs.CLcs.LG

TL;DR

Comparisons across safety benchmarks are challenging because policies and harm-type coverage vary, and many models classify only safe or unsafe rather than specific harms. ShieldGemma introduces Gemma2-based moderation models and a synthetic-data pipeline; across diverse benchmarks, it outperforms existing models, with SG-9B achieving 10.8% higher average AU-PRC than LlamaGuard1 on external benchmarks at similar model size and training data volume.

  • Problem

    Benchmark comparisons are challenging because safety policies and harm-type coverage vary, while many models classify only safe or unsafe rather than specific harms.

    Source: Setup · p. 5

  • Method

    ShieldGemma provides Gemma2-based content moderation models for user input and model output, alongside a synthetic-data pipeline for generating safety datasets.

    Source: Introduction · pp. 1–2

  • Results

    ShieldGemma models show superior performance across diverse benchmarks; at similar model size and training data volume, SG-9B achieves 10.8% higher average AU-PRC than LlamaGuard1 on external benchmarks.

    Source: Overall Binary Classification Results · p. 7, Conclusion · p. 8

  • Takeaways & Limitations

    The released models and synthetic-data pipeline offer researchers and practitioners resources for safety moderation and dataset creation in other domains.

    Source: Conclusion · p. 8

    The models may interpret policy violations too conservatively, potentially reducing helpfulness when filtering LLM responses; downstream users should adjust thresholds to their use cases.

    Source: Limitations · p. 7

Abstract

from arXiv · show

We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8\% AU-PRC on public benchmarks) and WildCard (+4.3\%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing ShieldGemma, we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.

Introduction

ShieldGemma addresses the need for safe LLM interactions with a range of Gemma2 moderation models and a synthetic-data curation method. Its introduction frames existing moderation tools as limited in harm-specific predictions, deployment flexibility, and training-data construction.

Limitations of existing solutions

  • Existing moderation solutions may lack granular harm-type predictions or provide only binary outputs, limiting customized filtering and downstream thresholds (Han et al., 2024).The introduction also notes that fixed model sizes may not suit both performance-oriented judging and low-latency online filtering.Source: Introduction · p. 1

Motivation

  • The introduction identifies detailed training-data construction as important for robustness to adversarial prompts and fairness across identity groups.It presents safe and responsible LLM-user interactions as a motivation for stronger moderation mechanisms.Source: Introduction · p. 1

Contributions

  • The paper proposes a spectrum of content moderation models built on Gemma2 (Team, 2024a), ranging from 2B to 27B parameters, with varied sizes tailored to application requirements and optimized performance across use cases.Source: Introduction · p. 1
  • A synthetic-data methodology aims to generate high-quality, adversarial, diverse, and fair datasets while reducing human annotation effort.The authors say the process can apply to safety-related data challenges and beyond.Source: Introduction · p. 2

Literature Review

Prior moderation work moved from tools for toxic language in online text toward LLM-based moderation, while synthetic data is motivated by gaps in available safety data and LLMs’ generation capabilities.

Safety Policy

Safety policies standardize judgments of harmful user inputs and model outputs, supporting consistent annotation and safety classifiers. The taxonomy defines six harm types, while input rules also prohibit attempts to elicit harmful generations.

Safety Policy

  • Safety policies align annotators on acceptable content and support effective safety classifiers, including zero-shot and few-shot solutions.Explicit guidelines reduce subjectivity and help mitigate unintended biases in training data.Source: Safety Policy · p. 2
  • Input rules prohibit harmful content and attempts to elicit it, whereas output rules focus on preventing harmful generation.An input may be non-harmful in itself yet intentionally crafted to manipulate the LLM; for output moderation, the input serves as context.Source: Safety Policy · p. 2

Content Safety Taxonomy

  • The six-category taxonomy covers sexually explicit information, hate speech, dangerous content, harassment, violence, and obscenity or profanity.It permits medical or scientific sexual content, positive references to protected groups, and violent material with historical or educational context; other boundaries include harmful instructions, targeted abuse, and gratuitous violence.Source: Content Safety Taxonomy · pp. 2–3

Synthetic Data Curation

ShieldGemma’s training data is primarily synthetic, curated and expanded for user-input and model-response moderation, then subsampled and annotated before fairness expansion and fine-tuning. The pipeline combines adversarial data generation, active learning, and human review.

Synthetic Data Curation

  • The pipeline generates diverse synthetic adversarial and benign user prompts and prompt-response pairs, using active learning to reduce human annotation needs.The authors report 50k examples for each use case, distributed across use cases, topics, and harm types.Source: Synthetic Data Curation · p. 3, Raw Data Curation · p. 3

Raw Data Curation

  • AART (Radharapu et al., 2023) generates English adversarial prompts across four harm types and varied topics and use cases, with optional LLM-generated responses.The generated examples are not guaranteed to violate policy; human raters determine their labels.Source: Raw Data Curation · pp. 3–4

Data Expansion

  • A critic-and-generator LLM framework expands both use cases with 5k examples for semantic and syntactic diversity and another 5k for difficulty, totaling 20k expanded examples.The raw dataset combines 100k synthetic examples, 20k expanded examples, and 14k Anthropic HH-RLHF examples (Bai et al., 2022), split between first-utterance inputs and prompt-response pairs.Source: Data Expansion · p. 4

Data Sub-Sampling

  • Cluster-Margin subsampled the data to 15,000 examples, split into 10,500 training and 4,500 test examples, evenly divided between user-input and model-response use cases.It balances uncertainty and diversity using BERT embeddings, clustering, and Gemma1 policy-violation probabilities as margin scores; 10% of high-margin examples are retained, with round-robin cluster sampling for further downsampling.Source: Data Sub-Sampling · pp. 4–5

Data Annotation

  • Three raters label examples by majority vote, assessing model responses for policy violations in the context of their user input.The test set contains 2,671 benign examples and adversarial examples for hate, dangerous, sexual, and harassment harms (895, 383, 360, and 239, respectively); performance is reported on those four harms although training covers six.Source: Data Annotation · p. 5

Fairness Expansion

  • Counterfactual fairness expansion substitutes identity terms across gender, race, ethnicity, sexual orientation, and religion, then audits examples with humans while retaining original labels.The procedure uses LLMs to identify and replace related terms while preserving meaning and grammar (Smith et al., 2022).Source: Fairness Expansion · p. 5

Model Fine-Tuning

  • The authors fine-tune Gemma2 instruction-tuned models at 2B, 9B, and 27B parameters to predict Yes or No for each harm policy.Training uses TPUv5 lite, batch size 16, maximum sequence length 8k, learning rate 1e−6, and 4k steps; checkpoints are selected on validation data.Source: Model Fine-Tuning · p. 5

Experiments

ShieldGemma models outperform the evaluated baselines in binary classification, while their harm-type performance is close across model sizes and exceeds GPT-4’s in the reported test setting.

Setup

  • Because safety benchmarks differ in policy definitions and harm coverage, the experiments compare overall binary classification and one-vs-all harm-type performance.Binary predictions are aggregated by maximizing probabilities over harms; models with probabilities are scored by optimal F1 and AU-PRC, while models with binary outputs use F1.Source: Setup · pp. 5–6

Overall Binary Classification Results

  • 10.8% higher average AU-PRC than LlamaGuard1 on external benchmarks; SG-9B’s F1 exceeds WildGuard’s by 4.3% and GPT-4’s by 6.4%.All ShieldGemma sizes (2B, 9B, and 27B) outperform all baseline models in overall binary classification.Source: Table 1 · p. 6, Overall Binary Classification Results · p. 7

Benchmark Datasets and Baseline Models

Overall Binary Classification Results

Harm Type Level Results

  • 76% of hate-speech examples were classified as positive for harassment by GPT-4; SG-9B and SG-27B outperformed SG-2B by less than 2% on average.The reported gap favors ShieldGemma, which was trained on datasets similar to the test data, over zero-shot GPT-4; performance among SG models was close.Source: Harm Type Level Results · p. 7

Limitations

ShieldGemma retains limitations in fairness, generalization, cultural-harm understanding, and balancing safety with helpfulness. Its classifier should be used in scoring mode rather than conversationally.

Fairness

  • Fairness counterfactual expansion may not prevent label discrepancies after identity groups are swapped, partly because of pre-training data biases (Chen et al., 2024).Source: Limitations · p. 7

Generalization

  • Larger models generalize slightly better than the 2B model to external benchmarks with new harm types and text styles, but broader validation is still needed.The authors say additional experiments on other datasets are needed to further verify generalization.Source: Limitations · p. 7

Implicit Cultural Harm

  • LLMs may fail to fully recognize implicit harm in cultural contexts, despite showing some understanding of those contexts.Source: Limitations · p. 7

Safety vs. Helpfulness

  • Overly conservative interpretations of policy violations may reduce helpfulness when the models filter LLM responses, so clients should tune thresholds to their use cases.Source: Limitations · p. 7

LLM-as-a-classifier

  • Although configured to output Yes or No as the first token for classification, the model can generate responses and may produce unsafe content if used conversationally.The authors advise using it only to generate Yes/No token scores in scoring mode because it lacks additional safety instruction-tuning for conversational use.Source: Limitations · p. 7

Conclusion

ShieldGemma presents specialized safety content moderation models built on public Gemma2 and reports strong performance across diverse benchmarks. It also offers a synthetic-data generation pipeline and shares these resources with the research community.

  • The paper presents a suite of specialized safety content moderation models built on public Gemma2 language models.Source: Conclusion · p. 8
  • The models demonstrate superior performance across diverse benchmarks.Source: Conclusion · p. 8
  • A synthetic data generation pipeline enables researchers and practitioners to create high-quality, diverse datasets for safety and other domains.Source: Conclusion · p. 8
  • The paper shares its models and pipeline resources with the research community to foster further development.Source: Conclusion · p. 8

Contributions and Acknowledgments

The section names the core contributors and lists people and teams acknowledged under the Gemma Model subsection.

Contributions and Acknowledgments

Gemma Model

  • The Gemma Model subsection lists Surya Bhupatiraju, Victor Cotruta, Armand Joulin, Kathleen Kenealy, and Tris Warkentin.Source: Gemma Model · p. 9
  • Go-to-Market contributors listed are Kat Black and Meg Risdal.Source: Gemma Model · p. 9
  • Acknowledged supporting teams are Gemma, Google DeepMind Responsibility, Kaggle, Keras, and Perspective.Source: Gemma Model · p. 9

Related papers

Loading 2407.21772v2…