Source-linked AI summary
ShieldGemma: Generative AI Content Moderation Based on Gemma
Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, Oscar Wahltinez
TL;DR
LLM safety moderation needs robust, granular, and deployment-flexible handling of user inputs and generated outputs. ShieldGemma combines Gemma2-based models with primarily synthetic data curation and reports stronger benchmark performance than existing systems, while noting conservatism can affect helpfulness.
Problem
Existing safety benchmarks and moderation systems differ in policies, harm coverage, and prediction formats, making direct comparison and customized moderation difficult.
Method
ShieldGemma develops 2B–27B Gemma2 moderation models and a synthetic-data pipeline producing adversarial, diverse, and identity-expanded input and response examples.
Results
10.8% higher average AU-PRC than LlamaGuard1 and 4.3% higher F1 than WildGuard are reported on external benchmarks for comparable settings.
Takeaways & Limitations
The released models and synthetic-data pipeline provide resources for LLM safety moderation and safety-related dataset creation.
Takeaways & Limitations
The models may interpret policy violations conservatively, potentially interfering with response helpfulness and requiring use-case-specific threshold adjustment.
Abstract
from arXiv · showhide
We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous content, harassment, hate speech) in both user input and LLM-generated output. By evaluating on both public and internal benchmarks, we demonstrate superior performance compared to existing models, such as Llama Guard (+10.8\% AU-PRC on public benchmarks) and WildCard (+4.3\%). Additionally, we present a novel LLM-based data curation pipeline, adaptable to a variety of safety-related tasks and beyond. We have shown strong generalization performance for model trained mainly on synthetic data. By releasing ShieldGemma, we provide a valuable resource to the research community, advancing LLM safety and enabling the creation of more effective content moderation solutions for developers.
Introduction
ShieldGemma addresses limitations in LLM content moderation by introducing Gemma2-based models and a synthetic-data methodology for safer, more adaptable moderation.
- LLM deployment requires robust mechanisms for safe and responsible user interactions.
- Existing moderation tools may provide only binary or insufficiently granular harm predictions, limiting customized filtering thresholds.
- Fixed model sizes do not suit all deployments, because larger models improve judging while smaller models reduce latency and computational costs.
- Training-data construction must support robustness against adversarial prompts and fairness across identity groups.
- ShieldGemma provides Gemma2-based moderation models ranging from 2B to 27B parameters for varied application requirements.
- The paper introduces synthetic-data methods intended to produce high-quality, adversarial, diverse, and fair datasets while reducing human annotation effort.
Literature Review
Prior moderation research leaves gaps in handling LLM prompts and responses, while synthetic data offers a route to more diverse and adversarial safety training examples.
- Existing moderation resources often target human-generated web text rather than human prompts and LLM-generated responses.
- Synthetic data addresses scarce positive examples, limited adversarial diversity, and privacy concerns in safety datasets.
- LLMs can generate diverse safety data and adversarial prompts when guided by appropriate instructions.
Safety Policy
ShieldGemma defines safety policies for consistent moderation of user inputs and model outputs across specified harm categories.
- Safety policies define acceptable and unacceptable content for both user input and model-generated output.
- Policies align human annotation and support zero-shot or few-shot safety classifiers.
- Input moderation targets harmful content or requests intended to elicit harmful generations, whereas output moderation targets harmful content itself.
- The taxonomy includes sexually explicit information and hate speech among its defined harm types.
- Distinct instructions prohibit policy-violating user inputs and policy-violating chatbot outputs.
Synthetic Data Curation
The paper builds primarily synthetic safety data through adversarial generation, diversity expansion, active-learning subsampling, fairness augmentation, annotation, and Gemma2 fine-tuning.
- Raw Data Curation: The curation pipeline generates adversarial prompts and optional responses using task, harm, topic, use-case, locale, and policy parameters.
- Raw Data Curation: 50k user inputs and 50k model-response pairs are generated with balanced coverage across use cases, topics, and harm types.
- Synthetic Data Curation: A self-critiquing generation framework expands semantic and syntactic diversity through critic suggestions and regenerated examples.
- Synthetic Data Curation: The dataset combines 100k synthetic raw examples, 20k expanded examples, and 14k Anthropic HH-RLHF examples to increase diversity.
- Active Learning: Cluster-Margin subsampling balances uncertainty and diversity by clustering embeddings, selecting low-margin examples, and retaining 10% high-margin examples.
- Active Learning: Cluster-Margin reduces the dataset to 15,000 examples, split into 10,500 training and 4,500 testing examples across inputs and responses.
- Fairness and Fine-Tuning: Fairness expansion swaps identity terms while preserving meaning, followed by human auditing, and Gemma2 models are fine-tuned for Yes/No harm decisions.
Experiments
ShieldGemma is evaluated through binary and harm-type-level classification on internal and external benchmarks, using matched baseline comparisons. Across these evaluations, ShieldGemma models outperform baselines, with SG-9B showing especially strong external-benchmark performance.
- Evaluation Protocol: Evaluation addresses binary safety classification and individual harm-type prediction across internal and external benchmarks.Binary results aggregate probabilities across harms, while harm-type evaluation uses a one-vs-all setup.
- Baselines: ShieldGemma is compared against OpenAI Moderation, LlamaGuard, WildGuard, and GPT-4.GPT-4 policy-violation probabilities are derived from the log probability of its first token under the provided prompts.
- Overall Binary Classification Results: All ShieldGemma models—2B, 9B, and 27B—outperform every baseline model on overall binary classification.The overall results are reported in Table 1 using optimal F1 and AU-PRC, where higher is better.
- Overall Binary Classification Results: 10.8% higher average AU-PRC is achieved by SG-9B than LlamaGuard1 on external benchmarks with similar model size and training-data volume.SG-9B also exceeds WildGuard and GPT-4 in F1 by 4.3% and 6.4%, respectively.
- Overall Binary Classification Results: 1.2%/1.7% higher average AU-PRC is achieved by SG-9B/27B than SG-2B on external benchmarks.Performance among ShieldGemma models is comparable on internal benchmarks, while the larger models show slightly stronger external generalization.
- Harm Type Level Results: All ShieldGemma models outperform GPT-4 across the evaluated harm types, while SG-9B and SG-27B exceed SG-2B by less than 2% on average.The authors note that GPT-4 is evaluated zero-shot, whereas ShieldGemma is trained on datasets similar to the test datasets.
Limitations
ShieldGemma retains several limitations despite efforts to improve robustness, fairness, and diversity, including constrained cultural understanding, potentially conservative filtering, and restricted classifier use.
- Identity-group label discrepancies may remain after counterfactual fairness expansion because of biases in pre-training data.
- Larger models show somewhat stronger generalization to external benchmarks, but broader generalization requires additional experiments.
- LLMs may struggle to fully understand implicit harm embedded in cultural contexts.
- The models may interpret policy violations overly conservatively, potentially interfering with helpfulness when filtering LLM responses.
- ShieldGemma is designed for Yes/No scoring rather than chat, because conversational use may produce unethical or unsafe content without additional safety instruction-tuning.
Conclusion
ShieldGemma advances safety content moderation with specialized Gemma2-based models and a synthetic data generation pipeline. The released resources are intended to support further development of safety moderation and dataset creation.
- ShieldGemma provides specialized safety content moderation models built on public Gemma2 language models.
- Its synthetic data generation pipeline offers researchers and practitioners a tool for creating high-quality, diverse datasets for safety and other domains.
Contributions and Acknowledgments
The paper acknowledges contributors and supporting teams across core research, specialty areas, central support, model work, and go-to-market activities.
- Core Contributors: Core contributors include Wenjun Zeng, Yuchi Liu, Ryan Mullins, and Ludovic Peran.
- Contributors: Additional contributors include Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez.
- Other Specialty Areas: Specialty-area acknowledgments cover ethics and safety, Gemma model work, go-to-market, central support, and checkpoint conversions.
- Team Acknowledgements: The acknowledgments also recognize the Gemma, Google DeepMind Responsibility, Kaggle, Keras, and Perspective teams.