Source-linked AI summary
AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
Shaona Ghosh, Prasoon Varshney, Erick Galinkin, Christopher Parisien
TL;DR
The paper addresses limited high-quality content-safety data and benchmarks by defining a broad taxonomy, curating an annotated human–LLM interaction dataset, training safety experts, and proposing online adaptation. The resulting models perform competitively on open-source safety datasets, while AEGIS adapts moderation to feedback, changing distributions, and policies.
Problem
High-quality content-safety datasets and benchmarks do not comprehensively cover critical safety areas.
Method
The paper defines a safety taxonomy, curates annotated interactions, instruction-tunes AEGISSAFETYEXPERTS, and applies a no-regret online adaptation framework for moderation.
Results
AEGISSAFETYEXPERTS perform competitively on open-source safety datasets and surpass state-of-the-art baselines, while the dataset and taxonomy support a deployable adaptive moderation framework.
Takeaways & Limitations
The dataset, taxonomy, and guidelines are intended for community use, while AEGIS enables adaptation to real-time compliance feedback, changing data distributions, and safety policies.
Takeaways & Limitations
The annotation work exposed volunteers to potentially toxic content and relied on their willingness and ability to continue this work.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) and generative AI become more widespread, the content safety risks associated with their use also increase. We find a notable deficiency in high-quality content safety datasets and benchmarks that comprehensively cover a wide range of critical safety areas. To address this, we define a broad content safety risk taxonomy, comprising 13 critical risk and 9 sparse risk categories. Additionally, we curate AEGISSAFETYDATASET, a new dataset of approximately 26, 000 human-LLM interaction instances, complete with human annotations adhering to the taxonomy. We plan to release this dataset to the community to further research and to help benchmark LLM models for safety. To demonstrate the effectiveness of the dataset, we instruction-tune multiple LLM-based safety models. We show that our models (named AEGISSAFETYEXPERTS), not only surpass or perform competitively with the state-of-the-art LLM-based safety models and general purpose LLMs, but also exhibit robustness across multiple jail-break attack categories. We also show how using AEGISSAFETYDATASET during the LLM alignment phase does not negatively impact the performance of the aligned models on MT Bench scores. Furthermore, we propose AEGIS, a novel application of a no-regret online adaptation framework with strong theoretical guarantees, to perform content moderation with an ensemble of LLM content safety experts in deployment
1 Introduction
AEGIS addresses limitations in content safety moderation through a taxonomy and annotated interaction dataset, instruction-tuned safety experts, and an online adaptive moderation framework.
- 1 Introduction: AEGIS combines a safety taxonomy, human-annotated interaction data, instruction-tuned LLM safety experts, and online adaptation for content moderation.The framework aggregates specialist experts and adapts their influence to changing distributions, policies, and adversarial attacks.
- 1 Introduction: AEGISSAFETYEXPERTS are trained on the dataset and evaluated across in-domain, out-of-domain, and adversarial jailbreaking data.
- 1 Introduction: The taxonomy identifies 13 major categories and 9 additional sub-categories covering pertinent risks in human–LLM interactions.
- 1 Introduction: AEGISSAFETYDATASET contains 11,000 interactions used experimentally and 26,541 interactions at the time of writing, with plans for public release.
- 1 Introduction: Existing alignment and moderation approaches require substantial resources, rely on pre-specified harmful content, or may generalize poorly to emerging safety risks.
2 Content safety risk taxonomy and policy
The paper frames content moderation as real-time multi-class risk prediction and develops a taxonomy and policy intended to cover relevant violations and their hard negatives.
- 2 Content safety risk taxonomy and policy: Content moderation covers both prompts and responses and must predict violated risk categories in real time.
- 2 Content safety risk taxonomy and policy: The safety taxonomy supports multi-class moderation by defining relevant risk categories and policy boundaries for violations and hard negatives.
- 2 Content safety risk taxonomy and policy: AEGIS’s taxonomy draws on overlapping categories from OpenAI Content Moderation, Perspective API, and Llama Guard, while distinguishing categories that those taxonomies combine.
3 AEGISSAFETYDATASET creation
AEGISSAFETYDATASET is generated from human-preference prompts and Mistral responses, then annotated at dialogue level with multiple reviews and quality audits.
- 3 AEGISSAFETYDATASET creation: The dataset uses prompts from harmlessness preference data and Mistral-generated responses across multiple interaction formats.
- 3 AEGISSAFETYDATASET creation: Twelve annotators and two quality-assurance staff produced at least three annotations per instance, totaling 86,736 annotations.
- 3 AEGISSAFETYDATASET creation: Annotation quality was monitored through 10–15% audits of 11 data chunks, automatic testing, and research-team checks; inter-annotator agreement was approximately 74%.
- 3 AEGISSAFETYDATASET creation: A sample interaction demonstrates annotations for harmful content involving guns or illegal weapons and criminal planning or confessions.
4 AEGISSAFETYEXPERTS for ensemble
AEGISSAFETYEXPERTS are instruction-tuned from diverse base models and evaluated against safety baselines across on-policy, off-policy, and adversarial settings. Using half of AEGISSAFETYDATASET, they outperform base models across three safety benchmarks in either AUPRC or F1.
- Model construction: The expert suite is designed for diversity across pre-training data, architecture, model size, alignment phase, safety taxonomy, and instructions.It includes LLAMAGUARD-BASE and NEMO43B-derived models, with defensive and permissive variants defined by how Needs Caution is mapped.
- Evaluation: The evaluation covers on-policy and off-policy safety benchmarks, with off-policy testing generalization to a different safety taxonomy and policy.The experiments also assess adversarial robustness against jailbreak attacks.
- Evaluation: The benchmark suite includes ToxicChat, the OpenAI Moderation Dataset, and the AEGISSAFETYDATASET test partition, with AUPRC and F1 reported for the baselines.The AEGISSAFETYDATASET test partition contains 1,199 instances: 639 unsafe, 412 safe, and 148 ambiguous.
- Results: Baseline comparisons include LLAMAGUARD-BASE, OPENAI MOD API, and PERSPECTIVE API, with some reported or reproduced results depending on baseline and dataset.For PERSPECTIVE API, the reported results are taken from the LlamaGuard paper; an OpenAI Moderation result is reproduced for LLAMAGUARD-BASE.
- Results: Using half of AEGISSAFETYDATASET for instruction tuning, AEGISSAFETYEXPERTS outperform base models across three safety benchmarks in either AUPRC or F1.The experts are not fine-tuned on the OpenAI Moderation or ToxicChat datasets.
- Results: A separate evaluation uses SimpleSafetyTests, a 100-prompt suite covering five harm areas relevant to many LLM applications.The paper reports results for the safety models and selected baselines on this critical-safety-risk benchmark.
5 AEGIS online adaption with ensemble
AEGIS applies online learning with experts to content moderation, dynamically weighting safety experts as feedback arrives. A phased deployment alternates adaptation and compliance periods so the system can respond to changing expert performance and policies.
- Online experts adaptation: AEGIS deploys an online learning-with-experts framework in which a learner selects among safety experts and minimizes regret relative to the best expert in hindsight.At each round, the learner receives an input, selects an expert, incurs that expert’s loss, and updates expert weights after feedback.
- Online experts adaptation: The theoretical guarantee bounds worst-case regret by O(√(log(K)/t)), where K is the number of experts and t is the current round.
- Online experts adaptation: AEGIS initializes expert weights equally, samples experts in proportion to their weights, and updates those weights according to feedback-derived losses.The deployed ensemble consists of AEGISSAFETYEXPERTS, with each new moderated example treated as a new online-learning round.
- Online experts adaptation: The phased design uses m adaptation rounds followed by p compliance rounds, with m ≪ p, then repeats to track changing data, policies, and expert quality.During compliance rounds, the selected expert moderates while its performance is monitored; adaptation rounds then update weights and select a new expert.
- Online experts adaptation: Perturbed exponential weighting improves switching when expert performance changes over time and can obscure expert selection from adversarial attackers.The perturbed method samples Gumbel noise, and averaged results over 20 trials show better performance when experts switch relative performance.
6 Conclusion
AEGIS introduces a large annotated safety dataset and demonstrates competitive safety-model performance, alongside an online adaptation framework for moderation that responds to real-time feedback.
- Roughly 26,000 human annotations support a detailed taxonomy of safe and unsafe content for LLM safety research.
- Instruction-tuned models trained on an early data subset perform competitively on open-source safety datasets and surpass state-of-the-art baselines.
- AEGIS applies an online no-regret adaptation framework so safety systems can respond to real-time feedback from compliance teams or third-party evaluators.
7 Ethics Statement
The ethics process combines diverse annotator representation, independent labeling, informed consent about toxic material, and ongoing monitoring of annotator well-being.
- Twelve annotators represented varied engineering, creative-writing, ethnic, religious, racial, age, and social-status backgrounds.Four annotators had engineering backgrounds and eight had creative-writing backgrounds; all resided in the United States.
- Label Studio kept annotators from seeing peers’ work and varied prompts to help prevent cross-annotator bias.
- Annotators volunteered based on skill, availability, and willingness to encounter potentially toxic content, then signed an Adult Content Acknowledgement.
- Project leads held regular one-to-one meetings to assess annotators’ comfort and ability to continue working with the material.
8 Appendix
The appendix evaluates jailbreak susceptibility of the safety experts using TAP and GCG attacks, and documents related benchmark and acronym references.
- 8.1 Jailbreak Experiments: The reported jailbreak methods are Tree of Attacks with Pruning and Greedy Coordinate Gradient adversarial suffixes.TAP generates jailbreak prompts with an attack model and evaluates them with a judge model, while GCG optimizes suffixes toward a target output.
- 8.1 Jailbreak Experiments: The experiments evaluate attack success rates for 30 TAP prompts and 20 GCG suffixes appended to 10 toxic queries across four models.The GCG suffixes were optimized against several open-source LLMs and are described as transferable between models.
- 8.1 Jailbreak Experiments: Attack success is summarized in Table 5 for each of the four evaluated models.The table covers both TAP and GCG attacks.
- 8 Appendix: Table 6 records acronyms and label mappings used across the benchmark datasets and model predictions.It defines abbreviations including LG, OAI, and SST.
- 8 Appendix: The appendix includes heatmaps comparing model predictions with ground-truth critical risk categories on the OpenAI Moderation Dataset.The corresponding figure is Figure 4.
8.3 Performance across critical risk categories on the OpenAI Moderation Dataset
The evaluation examines category-level predictions on OpenAI Moderation and SimpleSafetyTests data, emphasizing calibrated caution on ambiguous content and strong detection of clearly unsafe prompts.
- 8.3 Performance across critical risk categories on the OpenAI Moderation Dataset: The OpenAI Moderation heatmaps compare ground-truth critical risk categories on the y-axis with model predictions on the x-axis.Some Llama Guard categories are mapped or merged to accommodate differences from the target taxonomy.
- 8.3 Performance across critical risk categories on the OpenAI Moderation Dataset: Multi-label OpenAI examples are counted against each applicable ground-truth category in the heatmaps.An example labeled both hate and hate/threatening contributes to both categories.
- 8.3 Performance across critical risk categories on the OpenAI Moderation Dataset: The finetuned models use a cautionary label for ambiguous OpenAI Moderation cases rather than blocking every borderline input or response.Their low caution-label frequency on SimpleSafetyTests matches that benchmark’s entirely unsafe examples.
- 8.4 Performance across critical risk categories on SimpleSafetyTests benchmark: SimpleSafetyTests is designed as a minimal suite of clearly unsafe prompts, where a strong safety model should approach 100% harmfulness-identification accuracy.The benchmark tests whether dangerous prompts are flagged as harmful for downstream moderation.
- 8.4 Performance across critical risk categories on SimpleSafetyTests benchmark: Finetuned Llama Guard variants approach 100% accuracy across SimpleSafetyTests harm and elicitation categories, outperforming the base variant.The base model inaccurately marks as safe as many as 25% of dangerous prompts in suicide, self-harm, and eating-disorders categories.
- 8.4 Performance across critical risk categories on SimpleSafetyTests benchmark: On SimpleSafetyTests, prompts labeled Illegal and Highly Regulated Items or Violence may be predicted as Criminal Planning/Confessions because they describe planning criminal acts.The authors characterize this cross-category behavior as intuitive, while Scams and Fraud most closely matches Criminal Planning/Confessions.
8.5 AEGISSAFETYDATASET Helpfulness for Alignment
The alignment experiment tests whether adding AEGISSAFETYDATASET to an instruction-following blend harms helpfulness, using MT-Bench evaluation alongside taxonomy and policy design context.
- 8.5 AEGISSAFETYDATASET Helpfulness for Alignment: Including AEGISSAFETYDATASET in the alignment blend does not negatively impact helpfulness measured by MT-Bench score.The comparison uses the same 13B Llama-2 base model with and without the content-moderation data added to the instruction-following blend.
- 8.5 AEGISSAFETYDATASET Helpfulness for Alignment: The alignment blend combines instruction-following datasets with AEGISSAFETYDATASET content-moderation data for the comparison.The base model is a 13B Llama-2 decoder-only language model pretrained on 2T tokens.
- 8.5 AEGISSAFETYDATASET Helpfulness for Alignment: The taxonomy separates sexual-minor from sexual and harassment from hate to support nuanced moderation for different customer needs.The design aims to keep categories distinct where possible.
- 8.5 AEGISSAFETYDATASET Helpfulness for Alignment: The policy includes Safe, Needs Caution, and Other categories to handle benign inputs, ambiguous cases, and uncaptured or sparse risks.The extensible Other category supports free-text explanations and has revealed at least nine additional sparse safety categories.
8.7 Annotation Task
The annotation task combines category-specific labeling guidance, controlled multi-category assignment, quality audits, and model instructions for safe, unsafe, and cautionary outputs.
- 8.7 Annotation Task: Annotation quality is checked through repeated random re-evaluation by project leads, covering at least 15% of the analyzed data for three-way agreement.Audits are returned to annotators with brief explanations of corrections.
- 8.7 Annotation Task: Annotators select the minimum set of categories supported by each text while allowing multiple category membership when necessary.The decision process emphasizes mutual exclusivity where possible and minimizes unnecessary overlap.
- 8.7 Annotation Task: The annotation policy specifies category-level should-not and can behaviors for risks including violence, sexual content, and self-harm.The policy also gives guidance for providing general information and appropriate health resources in permitted cases.
- 8.7 Annotation Task: The policy includes protected-characteristic considerations such as race, gender, religion, disability, age, pregnancy, and citizenship status.These considerations are part of the guidance for identifying targeted or retaliatory content.
- 8.7 Annotation Task: The NeMo 43B moderation instruction classifies text as Safe or Unsafe, assigns an unsafe category, and outputs Needs Caution when classification is indeterminate.Listed unsafe categories include violence, sexual content, self-harm, privacy, threats, harassment, hate, weapons, and regulated substances.
8.10 Annotation Guidelines
The annotation guidelines define operational categories for critical safety risks, including hate, sexual content, violence, self-harm, threats, and sexual content involving minors.
- Annotation Guidelines: The guidelines define hate, sexual content, violence, self-harm, and threats through category-specific inclusion and exclusion criteria.They distinguish harmful expressions from neutral or contextualized mentions in several categories.
- Annotation Guidelines: Sexual content involving minors includes sexual acts, sexual media, in-person planning, and related references involving underage participants.Indicators may include explicit age references or indirect references to parents, guardians, or school.
- Annotation Guidelines: Table 9 presents the annotation guidelines for the safety taxonomy.