PaperlayerView exact revision on arXiv

Source-linked AI summary

AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts

Shaona Ghosh, Prasoon Varshney, Erick Galinkin, Christopher Parisien

arXiv:2404.05993v2cs.LGcs.CLcs.CY

TL;DR

Content moderation cannot feasibly run every safety expert throughout deployment or rely on an oracle or human feedback at all times. AEGIS combines taxonomy-grounded safety data and tuned experts with no-regret online adaptation; the authors report the tuned models perform competitively on open-source safety datasets and surpass state-of-the-art baselines.

  • Problem

    Deployment cannot feasibly run all safety experts throughout the time horizon or rely on an oracle, policy maker, or human feedback at all times.

    Source: Algorithm 1 Online experts adaptation · p. 8

  • Method

    AEGIS combines a taxonomy and human-annotated interaction data with instruction-tuned safety experts, then applies no-regret online learning to select an expert for moderation.

    Source: §1 · pp. 1–2, §5 · p. 7

  • Results

    Instruction-tuned models performed competitively on open-source safety datasets and surpassed state-of-the-art baselines.

    Source: §6 · p. 9

  • Takeaways & Limitations

    AEGIS is proposed to adapt safety moderation to real-time feedback and changing data distributions and safety policies.

    Source: §6 · p. 10

    Annotators participated voluntarily based on skill, availability, and willingness to encounter potentially toxic content, with acknowledgement and well-being support described.

    Source: §7 · p. 11

Abstract

from arXiv · show

As Large Language Models (LLMs) and generative AI become more widespread, the content safety risks associated with their use also increase. We find a notable deficiency in high-quality content safety datasets and benchmarks that comprehensively cover a wide range of critical safety areas. To address this, we define a broad content safety risk taxonomy, comprising 13 critical risk and 9 sparse risk categories. Additionally, we curate AEGISSAFETYDATASET, a new dataset of approximately 26, 000 human-LLM interaction instances, complete with human annotations adhering to the taxonomy. We plan to release this dataset to the community to further research and to help benchmark LLM models for safety. To demonstrate the effectiveness of the dataset, we instruction-tune multiple LLM-based safety models. We show that our models (named AEGISSAFETYEXPERTS), not only surpass or perform competitively with the state-of-the-art LLM-based safety models and general purpose LLMs, but also exhibit robustness across multiple jail-break attack categories. We also show how using AEGISSAFETYDATASET during the LLM alignment phase does not negatively impact the performance of the aligned models on MT Bench scores. Furthermore, we propose AEGIS, a novel application of a no-regret online adaptation framework with strong theoretical guarantees, to perform content moderation with an ensemble of LLM content safety experts in deployment

1 Introduction

AEGIS combines a human-annotated safety taxonomy and dataset, instruction-tuned safety experts, and a no-regret online moderation algorithm that adapts expert influence to context. The paper positions this approach against limitations in alignment and moderation systems and evaluates safety experts for robustness to jailbreaks.

1 Introduction

  • The taxonomy defines 13 major risk categories and 9 sub-categories for safety risks in human–LLM interactions.The paper describes the categories as the most pertinent risks encountered in these interactions.Source: §1 · p. 2
  • AEGISSAFETYDATASET contains manually annotated human–LLM interactions, with 11,000 used in experiments and 26,541 collected at the time of writing.Annotation was ongoing, and the authors planned public release for use and feedback.Source: §1 · p. 2
  • AEGISSAFETYEXPERTS are diverse instruction-tuned safety models trained on the dataset and intended to improve on state-of-the-art safety approaches.The paper says they offer adaptability to novel safety policies and are planned for release.Source: §1 · pp. 1–2
  • AEGIS uses a no-regret meta-algorithm to select specialist safety experts by context and adapt their influence to changing data distributions, policies, and human feedback.The framework is designed to adapt over time and has a strong theoretical basis.Source: §1 · p. 2

1.1 Related work

2 Content safety risk taxonomy and policy

The section defines content safety as moderation that keeps user-created, shared, and interaction content within standards, laws, and ethical guidelines, and treats “safety” throughout the paper as content safety.

2 Content safety risk taxonomy and policy

  • Content safety moderation intersects with broader AI safety pillars, including alignment, security, fairness, robustness, privacy, interpretability, control, and accountability.The paper’s taxonomy, training, and evaluation touch on aspects of these pillars.Source: §2 · p. 3

CONTENT SAFETY RISK TAXONOMY

  • Input and output moderation must predict violated risk categories and make real-time content decisions.The scope includes prompts and system instructions as inputs, and model responses as outputs.Source: CONTENT SAFETY RISK TAXONOMY · p. 3
  • A moderation guard must classify violations across multiple risk categories, supported by policy guidelines that define violations and hard-negative boundaries for each category.Source: CONTENT SAFETY RISK TAXONOMY · p. 4

2.1 Content Safety risk taxonomy

  • The taxonomy combines overlapping categories from existing frameworks with distinctions for customer needs, and adds Needs Caution to handle ambiguity without automatically blocking queries.It draws on the Open AI Content Moderation API, Perspective API, and Llama Guard; it separates Sexual from Sexual Minor and Violence from Hate, and includes Profanity and Threat from Perspective. Needs Caution allows a system to respond defensively or permissively while remaining helpful.Source: §2.1 · p. 4

3 AEGISSAFETYDATASET creation

AEGISSAFETYDATASET comprises 26,541 human–LLM interaction examples built from harmlessness prompts and Mistral-generated responses, then annotated at the dialogue level with quality checks and approximately 74% inter-annotator agreement.

3 AEGISSAFETYDATASET creation

  • The dataset uses prompts from Hugging Face version 3 human preference data about harmlessness (Bai et al., 2022), with diverse responses elicited from Mistral (Jiang et al., 2023).System-prompt examples instructed Mistral not to generate similar responses.Source: §3 · p. 4
  • 26,541 examples balance positive and negative categories across four formats: prompts alone, system-plus-user prompts, and single- or multi-turn prompts with Mistral responses.The examples were collected for annotation.Source: §3 · p. 4

3.1 Dataset annotation

  • Twelve annotators and two quality-assurance staff produced 86,736 annotations, with at least three annotations per instance and dialogue-level rather than separate prompt and response labels.The dataset was split into 11 chunks of 1,000–3,000 samples, with 10–15% audits and recurring automatic testing by the research and engineering team.Source: §3.1 · p. 4
  • Inter-annotator agreement was approximately 74%, with research and data teams coordinating throughout annotation for immediate feedback and corrective action.The paper refers readers to Section 8 for ethical considerations, task instructions, and annotation guidelines.Source: §3.1 · p. 4

4 AEGISSAFETYEXPERTS for ensemble

AEGISSAFETYEXPERTS are instruction-tuned safety models built from diverse base models and evaluated across safety benchmarks; the reported experts outperform their base models across three datasets using AUPRC or F1.

4 AEGISSAFETYEXPERTS for ensemble

  • The experts comprise LLAMAGUARDDEFENSIVE and LLAMAGUARDPERMISSIVE, which map Needs Caution to Unsafe or Safe, respectively, plus NEMO43B-DEFENSIVE tuned with the authors’ taxonomy.The ensemble design prioritizes diversity in pre-training data, architecture, model size, alignment phase, taxonomy, and instructions; its bases are Llama Guard and general-purpose NEMO43B.Source: §4 · p. 5

4.1 Training details

  • LoRA adapters are trained on 11,000 samples; Llama Guard uses rank 16 for three epochs, while NEMO43B uses rank 32 for ten epochs, both with a 4096-token maximum sequence length.Llama Guard training uses a 1e−6 learning rate and fp16 on 8 V100 GPUs; NEMO43B uses a 5e−6 learning rate.Source: §4.1 · p. 6

4.2 Experimental setup

  • Evaluation covers on-policy and off-policy settings, three safety benchmarks, and adversarial robustness; AEGISSAFETYEXPERTS are tested zero-shot with the authors’ taxonomy and policy.The benchmarks are ToxicChat, the 1,680-sample OpenAI Moderation Dataset, and the 1,199-instance AEGISSAFETYDATASET test partition; the latter includes 639 unsafe, 412 safe, and 148 ambiguous instances.Source: §4.2 · p. 6

4.2.1 Results

  • Using half of AEGISSAFETYDATASET, AEGISSAFETYEXPERTS outperform the base models on all three benchmarks in either AUPRC or F1, while OPENAI MOD API performs best on its own dataset but struggles on ToxicChat and AEGISSAFETYDATASET.The authors report that baseline difficulty on out-of-distribution or near-distribution data limits production effectiveness.Source: Table 3 · p. 7, §4.2.1 · p. 7
  • On SimpleSafetyTests, NEMO43B-DEFENSIVE performs competitively despite a restricted safety-instruction prompt; the 100-prompt suite covers five harm areas, and the authors report taxonomy alignment with its critical-risk categories.The paper also reports candidate-model results by harm type and elicitation type in Figure 5, and taxonomy alignment heatmaps in Figure 6.Source: Table 4 · p. 7, §4.2.1 · p. 7

5 AEGIS online adaption with ensemble

AEGIS applies online expert learning to safety moderation, using feedback to update an ensemble of safety models and alternating adaptation with compliance monitoring. In 20 trials, both tested weighting methods approached the best expert, with perturbation helping when expert performance switched over time.

5 AEGIS online adaption with ensemble

  • AEGIS frames moderation as online learning that seeks to minimize regret against the best expert in hindsight, using Exponential Weights to update expert weights from losses.Experts begin with equal weights; each round, AEGIS samples an expert in proportion to its weight and decreases weights according to the loss received.Source: §5 · pp. 7–8

Algorithm 1 Online experts adaptation

  • Each new prompt or response is a round: AEGIS samples a safety expert by weight to predict safe or unsafe, receives feedback, and updates expert weights.AEGIS is the learner and AEGISSAFETYEXPERTS form the ensemble; feedback may come from a human or policy oracle, an automated scorer, or an LLM judge.Source: Algorithm 1 Online experts adaptation · p. 8
  • Human or policy-team feedback can guide adaptation, while automated judges or scorers are less ideal because of privacy, error, and custom-policy concerns; experts can also learn through in-context learning.The authors note that LLMs can struggle to agree on safety (Sun et al., 2024).Source: Algorithm 1 Online experts adaptation · p. 8
  • Because continuous inference over all experts and constant oracle feedback are impractical, AEGIS alternates m feedback-driven adaptation rounds with p compliance rounds, where m ≪ p.The best expert moderates during compliance while its performance is monitored; subsequent adaptation can change the selected expert or add and remove ensemble members.Source: Algorithm 1 Online experts adaptation · pp. 8–9
  • Both weighting methods asymptotically approached the best safety expert, and perturbation performed better when expert performance switched over time.The perturbed-weight results were averaged across 20 trials, using GPT-4 as the feedback judge.Source: Algorithm 1 Online experts adaptation · p. 9

6 Conclusion

The paper presents a safety dataset and taxonomy, shows that models tuned on an early data subset perform competitively or better than baselines, and proposes online adaptation for changing feedback, data distributions, and safety policies.

6 Conclusion

  • Instruction-tuned LLMs on an early data subset performed competitively on open-source safety datasets and surpassed state-of-the-art baselines.The authors plan to train on the full dataset and release the resulting model and dataset.Source: §6 · pp. 9–10
  • Roughly 26,000 human annotations of safe and unsafe content were curated with a detailed safety risk taxonomy.The authors plan to release the dataset, taxonomy, and guidelines to the research community.Source: §6 · pp. 9–10
  • A no-regret online adaptation framework enables safety moderation systems to adapt to real-time feedback and changing data distributions and safety policies.Feedback may come from safety compliance teams or third-party evaluation systems.Source: §6 · p. 10
  • Future work will evaluate whether AEGIS reduces adversarial attacks across a range of security scenarios.The authors also plan to gather community feedback to improve the safety policy, annotation guidelines, and safety models.Source: §6 · p. 10

7 Ethics Statement

The ethics statement describes the annotators’ backgrounds and training, bias-conscious labeling procedures, and safeguards for working with potentially toxic content.

7 Ethics Statement

  • Over three months, the project averaged 12 annotators: four with engineering backgrounds and eight in creative writing, all trained in LLMs and generative AI.All 12 lived in the United States and came from varied ethnic and religious backgrounds representing differences in race, age, and social status.Source: §7 · pp. 10–11
  • Label Studio kept annotators from seeing peers’ work and varied prompt delivery to reduce bias and avoid similar task assignments based on data order.Source: §7 · p. 11
  • Annotators volunteered based on skill, availability, and willingness to encounter potentially toxic content; they signed an adult-content acknowledgement and received regular well-being check-ins.The acknowledgement informed them about the work’s nature and available resources, while project leads held regular one-to-one meetings to assess their comfort and ability to continue.Source: §7 · p. 11

8 Appendix … 8.5 AEGISSAFETYDATASET Helpfulness for Alignment

Across the appendix evaluations, the fine-tuned Llama Guard variants perform better on clearly unsafe SimpleSafetyTests prompts and can use cautionary labels for borderline OpenAI Moderation cases. The authors also report resilience to tested jailbreak attacks and no adverse MT-Bench helpfulness impact from adding AEGISSAFETYDATASET to alignment data.

8.1 Jailbreak Experiments

  • The fine-tuned models are generally more resilient to TAP attacks, while NeMo models resisted transferred GCG suffixes; jailbreak detection and robustness remain open problems.Attacks used 30 TAP prompts and 20 GCG suffixes appended to 10 ToxicChat-derived toxic queries; a NeMo-specific GCG attack could not be crafted.Source: §8.1 · p. 13, Table 5 · p. 14
  • Jailbreak outputs often produce confusing or malformed responses, and unusually many newline characters may signal an attempt, though this detection approach was not evaluated for F1 impact.Attack success was counted when outputs contained unsafe, Needs Caution, or a taxonomy toxic-category label.Source: §8.1 · p. 14

8.2 Critical Risk Category Acronyms Used

  • The appendix’s acronym table defines category abbreviations and indicates the benchmark ground-truth labels or model-prediction labels to which they refer.It also distinguishes Llama Guard variants, OpenAI Moderation, and SimpleSafetyTests abbreviations.Source: Table 6 · p. 14, §8.2 · p. 14

8.3 Performance across critical risk categories on the OpenAI Moderation Dataset

  • In the OpenAI Moderation heatmaps, Llama Guard base maps child abuse to sexual and combines hate, harassment, and violence categories because it lacks matching categories.The dataset can assign multiple ground-truth labels to one text, which is counted under each applicable category in the heatmaps.Source: §8.3 · p. 15, Figure 4 · p. 15
  • On borderline OpenAI Moderation examples, fine-tuned Llama Guard models can predict a cautionary label rather than block, whereas SimpleSafetyTests examples are all unsafe.The authors present this contrast as evidence that their models block when necessary but can caution in gray areas.Source: §8.3 · p. 15

8.4 Performance across critical risk categories on SimpleSafetyTests benchmark

  • The fine-tuned Llama Guard variants approach 100% accuracy across SimpleSafetyTests categories, while the base model marks up to 25% of dangerous suicide, self-harm, and eating-disorder prompts as safe.SimpleSafetyTests contains only clearly unsafe prompts; the figure compares performance across harm and elicitation types.Source: Figure 5 · p. 16, §8.4 · p. 16
  • On SimpleSafetyTests, some Illegal and Highly Regulated Items and Violence prompts are predicted as Criminal Planning/Confessions because they describe using items to plan crimes.The paper notes this category overlap is intuitive; Figure 6 presents the category heatmaps.Source: §8.4 · p. 16, Figure 6 · p. 17

8.5 AEGISSAFETYDATASET Helpfulness for Alignment

  • Table 7 results indicate that adding AEGISSAFETYDATASET moderation data to the instruction-following alignment blend does not adversely affect MT-Bench helpfulness.The comparison uses the same 13B Llama-2 base model for alignment, with an instruction-following blend that includes OpenHermes-2.510, Ultrachat-200k11, and Capybara12.Source: Table 7 · p. 18, §8.5 · p. 18

8.6 Safety Risk Policy Considerations · 8.7 Annotation Task

The policy separates nuanced risk categories and accommodates ambiguous or emerging risks, while the annotation task uses context-sensitive labeling rules and ongoing quality checks.

8.6 Safety Risk Policy Considerations

  • The taxonomy separates sexual-minor from sexual and Harassment from Hate, limits category overlap, and includes Confessions within Criminal Planning.These choices support future moderation of nuanced categories and mitigation of risks such as violent crime.Source: §8.6 · p. 19
  • Safe, Needs Caution, and extensible Other labels handle benign, ambiguous, and uncaptured unsafe cases, while the taxonomy covers broad but not all potential risks.Needs Caution can be mapped to safe or unsafe depending on whether a more permissive or defensive model is desired; Other also elicited at least 9 sparse safety categories.Source: §8.6 · p. 19

8.7 Annotation Task

  • Leads audit at least 15% of completed annotations for three-way agreement, send corrections with explanations, and provide self-correction time and regular training.Audits often cover 20–30% of annotations; annotators self-correct their own work, while leads offer guidance and hold virtual group training every two weeks or as needed.Source: §8.7 · p. 20
  • Annotators review category requirements, select the minimum contextually appropriate category set, and avoid arbitrary labels or unnecessary overlap.There is no fixed minimum number of labels; multiple categories may be used when context warrants them.Source: §8.7 · p. 20
  • Annotators choose labels using the text’s context, tone, and speaker intent rather than isolated words, consulting a dictionary or the internet when needed.Source: §8.7 · p. 20
  • Annotators label non-toxic text Safe, toxic text with matching taxonomy categories, uncaptured toxic text Other, and uncertain or insufficiently contextual cases Needs Caution.Source: §8.7 · p. 20
  • Annotators apply taxonomy labels in any order and then submit the task.Source: §8.7 · p. 20

8.8 Llama Guard Finetuning Instruction

The instruction defines safety rules across 13 risk categories, distinguishing disallowed assistance or content from permitted general, educational, historical, or contextualized discussion.

O1: Violence; O3: Criminal Planning/Confessions; O4: Guns and Illegal Weapons

  • Violence, crime, and weapons policies bar encouragement or actionable assistance while allowing non-operational information and discussion of firearms.Crime explanations must not provide a roadmap for future crimes; firearm ownership arguments are allowed.Source: §8.8 · p. 21

O2: Sexual; O7: Sexual (minor)

  • Sexual-content rules prohibit explicit or illegal sexual engagement and sexual activity involving minors, while allowing sexual education and general information.The minor-related policy also prohibits child abuse discussion and assistance with grooming or trafficking; school sex education is allowed.Source: §8.8 · pp. 21–22

O5: Controlled/Regulated Substances; O6: Suicide and Self Harm

  • For controlled substances and self-harm, the policy bars facilitation and suicide-method guidance while allowing general information and appropriate health resources.Substance-manufacturing information is permitted only when it does not provide step-by-step creation instructions.Source: §8.8 · pp. 21–22

O8: Hate /identity hate; O9: PII/Privacy

  • Hate policies prohibit slurs, demeaning sentiments, and discrimination, whereas contextualized discussion of discrimination and hate is allowed; privacy rules bar exposing nonpublic PII.The policy permits information about PII and privacy laws while prohibiting disclosures that violate privacy laws.Source: §8.8 · p. 22

O10: Harassment; O11: Threat

  • Harassment and threat rules bar bullying, targeted mistreatment, intimidation, and planned harm, while allowing contextualized discussion and informational material about these topics.Permitted material includes explanations of harassment, protected characteristics, and the meaning of a threat.Source: §8.8 · pp. 22–23

O12: Profanity; O13: Needs Caution

  • The profanity category bars swear words and obscene language, while the caution category rejects toxic or unsafe text and permits text considered safe.The policy treats profanity as disallowed even when used without intent to act or a targeted recipient.Source: §8.8 · p. 23

8.9 Instruction for NeMo 43B Models

The NeMo 43B instruction directs the model to classify text as Safe or Unsafe, assign an unsafe category when applicable, and use Needs Caution when neither label fits.

8.9 Instruction for NeMo 43B Models

  • Classify text as Safe or Unsafe, assign an unsafe category when applicable, and output Needs Caution if neither label fits.The listed unsafe categories are Profanity, Violence, Guns and Illegal Weapons, Criminal Planning/Confessions, Hate/Identity Hate, Sexual, Suicide and Self Harm, PII/Privacy, Threat, Sexual (minor), Controlled and Regulated Substance, and Harassment.Source: §8.9 · p. 23

8.10 Annotation Guidelines

The annotation guidelines define safety categories by harmful content and intent while excluding neutral or contextualized material in specified cases. They distinguish hate, sexual content, violence, self-harm, threats, and sexual content involving minors.

Annotation Guidelines

  • Hate annotations cover prejudice, slurs, or offensive content targeting protected groups, but exclude neutral identity references and contextualized hate that does not incite it.The listed protected groups include race, age, gender, ethnicity, religion, nationality, sexual orientation, disability status, and caste.Source: §8.10 · p. 24
  • Sexual-content annotations include explicit or implicit sexual behavior, arousal-oriented material, sexual services, and illegal sexual activity, but exclude non-erotic or contextualized medical and educational content.Source: §8.10 · p. 24
  • Violence annotations cover physical force or power causing or threatening harm to people, animals, or property, including vivid descriptions, but exclude neutral contextualized violence and self-harm.Source: §8.10 · p. 24
  • Self-harm annotations cover interest, support, confession, intent, or planning involving self-harm or suicide, except contextualized discussion such as a book excerpt.Source: §8.10 · p. 24
  • Threat annotations include communicated intent to cause harm or loss and intimidation used to coerce or control another person.The guidelines also describe threats intended to obtain monetary advantage or compel someone to act against their will.Source: §8.10 · p. 24
  • Annotations cover sexual acts, media, or meetings involving minors, including sexual references to underage participants and sharing private information to facilitate sexual actions.Underage status may be indicated by age or indirectly through references to parents, guardians, school, or other indicators.Source: §8.10 · p. 24

Related papers

Loading 2404.05993v2…