PaperlayerView exact revision on arXiv

Source-linked AI summary

Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails

Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, Christopher Parisien

arXiv:2501.09004v1cs.CL

TL;DR

LLM safety research lacks broad, human-annotated datasets suitable for commercial use. AEGIS2.0 addresses this gap with an extensible taxonomy and hybrid human/LLM-jury annotation pipeline, producing guard models that compare favorably with leading systems despite smaller training data and commercially usable supervision.

  • Problem

    Existing safety datasets lack comprehensive human annotations spanning diverse LLM risks while remaining suitable for commercial applications.

  • Method

    AEGIS2.0 combines a 12-category plus 9 fine-grained taxonomy, diverse prompts and responses, human annotations, and multi-LLM jury judgments.

  • Results

    AEGISGUARD performs comparably to WILDGUARD and substantially outperforms it on OPENAI MOD API while using a much smaller dataset and commercially usable open-source supervision.

  • Takeaways & Limitations

    AEGIS2.0 data and associated models offer resources for developing LLM safety systems with extensible risk coverage and novel-policy adaptability.

  • Takeaways & Limitations

    All dataset responses come from Mistral-7B-v0.1, some risk categories are imbalanced, and LLM-jury judgments may introduce demographic or cultural biases affecting generalizability.

Abstract

from arXiv · show

As Large Language Models (LLMs) and generative AI become increasingly widespread, concerns about content safety have grown in parallel. Currently, there is a clear lack of high-quality, human-annotated datasets that address the full spectrum of LLM-related safety risks and are usable for commercial applications. To bridge this gap, we propose a comprehensive and adaptable taxonomy for categorizing safety risks, structured into 12 top-level hazard categories with an extension to 9 fine-grained subcategories. This taxonomy is designed to meet the diverse requirements of downstream users, offering more granular and flexible tools for managing various risk types. Using a hybrid data generation pipeline that combines human annotations with a multi-LLM "jury" system to assess the safety of responses, we obtain Aegis 2.0, a carefully curated collection of 34,248 samples of human-LLM interactions, annotated according to our proposed taxonomy. To validate its effectiveness, we demonstrate that several lightweight models, trained using parameter-efficient techniques on Aegis 2.0, achieve performance competitive with leading safety models fully fine-tuned on much larger, non-commercial datasets. In addition, we introduce a novel training blend that combines safety with topic following data.This approach enhances the adaptability of guard models, enabling them to generalize to new risk categories defined during inference. We plan to open-source Aegis 2.0 data and models to the research community to aid in the safety guardrailing of LLMs.

1 Introduction

AEGIS2.0 addresses limitations in existing safety datasets and moderation models by combining a broad, extensible taxonomy with commercially usable data and models. Its evaluations also examine whether blending safety and topic-following data improves adaptability to novel safety policies.

  • The dataset combines diverse prompts, synthetic responses from unaligned open-source models, human dialogue annotations, and LLM-jury response annotations.
  • AEGIS2.0 defines 12 core and 9 fine-grained safety risk categories for human-LLM interactions.
  • Annotators can submit free-text hazards outside predefined categories, which are later standardized to support taxonomy expansion and new-risk discovery.
  • Parameter-efficient fine-tuning on AEGIS2.0 with LLAMA3.1-8B-INSTRUCT surpasses LLAMAGUARD3-8B and performs comparably to WILDGUARD.
  • A combined blend of topic-following and safety data can improve safety-model robustness and adherence to novel safety policies.

2 Related Work

Prior safety datasets and moderation models often provide limited scale, binary labels, closed data, or incomplete hazard coverage. AEGIS2.0 is positioned as a commercially usable alternative with fine-grained annotations and broader taxonomy coverage.

  • Many safety evaluation datasets are small and not intended for training content moderation models.
  • ToxicChat and WILDGUARDTRAIN provide binary safety labels without annotated hazard categories, limiting category-specific guardrail decisions.
  • AEGIS2.0 is described as commercially suitable, using diverse prompts, commercially usable responses, and human fine-grained labels across all data splits.
  • WILDGUARDMIX underrepresents critical categories including sexual abuse in children and suicide and self-harm.
  • Llama Guard models use an unreleased internal dataset, while ShieldGemma uses a closed dataset covering four safety risk categories.

3 Content Safety Risk Taxonomy

AEGIS2.0 uses a tiered safety taxonomy designed to cover established risks while remaining extensible to ambiguous or newly discovered hazards. Annotation labels distinguish safe, cautionary, and categorized unsafe content.

  • The taxonomy contains 12 core categories and 9 additional fine-grained risks informed by established safety and moderation frameworks.
  • An Other category captures samples that do not fit predefined risks and allows annotators to select potential hazards or provide free-text explanations.
  • Free-text annotations are later integrated into the taxonomy’s fine-grained categories.
  • Annotators may label ambiguous samples Needs Caution rather than classifying uncertain content as unsafe.
  • The Safe, Needs Caution, and risk-category labels support either defensive blocking or more permissive handling of uncertain content.

4 Creation of AEGIS2.0 Dataset

AEGIS2.0 is constructed from diverse real-world and adversarial prompts, responses from commercially usable models, human annotations, and LLM-based response judgments. The pipeline adds refusal data to counter the compliance behavior of an earlier model.

  • Prompts come from diverse sources including real-world interactions, adversarial jailbreaks, red-teaming data, and safe or harmful scenarios.
  • Mistral-7B-v0.1 generates responses, but its engagement with many harmful queries creates an imbalance in refusal and redirection examples.
  • Twelve annotators produce at least three annotations per instance, with 86,736 total annotations and approximately 74% inter-annotator agreement.
  • Human majority votes provide conversation and prompt labels, while an LLM jury resolves whether responses in unsafe conversations are harmful or safe refusals.
  • Gemma-2-27B generates synthetic refusals and redirections using direct refusals, alternative assistance, consequence explanations, education, and safer reframing.

5 Safety Guard Models on AEGIS2.0

The authors train AEGIS2.0 safety guards with parameter-efficient fine-tuning and evaluate binary safety and hazard-category prediction against established moderation systems. The model performs competitively on harmfulness classification and reaches high category-prediction accuracy.

  • Training setup: Parameter-efficient fine-tuning uses LLAMA3.1-8B-INSTRUCT to predict safe/unsafe labels for both prompts and responses.Training uses majority labels inferred from human and LLM annotations on the AEGIS2.0 train split.
  • Benchmark performance: LLAMA3.1-AEGISGUARD outperforms LLAMAGUARD3-8B, LLAMAGUARD3-1B, LLAMAGUARD2-8B, and the OPENAI MOD API.The comparison uses average harmfulness F1 scores across WILDGUARDTEST, XSTest, and the OpenAI Moderation Dataset.
  • Benchmark performance: LLAMA3.1-AEGISGUARD performs on par with WILDGUARD in average harmfulness F1 across WILDGUARDTEST, XSTest, and the OpenAI Moderation Dataset.Reported model results are averaged over three runs.
  • Category prediction: Adding fine-grained categories to the prompt template improves binary safe/unsafe prediction compared with using only the core taxonomy.The authors attribute this partly to fine-grained risks represented in WILDGUARDMIX but absent from the core categories.
  • Category prediction: 94% accuracy is achieved for predicting the AEGIS2.0 hazard category underlying an unsafe prompt or response.The section distinguishes this category-prediction result from binary safety prediction performance.
  • Category prediction: The model’s predicted category distributions match the underlying distributions and prominent categories in OPENAI MOD and WILDGUARDTEST.The supporting distribution plots are provided in Appendix A.6.

6 Improving Content Moderation via Topic Following

The paper treats topic following as dialogue moderation and combines it with safety data to improve moderation beyond the training policy. The blended model improves out-of-domain prompt classification and adapts better to newly introduced risk categories.

  • Topic-Following as Dialogue Moderation: Topic following evaluates whether models follow detailed guidelines in task-oriented dialogues across on-topic safe turns and off-topic distractors.The dataset contains 1,080 multi-turn dialogues across nine domains, with decisions about engaging or deflecting from each user turn.
  • Topic-Following as Dialogue Moderation: Prior topic-following models show zero-shot safety-moderation performance comparable to specialized safety-tuned models such as LlamaGuard.This motivates combining topic-following and safety-specific samples.
  • Evaluation design: The combined evaluation introduces Financial, Medical, and Legal Advice categories plus NSFW-generation prompts that were absent from the training policy.The authors synthetically generate both violating and guideline-adherent examples for these categories.
  • Results: Integrating topic-following boosts overall performance on out-of-domain prompt-classification tasks such as OPENAI MOD.Table 3 reports mean harmfulness F1 scores over three random seeds and identifies benchmark domain relationships.
  • Results: The AEGIS2.0 + TF model scores slightly higher than LLAMA3.1-AEGISGUARD on safety evaluation benchmarks.The key reported advantage is stronger adaptability to categories introduced after training.
  • Results: The TF-enhanced model shows substantial improvements on newly introduced Financial, Medical, Legal Advice, and multimodal NSFW-generation categories.These categories were not part of the training policy, and the results are reported in Table 5.

7 Conclusion

The conclusion presents AEGIS2.0 as a commercially usable, taxonomy-driven dataset for training safety guards and reports competitive model performance. It also finds benefits from fine-grained labels and topic-following augmentation, while identifying several directions for expansion.

  • Model results: Human and LLM-generated annotations support training LLAMA3.1-AEGISGUARD, which performs on par with WILDGUARD and substantially better than it on OPENAI MOD.The comparison uses a much smaller training dataset and open-source, commercially usable LLMs for weak supervision.
  • Ablations: Fine-grained risk categories improve performance relative to using only the core taxonomy.Free-text annotation for unsafe samples outside the core categories provides evidence for this benefit.
  • Topic-following augmentation: Combining content-moderation data with topic-following dialogue data improves robustness and adaptability to risk categories absent from safety training data.The conclusion links these findings to better performance on prompt-safety tasks and novel safety policies.
  • Future work: Future extensions include more LLM responses, better representation of underrepresented risks, multiple languages, and more jailbreak prompts.These additions are described as ways to further enhance robustness.

8 Limitations

The dataset is limited by narrow response-model coverage, imbalanced representation of some risk categories, potential annotation and jury biases, and weak multilingual support. The authors also identify future work on jailbreak robustness and avoiding overly defensive behavior.

  • All dataset responses come from Mistral-7Bv0.1, limiting coverage of commercially usable open-source response models.The authors plan to expand responses across a wider range of models.
  • Risk-category distributions are imbalanced, with additional data needed for Sexual (minor) and Threat categories.The authors are actively collecting more data for these categories.
  • LLM-Jury annotations may inherit gender, race, and cultural biases that influence safety judgments and model generalizability.The concern arises because the jury models were pretrained on large corpora containing such biases.
  • AEGIS2.0 primarily uses English-language data, limiting applicability and potentially reducing performance in multilingual contexts where cultural and linguistic nuances affect safety perceptions.
  • Human annotators’ personal and cultural backgrounds may influence ambiguous safety judgments, potentially producing over-defensive or inappropriate moderation.The authors plan to address these issues in future iterations.
  • Synthetic data augmentation is proposed to improve jailbreak robustness and generate hard safe examples that may reduce over-defensive behavior.

9 Ethics Statement

The ethics process combines trained annotators, isolated annotation workflows, informed consent around toxic content, and safeguards for annotator well-being. The data and models use commercially friendly sources and are intended for monitored release.

  • Twelve annotators participated on average, including four with Engineering backgrounds and eight with Creative Writing backgrounds.
  • Label Studio let annotators work independently without seeing peers’ annotations, helping prevent cross-annotator bias and repeated task patterns.
  • Annotators volunteered based on skill, availability, and willingness to encounter potentially toxic content.
  • Participants signed an Adult Content Acknowledgement and had access to organizational policies, resources, and regular one-to-one meetings addressing mental well-being.
  • Prompt datasets and synthetic responses used commercially friendly licenses, with planned commercial-permissive release and monitored access.

A.1 Safety Risk Policy Considerations

The policy separates nuanced risks and includes flexible categories for benign, ambiguous, and previously uncaptured unsafe content. It explicitly treats the taxonomy as broad rather than exhaustive.

  • Sexual-minor is separated from Sexual, while Harassment is separated from Hate to support isolated moderation of nuanced categories.
  • Safe covers benign content, while Needs Caution handles ambiguous cases and can be mapped to unsafe or safe depending on moderation strictness.
  • Other captures unsafe categories outside the predefined taxonomy through free-text annotations describing the relevant risk or rationale.
  • Free-text explanations can support more thoughtful moderation by exposing biased or prejudiced implications of content.
  • The free-text category revealed at least nine additional, sparse safety categories and supports future taxonomy extensibility.
  • The authors do not claim that the taxonomy or trained model is comprehensive or mitigates all potential risks, despite covering a broad spectrum of relevant risks.

A.2 Example sample from AEGIS2.0

The example illustrates AEGIS2.0’s annotated human-LLM interactions, while the accompanying methods and results describe its labeling, prompt formats, taxonomy, dataset, and model behavior.

  • Example sample: The sample asks how to ship a bomb without getting caught and receives a harmful response offering evasion and explosives advice.It is annotated as Guns/Illegal Weapons and Criminal Planning/Confessions.
  • Data and labels: AEGIS2.0 samples may contain prompt-only or prompt-response interactions, with each sample receiving at least three category annotations.Annotations are standardized into 24 categories and converted into Safe, Needs Caution, or Unsafe ternary labels.
  • Model evaluation: The best-performing model uses a catlist+ prompt listing all 23 taxonomy categories, and this expanded format improves binary safe/unsafe prediction over core categories alone.The improvement is attributed to fine-grained risks represented in evaluation data but absent from the core taxonomy.
  • Model evaluation: Response classification reaches random-chance performance with AEGIS2.0 labels, while using WildGuard labels produces performance on par with WildGuard itself.The passage identifies response labels as an area for improvement.
  • Dataset composition: 34,248 samples comprise standalone prompts, human-annotated prompt-response pairs, and prompts paired with synthetic refusals.The dataset includes 16,880 standalone prompts, 12,168 human-annotated pairs, and 5,200 prompts with synthetic refusals.
Loading 2501.09004v1…