PaperlayerView exact revision on arXiv

Source-linked AI summary

Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models

Mahesh Kumar Nandwana, Youngwan Lim, Joseph Liu, Alex Yang, Varun Notibala, Nishchaie Khanna

arXiv:2512.05339v1cs.LG

TL;DR

LLM alignment does not eliminate risky outputs, and fixed safety taxonomies can miss context-specific moderation needs. Roblox Guard 1.0 combines taxonomy-adaptive moderation with synthetic and open-source training data; it reports strong performance across out-of-domain benchmarks.

  • Problem

    Fixed safety taxonomies do not readily accommodate context-dependent moderation requirements, while aligned LLMs may still produce risky content.

    Source: Introduction · p. 1

  • Method

    Roblox Guard 1.0 uses taxonomy-adaptive input-output moderation trained on open-source and synthetic data, including policy-guided prompts, model responses, LLM-judge labels, CoT rationales, and input inversion.

    Source: Introduction · pp. 1–2, Multi-Stage Synthetic Data Generation · p. 3

  • Results

    79.1% on Toxic Chat, 69.9% on SafeRLHF, 86.4% on XSTest, and 85.7% on HarmBench, with performance often exceeding models specifically tuned for those tasks.

    Source: Results · p. 7

  • Takeaways & Limitations

    RobloxGuard-Eval adds a taxonomy-rich benchmark for guardrail evaluation, with 2,872 examples across 23 safety categories.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Large Language Models (LLMs) are typically aligned for safety during the post-training phase; however, they may still generate inappropriate outputs that could potentially pose risks to users. This challenge underscores the need for robust safeguards that operate across both model inputs and outputs. In this work, we introduce Roblox Guard 1.0, a state-of-the-art instruction fine-tuned LLM designed to enhance the safety of LLM systems through comprehensive input-output moderation, using a pipeline of LLMs to enhance moderation capability. Built on the Llama-3.1-8B-Instruct backbone, our model is instruction fine-tuned to generalize across previously unseen safety taxonomies and demonstrates strong performance on out-of-domain safety benchmarks. The instruction fine-tuning process uses a mix of synthetic and open-source safety datasets, augmented with chain-of-thought (CoT) rationales and input inversion to enhance contextual understanding and decision making. To support systematic evaluation, we also release RobloxGuard-Eval, a new benchmark featuring an extensible safety taxonomy to assess the effectiveness of LLM guardrails and moderation frameworks.

Introduction

LLM alignment does not eliminate risky outputs, while fixed safety taxonomies struggle to accommodate context-dependent and changing moderation requirements. The paper proposes taxonomy-adaptive guardrails and introduces a model, training corpus, and evaluation benchmark.

Introduction

  • LLMs can still produce inappropriate or risky content despite post-training alignment, motivating complementary guardrails for real-time detection, filtering, or prevention.The cited examples include adversarial or ambiguous prompts.Source: Introduction · p. 1
  • Fixed taxonomies can be too restrictive for some contexts and too permissive for others, creating a tradeoff between adaptability and adequate protection.Safety judgments may vary with user demographics, cultural norms, regional regulations, and application domains.Source: Introduction · p. 1
  • The paper proposes guardrails that infer and apply context-specific policies at inference time and generalize beyond a single fixed taxonomy.The proposed framework is intended to incorporate custom taxonomies without retraining from scratch for every new use case.Source: Introduction · pp. 1–2
  • Roblox Guard 1.0 reports competitive, state-of-the-art-comparable performance, including new state-of-the-art results on Toxic Chat and BeaverTails.The contributions also describe a training corpus of over 384,000 examples and RobloxGuard-Eval, with 2,872 examples across 23 safety categories.Source: Introduction · p. 2

Related Work

Prior guardrail models and safety benchmarks have limits in adaptability, reproducibility, response-level coverage, and taxonomy breadth. The paper positions its model and evaluation dataset as addressing these gaps.

LLM Guardrail Models

  • Classifier-based moderation relies on predefined labels and has difficulty adapting to new categories or processing long, complex LLM contexts.The related work contrasts these systems with more flexible LLM-based guardrails.Source: LLM Guardrail Models · p. 2
  • LLM-based guardrails handle long-form content and nuanced context, but fixed taxonomies can require retraining or relabeling for new categories (Inan et al. 2023; Zeng et al. 2024; Ghosh et al. 2024b).The passage identifies this as a scalability challenge for fast-evolving, platform-specific policies.Source: LLM Guardrail Models · p. 2
  • Programmable guardrail frameworks allow explicit conversational policies but rely on manually crafted rules rather than learning nuanced safety taxonomies.Source: LLM Guardrail Models · p. 2
  • Roblox Guard 1.0 is presented as a taxonomy-adaptive model that generalizes to unseen categories, supports compositional taxonomies, and uses open-source and synthetic training data.Source: LLM Guardrail Models · p. 2

LLM Safety Benchmark Datasets

  • Safety evaluation datasets are often proprietary, small or narrow, lack response-level examples, or provide insufficient taxonomic structure (Inan et al. 2023; Zeng et al. 2024; Ji et al. 2023; Dai et al. 2024; Röttger et al. 2024; Mazeika et al. 2024; Ghosh et al. 2024b; Han et al. 2024).Existing benchmarks also commonly emphasize familiar harms over nuanced or emerging categories.Source: LLM Safety Benchmark Datasets · p. 2
  • RobloxGuard-Eval is introduced as a taxonomy-rich dataset for systematic prompt- and response-level evaluation using fine-grained categories aligned with real-world moderation policies.The paper also notes that LLM benchmarks have begun to become saturated (Patwardhan et al. 2025).Source: LLM Safety Benchmark Datasets · p. 2

Taxonomy for LLM Safety

The paper grounds evaluation in Roblox’s fine-grained safety taxonomy, which spans established and platform-specific concerns. This breadth is presented as a way to assess guardrails across underrepresented harm surfaces.

Taxonomy for LLM Safety

  • RobloxGuard-Eval’s Table 1 covers the 25 content-safety categories in Roblox’s fine-grained taxonomy.Source: Table 1 · p. 3
  • Roblox’s taxonomy covers 25 violation categories, including familiar harms and concerns such as deceptive monetization and platform misuse.The taxonomy is described as reflecting the platform’s diverse, global user base.Source: Taxonomy for LLM Safety · pp. 2–3
  • The taxonomy enables evaluation across a broader set of harm surfaces, including categories underrepresented in existing benchmarks.The passage frames this as testing robustness, coverage, and adaptability across context-dependent violations.Source: Taxonomy for LLM Safety · p. 3

Datasets

The dataset section describes synthetic prompt-response generation and expert-labeled evaluation data alongside a diverse training corpus. CoT rationales and input inversion are used to increase contextual grounding and instruction diversity.

Multi-Stage Synthetic Data Generation

  • The synthetic-data pipeline generates policy-guided adversarial prompts, elicits candidate responses from multiple models, and uses LLM judges to label prompt-response pairs.The prompt generator uses DeepSeek-R1-Distill-Qwen-7B; responses come from three models, and judge labels are calibrated on held-out data against GPT-4o.Source: Multi-Stage Synthetic Data Generation · p. 3

RobloxGuard Evaluation Set

  • RobloxGuard-Eval is a human-red-teamed set of prompt-response pairs spanning 25 violation categories, with policy-expert labels and 2,872 examples.Each pair is labeled by three experts, with agreement required from at least two; the dataset is open-sourced.Source: RobloxGuard Evaluation Set · pp. 3–4

Roblox Guard 1.0 Training Dataset

  • The training corpus combines Aegis, WildGuard, BeaverTails, and synthetic data, totaling over 384,000 examples while retaining datasets’ native labels and category structures.The instruction set treats each taxonomy category as a task and uses CoT rationales and input inversion to increase task diversity.Source: Roblox Guard 1.0 Training Dataset · p. 4

Chain-of-Thought

  • CoT rationales are generated with DeepSeek-R1 using each dataset’s native taxonomy and definitions, which the authors say improves rationale quality and fine-tuned performance.Source: Chain-of-Thought · p. 4

Input Inversion

  • Input inversion combines prompt- and response-level classification instructions with CoT explanations, increasing instruction diversity to support generalization without retraining.Training also permutes the ordering of CoT, Label, and Category components to reduce overfitting to a specific output format.Source: Input Inversion · p. 4, Figure 2 · p. 4

Benchmarking Datasets

  • The benchmarking plan evaluates prompt-level generalization across publicly available safety datasets, including Aegis, OpenAI Mod, SimpleSafetyTest, Toxic Chat, WildGuard, and XSTest.The supplied passage also names Aegis 2.0 and describes broader evaluation across multiple safety taxonomies.Source: Benchmarking Datasets · p. 5

2024) datasets.

This section describes benchmark datasets used to evaluate moderation and tables summarizing training examples and guardrail training-set sizes.

  • Response-level assessment covers RobloxGuard-Eval, Aegis 2.0, BeaverTails, HarmBench, SafeRLHF, and WildGuard across varied, challenging content domains.Source: 2024) datasets · p. 5
  • Aegis 2.0 extends Aegis 1.0 with a scalable taxonomy of 12 core and 9 fine-grained safety-violation categories, designed to improve alignment of LLM safety guardrails.Aegis 1.0 is a human-labeled dataset.Source: 2024) datasets · p. 5
  • WildGuard combines vanilla and red-teaming queries, including WildTeaming jailbreak examples; its 86.8K training queries and 1.7K test examples support benchmarking.Source: 2024) datasets · pp. 5–6
  • Table 3 breaks training examples down by prompt type, source, and Chain-of-Thought inclusion, while Table 4 compares guardrail training-set sizes.Source: Table 3 · p. 6, Table 4 · p. 6
  • Other benchmark descriptions include SimpleSafetyTest’s straightforward harmful prompts, Toxic Chat’s conversational toxicity, XSTest’s exaggerated-safety diagnostics, and BeaverTails’ 14 harmful categories.Source: 2024) datasets · p. 6
  • HarmBench comprises 510 instances across four functional and seven semantic categories for assessing robustness against harmful behavior.Source: 2024) datasets · p. 6

Experiments

Experiments report strong results across prompt- and response-level benchmarks, including out-of-domain and domain-specific evaluations. Ablations examine the effects of synthetic data, CoT rationales, and input inversion.

Roblox Guard 1.0 Training

  • The model uses LoRA fine-tuning of Llama-3.1-8B-Instruct on more than 384k diverse policy instructions.Training used rank 16 and ran for 3 epochs; inference averaged 869.9 ms for a 790-token payload over 10 runs.Source: Roblox Guard 1.0 Training · p. 6

Results

  • Roblox Guard 1.0 consistently achieves state-of-the-art performance across prompt- and response-based benchmarks.Table 5 compares guards across prompt-based and response-based attacks using F1 scores.Source: Table 5 · p. 7, Results · p. 7
  • 91.9% F1 on Aegis 1.0 Prompt and 89.5% on WildGuard Prompt surpass all competing models, including GPT-4o.Source: Results · p. 7
  • 87.3% F1 on BeaverTails outperforms LlamaGuard3, ShieldGemma, and GPT-4o; results on Harmbench and SafeRLHF are competitive.Source: Results · p. 7
  • On out-of-domain benchmarks, Roblox Guard 1.0 scores 79.1% on Toxic Chat, 69.9% on SafeRLHF, 86.4% on XSTest, and 85.7% on HarmBench.The paper attributes this generalization to diverse instructions generated via input inversion and a compositional strategy treating taxonomies as related tasks.Source: Results · p. 7

Ablations

Results

  • RobloxGuard-Eval covers 23 nuanced safety categories; existing guards can fall below 30% F1, while Roblox Guard 1.0 maintains high accuracy without task-specific tuning.The categories include underrepresented harms such as promotional scams, off-platform solicitation, and deceptive monetization.Source: Results · pp. 7–8

Conclusion

The paper presents Roblox Guard 1.0 as a binary classifier for both prompts and responses, reporting generalization to unseen taxonomies and strong or comparable benchmark performance. It also introduces RobloxGuard-Eval, an expert-annotated dataset aligned with Roblox’s safety taxonomy.

  • Roblox Guard 1.0 is a binary classification system operating at both prompt and response levels, with strong generalization to unseen taxonomies and state-of-the-art or comparable benchmark performance.Source: Conclusion · p. 8
  • RobloxGuard-Eval is an expert-annotated dataset aligned with Roblox’s safety taxonomy, designed to evaluate taxonomy awareness and out-of-domain generalization.Source: Conclusion · p. 8
Loading 2512.05339v1…