FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation

1 Introduction

• Most moderators use binary labels tied to one safety policy, although enforcement strictness varies across platforms and changes over time.The same prompt–response pair may be removed in a strict setting but allowed in a looser one.Source: §1 · pp. 1–2
• Existing moderators show cross-strictness inconsistency, with prompt-moderation F1 drops of 19.2% for Qwen3Guard and 15.7% for BingoGuard.FlexBench evaluates strict, moderate, and loose regimes; the marked best-to-worst drops are shown in Figure 2.Source: Figure 2 · p. 2, §1 · p. 2
• FlexGuard predicts a calibrated continuous risk score intended to reflect severity, then supports strictness-specific decisions through thresholding.Its training uses rubric-grounded score supervision and risk-alignment optimization.Source: §1 · p. 2
• FlexGuard improves average moderation performance and worst-regime robustness on FlexBench and additional public benchmarks.The introduction reports both gains without specifying numerical values.Source: §1 · p. 2

2.1 LLM based Content Moderators

• Moderation models have added richer training data, finer-grained tasks, and reasoning-oriented fine-tuning or reinforcement learning, while most still formulate moderation as binary classification.Some severity-prediction approaches assess risk only after an instance is classified as unsafe.Source: §2.1 · pp. 2–3

2.2 Content Moderation Benchmarks

• Existing moderation benchmarks cover varied content and scenarios, but generally use fixed binary labels or predefined severity classes rather than evaluating varying enforcement strictness.The passage notes that even benchmarks with severity annotations retain a fixed evaluation setup.Source: §2.2 · p. 3

3 FlexBench

• FlexBench addresses the lack of benchmarks for strictness-adaptive moderation by evaluating moderators across strict, moderate, and loose enforcement regimes.The benchmark is intended to assess robustness under differing deployment requirements.Source: §3 · p. 3

3.1 Strictness-Adaptive Moderation

• Strictness-adaptive moderation evaluates predictions against labels induced by a deployment’s strictness parameter, rather than a single fixed safety definition.The task expects robust performance as the strictness parameter varies.Source: §3.1 · p. 3

3.2 Benchmark Construction

• FlexBench contains 4K human-annotated instances—2K prompts and 2K prompt–response pairs—and covers seven core risk categories.Its categories use five severity tiers, with rubrics for scoring prompts and assistant responses.Source: §3.2 · pp. 3–4
• The five severity tiers map to three regimes: strict treats only BENIGN as safe, moderate adds LOW, and loose treats through MODERATE as safe.HIGH and EXTREME are unsafe in the loose regime.Source: §3.2 · p. 3
• Six trained annotators verify and correct LLM-generated category and severity labels in two rounds, with discrepancies resolved by a senior annotator.The workflow reports 69.9% LLM–human agreement for prompts and 63.2% for responses.Source: §3.2 · p. 4

3.3 Initial Findings

• All evaluated state-of-the-art moderators show substantial cross-strictness inconsistency; Qwen3Guard prompt-moderation F1 drops 19.2% from its best to worst regime.The response-moderation drop is 14.8%; threshold tuning and rubric-conditioned prompting do not yield stable behavior as strictness shifts.Source: §3.3 · p. 5

4.1 Continuous Risk Scoring

• FlexGuard predicts a calibrated 0–100 risk score and category, enabling deployment-specific decisions by thresholding the score.Higher scores indicate higher risk severity, and the threshold can shift with enforcement requirements.Source: §4.1 · p. 5

4.2 Rubric-Guided Score Distillation Pipeline

• On Aegis2.0 and WildGuardMix training splits, rubric judges produce categories, severity scores, and rationales; source-label calibration improves LLM–human agreement.The judge is selected by comparison with human annotations on 1,000 held-out instances.Source: §4.2 · pp. 5–6

4.3 Risk Alignment Training

• Training first uses SFT for rubric-grounded rationales and category/score outputs, then GRPO optimizes category accuracy alongside a dense reward that decreases with score error.The score-error reward supplies dense learning signals and is intended to reduce sensitivity to occasional label noise.Source: §4.3 · p. 6

4.4 Adaptive Threshold Selection

• Rubric thresholding sets strict, moderate, and loose cutoffs to 20, 40, and 60 (or a default 40); validation-based thresholding selects the value maximizing F1.Validation-based selection applies when binary safety labels under the target strictness are available.Source: §4.4 · p. 6

5.1 Experimental Setup

• The setup compares rubric-prompted, logit-thresholded, and level-thresholded moderators, alongside public prompt- and response-moderation benchmarks, using unsafe-class F1.Baseline thresholds are selected by strictness regime on the FlexBench validation split.Source: §5.1 · pp. 6–7

5.2 Overall Performance

• 5.85% over Doubao-1.8 on prompt moderation and 9.64% over GPT-5 on response moderation, FlexGuard with calibrated thresholds achieves the best Average and Worst F1 on FlexBench.Table 1 reports harmfulness F1 across three strictness regimes.Source: Table 1 · p. 7, §5.2 · p. 7
• Qwen3Guard’s prompt-moderation F1 decreases by 19.2% from strict to loose, while rubric-prompted and level-based baselines also show cross-regime inconsistencies.The reported pattern indicates that prompt adjustment or discrete severity prediction alone does not yield stable strictness adaptation.Source: §5.2 · p. 7
• FlexGuard achieves strong average performance across public prompt- and response-moderation benchmarks using benchmark-specific validation thresholds or a default threshold of tτ = 40.These evaluations use each benchmark’s original binary labels.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Ablation Study

• Beta targets alone yield limited robustness; rubric-guided scores improve results, calibration especially helps in looser settings, and GRPO adds the largest gains.GRPO without the score-regression component degrades performance.Source: §5.3 · p. 8, Table 3 · p. 9

5.4 Additional Analysis

• Doubao-1.6-Pro has the highest judge–human agreement for prompt and response annotation, calibration improves agreement for all judges, and judge scores track human severity monotonically.The monotonicity analysis uses a 1,000-instance human-annotated subset.Source: §5.4 · p. 8, Table 4 · p. 9, Table 5 · p. 9
• Across three backbones, FlexGuard maintains a similar trend across strictness regimes, while Qwen3-4B shows a noticeable performance drop, particularly for prompt moderation.Figure 4 presents backbone performance on FlexBench across the three regimes.Source: §5.4 · p. 9

• FlexBench supports controlled comparison of moderators under three strictness regimes.Source: §6 · p. 9
• Figure 4 evaluates FlexGuard with different backbones on FlexBench across the three regimes.Source: Figure 4 · p. 9

• Benchmark construction and experiments use English-only data, so effectiveness is validated only for English moderation, not multilingual or code-mixed settings.Source: §7 · p. 9
• The risk-score distillation pipeline uses Aegis2.0 and WildGuardMix, without systematic evaluation of other sources or shifted training distributions.Source: §7 · p. 9
• The alignment stage uses GRPO with a designed score-regression reward; alternative post-training algorithms such as DAPO/GSPO-style variants are unexplored.The authors note these alternatives may improve robustness or reduce sensitivity to noisy pseudo labels.Source: §7 · pp. 9–10

8 Ethical Considerations

• FlexBench uses public benchmarks, avoids private user logs and proprietary platform data, and follows source-compatible licensing and redistribution terms.The release policy is restricted to licenses and terms compatible with the source datasets.Source: §8 · p. 10
• Six professional annotators receive taxonomy and rubric training, content warnings, breaks, opt-outs, distress escalation, senior review, and compensation intended to be fair.A two-round review process with senior adjudication is used to reduce individual burden and improve label quality.Source: §8 · p. 10
• Severity tiers and strictness regimes are operationalizations, not universal standards, and residual subjectivity and annotator or rubric bias may remain.The authors recommend recalibrating thresholds and validating behavior for each policy and deployment context.Source: §8 · p. 10
• FlexBench and FlexGuard carry dual-use risks, including facilitating censorship or probing decision boundaries for evasion.The authors recommend rate limiting, monitoring for systematic probing, and human oversight for borderline cases.Source: §8 · p. 10
• Potential release mechanisms include documentation discouraging evasion optimization and restricting access to the most operationally harmful examples while preserving research utility.The authors state that such mechanisms would be considered if FlexBench or model artifacts are released.Source: §8 · p. 10
• AI assistants were used for language polishing and minor code-editing suggestions; the authors say they developed and validated the research contributions and reviewed final outputs.The stated contributions include benchmark design, taxonomy, data selection, annotation, training, experiments, and analysis.Source: §8 · p. 10

A Dataset Details

• FlexBench’s prompt and response splits draw from public moderation datasets including XSTest, ToxicChat, WildGuardTest, OpenAI Moderation, Aegis2.0, PKU-SafeRLHF, HarmBench, and BeaverTails.The passage identifies the prompt-split sources and begins listing response-split sources; the accompanying source list names further datasets.Source: §A.1 · p. 12
• XSTest is an adversarial jailbreaking benchmark with 5k samples, while WildGuardTest contains 30k human-verified examples from real-world online interactions.WildGuardTest emphasizes naturally occurring toxic content, including hate speech and harassment.Source: §A.1 · p. 12
• ToxicChat provides 110k context-aware dialogue turns from 10k multi-round chats, while OpenAI Moderation contains millions of samples with fine-grained safety-risk labels.ToxicChat labels account for conversational context; OpenAI Moderation uses human-model collaborative annotation.Source: §A.1 · p. 12
• Aegis2.0 has 50k samples with dynamic adversarial and multilingual coverage; PKU-SafeRLHF includes 19 harm categories, three severity levels, and 44.6k refined prompts.BeaverTails contains 334.4k instances and separates helpfulness and harmlessness annotations.Source: §A.1 · pp. 12–13
• The described curation pipeline contains 50k user input examples and 50k LLM response examples across four harm categories, with human-verified labels.The categories are sexually explicit, dangerous content, hate, and harassment.Source: §A.1 · p. 13

A.2 FlexBench Statistics

• Table 6 reports instance counts broken down by risk severity, category, and data source.The supplied passage describes the table’s scope but gives no individual count values.Source: §A.2 · p. 13

A.3 Training Corpus Sources

• FlexGuard’s training corpus uses the Aegis2.0 and WildGuardMix training splits, then removes exact-string user-query overlaps with FlexBench.The collection follows Sreedhar et al. (2025).Source: §A.3 · p. 13
• Aegis2.0 contains human-LLM safety interactions labeled with 12 core hazard categories and an extension of nine fine-grained risks.It includes dialogue-level human annotations and turn-level response labels derived through a jury-of-LLM procedure.Source: §A.3 · p. 13
• WildGuardMix has roughly 92K examples, including about 87K training and 5.3K test examples, balanced across direct and adversarial prompts.Its prompt-response pairs include compliant and refusal-style responses, and its sources include synthetic, real-world, and annotator-written safety data.Source: §A.3 · pp. 13–14

B Implementation Details

• FlexGuard training uses SFT warm-up followed by GRPO alignment, with rubric-distilled risk categories, scores, and rationales balanced across five severity tiers.Pseudo risk scores are divided into five equal-width bins of 20 points, then downsampled to balance the tier distribution.Source: B Implementation Details · p. 14
• BingoGuard-Llama3.1-8B and a post-trained Llama-3.1-8B-Instruct PKU-SafeRLHF baseline use their original paper’s training hyperparameters and prompt templates.Both baselines use the same inference settings as FlexGuard.Source: B Implementation Details · p. 14
• The implementation specifies TRL and VERL training settings and uses vLLM for inference.TRL uses batch_size_per_gpu 2, 1 epoch, and learning_rate 5 × 10−4; VERL uses max_prompt_length 2046, max_response_length 512, and learning_rate 1 × 10−6. vLLM uses temperature 1.0, top_p 0.9, max_tokens 1024, logprobs 5, tensor_parallel_size 8, bfloat16, and repeat 3.Source: B Implementation Details · p. 14

C More Results

• All baselines lag FlexGuard in Average and Worst F1 for prompt and response moderation when using native binary predictions without strictness adaptation.Table 7 reports harmfulness F1 across three strictness regimes; the supplied passage gives no exact scores.Source: C More Results · p. 14, Table 7 · p. 15
• For a torture-instruction response, FlexGuard identifies VIO as the sole risk category and assigns a score of 94, citing rubric evidence for extreme risk.Its reasoning links high thematic proportion, concentrated violent content, and actionable method details to the score, which can be thresholded for different strictness requirements.Source: C More Results · pp. 14–15, Table 8 · p. 16

D.1 LLM Annotation

• Annotation prompts derive from expert-designed rubrics that define five severity tiers across seven safety categories.The prompt workflow identifies risk categories, extracts relevant elements, analyzes intent, estimates risk-theme proportion, calculates a continuous score, selects a disposition, and generates a result.Source: §D.1 · pp. 15–16

D.2 FlexGuard

• FlexGuard uses the same response-moderation system-prompt template for training and inference, replacing “Assistant” with “User” for prompt moderation.Source: §D.2 · p. 22
• FlexGuard assigns a 0–100 risk score across seven unsafe-content categories, with score bands from benign/negligible to extreme/severe and the strongest harmful portion guiding the score.SAFE responses must score 0–20; outputs include the category or categories and an integer risk score.Source: §D.2 · pp. 22–23

Prompt Moderation (Strict)

• The strict prompt allows refusals or high-level, non-actionable, or benign content, but requires refusal for actionable or detailed guidance enabling wrongdoing.It requires exactly one output word: ALLOW or REFUSE.Source: Prompt Moderation (Strict) · p. 23

Prompt Moderation (Moderate)

• The moderate prompt uses the same stated decision boundary as the strict prompt: allow refusals or non-actionable benign content, and refuse actionable guidance enabling wrongdoing.It also requires exactly one output word: ALLOW or REFUSE.Source: Prompt Moderation (Moderate) · p. 24

Prompt Moderation (Loose)

• The loose prompt allows partial or incomplete non-executable information without a full actionable workflow, but refuses executable procedures, calls to action, or near-complete guides.The output remains restricted to one word: ALLOW or REFUSE.Source: Prompt Moderation (Loose) · pp. 24–25