PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3Guard Technical Report

Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, Baosong Yang, Chen Cheng, Jialong Tang, Jiandong Jiang, Jianwei Zhang, Jijie Xu, Ming Yan, Minmin Sun, Pei Zhang, Pengjun Xie, Qiaoyu Tang, Qin Zhu, Rong Zhang, Shibin Wu, Shuo Zhang, Tao He, Tianyi Tang, Tingyu Xia, Wei Liao, Weizhou Shen, Wenbiao Yin, Wenmeng Zhou, Wenyuan Yu, Xiaobin Wang, Xiaodong Deng, Xiaodong Xu, Xinyu Zhang, Yang Liu, Yeqiu Li, Yi Zhang, Yong Jiang, Yu Wan, Yuxin Zhou

arXiv:2510.14276v1cs.CL

TL;DR

Existing guardrails struggle with policy-dependent labels and full-response-only checking. Qwen3Guard introduces generative tri-class moderation and token-level streaming detection, achieving strong multilingual benchmark performance while retaining practical deployment flexibility. Its streaming variant remains subject to reduced robustness under partial context.

  • Problem

    Existing guard models use inconsistent safety policies and generally require complete responses, limiting adaptable moderation and real-time monitoring.

  • Method

    Qwen3Guard combines instruction-following Generative Qwen3Guard with token-level Stream Qwen3Guard for incremental safety detection.

  • Results

    Qwen3Guard achieves state-of-the-art performance across English, Chinese, and multilingual safety benchmarks, while Stream Qwen3Guard supports real-time monitoring with modest degradation.

  • Takeaways & Limitations

    The two variants support fine-grained, multilingual moderation and enable streaming intervention without retraining the underlying model.

  • Takeaways & Limitations

    Stream Qwen3Guard can be less robust for ambiguous or context-dependent phrases under partial context, especially in out-of-distribution or low-resource languages.

Abstract

from arXiv · show

As large language models (LLMs) become more capable and widely used, ensuring the safety of their outputs is increasingly critical. Existing guardrail models, though useful in static evaluation settings, face two major limitations in real-world applications: (1) they typically output only binary "safe/unsafe" labels, which can be interpreted inconsistently across diverse safety policies, rendering them incapable of accommodating varying safety tolerances across domains; and (2) they require complete model outputs before performing safety checks, making them fundamentally incompatible with streaming LLM inference, thereby preventing timely intervention during generation and increasing exposure to harmful partial outputs. To address these challenges, we present Qwen3Guard, a series of multilingual safety guardrail models with two specialized variants: Generative Qwen3Guard, which casts safety classification as an instruction-following task to enable fine-grained tri-class judgments (safe, controversial, unsafe); and Stream Qwen3Guard, which introduces a token-level classification head for real-time safety monitoring during incremental text generation. Both variants are available in three sizes (0.6B, 4B, and 8B parameters) and support up to 119 languages and dialects, providing comprehensive, scalable, and low-latency safety moderation for global LLM deployments. Evaluated across English, Chinese, and multilingual benchmarks, Qwen3Guard achieves state-of-the-art performance in both prompt and response safety classification. All models are released under the Apache 2.0 license for public use.

1 Introduction

Qwen3Guard addresses inconsistent safety policies and the inability of existing guards to moderate streaming outputs. It introduces fine-grained multilingual classification and specialized generative and streaming variants.

  • Motivation: Existing guard models apply inconsistent safety policies and typically cannot assess responses until generation is complete.These limitations undermine reliable policy interpretation and hinder timely intervention.
  • Contributions: Qwen3Guard adds a controversial category alongside safe and unsafe to support adaptable moderation across contexts.The category captures cases whose safety labels vary with context or policy.
  • Contributions: Generative Qwen3Guard reformulates safety classification as instruction following, while Stream Qwen3Guard performs token-level detection during generation.The two variants target full input/output classification and real-time streaming intervention, respectively.
  • Contributions: Qwen3Guard is offered in 0.6B, 4B, and 8B parameter sizes and supports 119 languages and dialects.The size range targets diverse deployment constraints, while multilingual coverage supports global applications.
  • Results: Qwen3Guard achieves state-of-the-art performance across English, Chinese, and multilingual safety benchmarks for prompt and response classification.The reported evaluation covers both Generative and Stream variants across diverse safety tasks.

2 Safety Policy

The safety policy defines what content requires moderation and uses three severity levels to accommodate application-specific risk tolerance. It covers harmful inputs and outputs across broad societal and ethical categories.

  • Policy Principles: The policy guides guardrail alignment, training-data annotation, and interpretation of safety evaluations.It defines dialogues considered unsafe or warranting caution.
  • Policy Principles: Qwen3Guard detects harmful topics or elicitation attempts in inputs and harmful information or advice in outputs.Input and output classification apply different moderation targets.
  • Severity Levels: Applications can selectively enforce Safe, Controversial, or Unsafe severity levels according to their risk tolerance.Safe content is generally acceptable, Unsafe content is harmful across most scenarios, and Controversial content depends on context or disagreement.
  • Safety Categories: The policy includes categories such as violence, non-violent illegal acts, sexual content, personal information, self-harm, unethical acts, political misinformation, and copyright violation.Jailbreak is reserved for input classification because it concerns attempts to override system prompts or model conditioning.
  • Safety Categories: Jailbreak labels apply only to inputs, while output categorization depends on the potential harm caused by the generated response.This separates prompt-manipulation attacks from harms in assistant outputs.

3 Generative Qwen3Guard

Generative Qwen3Guard performs instruction-following safety assessment using policy-guided data and refinement procedures. Its training pipeline constructs controversial labels and reduces annotation noise through cross-partition distillation.

  • Task Definition: Generative Qwen3Guard is built on instruction-tuned Qwen3 models and trained with supervised fine-tuning for generative moderation.The model receives moderation instructions and target content, then generates a predefined-format assessment.
  • Task Definition: The generated assessment covers the latest user query or assistant response, with refusal detection additionally reported for responses.Instructions include task definition, policies, categories, dialogue context, and output formatting.
  • Data Collection: The dataset combines more than 1.19M human-annotated and synthetically generated positive and negative samples across languages.Multilingual content was translated into 15 additional languages and validated using language-mixing detection, an LLM judge, and manual review.
  • Data Collection: Prompt synthesis uses a fine-grained safety taxonomy, keyword guidance, and paired safe-unsafe examples to broaden coverage while reducing lexical shortcuts.Paired examples share surface structure, such as “How to make a bomb” and “How to make a cake.”
  • Data Collection: Multiple Qwen models generate preliminary labels, which are aggregated by voting; safety-level labels exceed 0.9 F1 on a human-annotated validation set.Category and refusal labels use the most frequently predicted output.
  • Training: The pipeline addresses scarce controversial examples and annotation noise through reweighted cross-partition labeling and knowledge distillation.Strict and Loose models annotate held-out partitions, conflicts become Controversial, and Qwen3-32B distillation reduces annotation errors.

3.4 Evaluation

Qwen3Guard-Gen is evaluated on English, Chinese, and multilingual prompt and response safety benchmarks, including reasoning-trace moderation. It achieves strong cross-language performance while addressing policy inconsistency through a controversial category and configurable strict or loose treatment.

  • Evaluation Setup: The evaluation spans English prompt and response benchmarks, Chinese datasets, multilingual RTP-LX and PolyGuard-Response tests, and a manually annotated reasoning-trace dataset called Think.English benchmarks include ToxicChat, OpenAIModeration, Aegis, HarmBench, SafeRLHF, Beavertails, XSTest, and WildguardTest.
  • Main Results: Qwen3Guard-Gen achieves top performance on 8 of 14 public English benchmarks, while its 0.6B model rivals or exceeds guards more than 10× larger.The evaluation covers both prompt and response classification.
  • Main Results: Qwen3Guard-Gen outperforms prior guards on 6 of 10 major languages for prompt classification and achieves top performance across all 10 for response classification.The model also maintains strong generalization on languages with limited training coverage.
  • Policy Adaptability: The controversial label supports policy-specific decisions: treating controversial samples as unsafe suits stricter Aegis, whereas treating them as safe suits more permissive OpenAIMod.Existing guards show policy inconsistency; WildGuard-7B aligns with Aegis but is overly conservative on OpenAIMod.
  • Ablation Study: Across most datasets, strict or loose handling of controversial cases outperforms binary classification without that label, especially on permissive ToxicChat and OpenAIModeration.The table reports F1 scores for prompt and response classification, with the best mode selected per benchmark.
  • Ablation Study: Distillation improves Qwen3Guard-Gen by +0.47/+1.10 points on prompts and +0.5/+0.76 points on responses by filtering noisy annotations.The reported improvements make category decision boundaries more separable.

3.5 Application I: Safety RL with Generative Qwen3Guard

This application uses Generative Qwen3Guard as a safety reward in reinforcement learning for Qwen3-4B. It compares a safety-only objective with a hybrid objective that also rewards helpfulness and penalizes refusal.

  • Application: Generative Qwen3Guard provides safety judgments as a reward signal for aligning Qwen3-4B against harmful or adversarial prompts.The training aims to preserve safety without producing blanket refusals that harm user experience.
  • Reward Design: The guard-only reward maximizes response safety by treating only Qwen3Guard’s Safe prediction as safe; Unsafe and Controversial predictions are not safe.The reward is defined over prompt x, thinking content t, and final output y.
  • Reward Design: The hybrid reward jointly optimizes high safety, high helpfulness, and low refusal rate to mitigate degeneration from safety-only optimization.WorldPM-Helpsteer2 supplies the helpfulness score, while Qwen3Guard supplies safety and refusal predicates.
  • Training: Training uses GSPO with eight Qwen3-4B responses per WildJailbreak prompt across thinking and non-thinking modes.Samples with uniform outcomes across all eight responses are filtered out.
  • Evaluation: Evaluation uses 2,000 harmful and 210 benign WildJailbreak prompts, with safety judged by Qwen3-235B-Instruct-2507 and WildGuard rather than Qwen3Guard.General utility is additionally measured on Arena-Hard-v2, AIME-25, LiveCodeBench-V6, and GPQA.

3.6 Experiment Results

Safety RL reveals a trade-off between safety and refusal under the guard-only reward, while the hybrid reward improves safety without the same degradation. Training dynamics show steadily increasing safety while avoiding over-refusal.

  • Experiment Results: The guard-only reward achieves near-perfect safety but produces an extremely high refusal rate.Arena-Hard-v2 win rate declines slightly, while AIME25, LCB-v6, and GPQA are not noticeably affected.
  • Experiment Results: The hybrid reward raises WildGuard safety from approximately 60 to over 97 across thinking and non-thinking modes while penalizing excessive refusal.Arena-Hard-v2 response quality shows marginal improvement under the WorldPM signal.
  • Experiment Results: The hybrid framework produces a model that is simultaneously safer, more helpful, and retains high general capability.A qualitative before-and-after safety-RL case study is provided in Figure 13.
  • Training Dynamics: Figure 6 tracks safety rate and refusal rate over RL training steps and shows that hybrid reward avoids over-refusal while steadily enhancing safety.The dynamics are measured by Qwen3Guard-Gen-4B.

4 Stream Qwen3Guard

Stream Qwen3Guard performs real-time safety moderation by classifying prompts and generated tokens through parallel pathways. It uses rollout-based and judge-based token annotation, and maintains strong performance with lower scaling overhead than Generative Qwen3Guard.

  • Architecture: Stream Qwen3Guard evaluates user prompts and each generated response token, enabling immediate intervention during streaming generation.The architecture uses separate Prompt Moderator and Response Moderator heads.
  • Architecture: The model attaches two classification heads to a pre-trained Qwen3 backbone, processing query and response representations through parallel pathways.The heads predict harm severity levels and safety categories for both streams.
  • Data Collection: Token-level annotations are created by identifying unsafe-content boundaries through rollout-based safety assessment followed by LLM-as-judge verification.For each token prefix, multiple continuations are generated and assessed before the judge checks the prefix itself.
  • Data Collection: 85% is the rollout safety-violation threshold found in pilot experiments to align strongly with human safety annotations.The threshold is applied to Generative Qwen3Guard safety predictions for rollout assessment.
  • Evaluation: Stream Qwen3Guard achieves nearly 86.0% exact hit rate for response-only boundary detection and detects unsafe content within 128 tokens in approximately 66.8% of thinking cases.Thinking content is harder to moderate because reasoning traces are informal and unstructured.
  • Evaluation: Stream Qwen3Guard’s processing time scales nearly linearly with response length, whereas Generative Qwen3Guard incurs substantially higher computational overhead as responses grow longer.Stream Qwen3Guard shows only a marginal performance decline compared with the full-context Generative variant.

5 Related Work

Related guard-model work primarily uses instruction-following classification, while prior token-level approaches rely on approximate or indirect labels. Stream Qwen3Guard instead estimates incomplete-sentence labels using rollouts and Generative Qwen3Guard predictions.

  • Existing Guard Models: Most existing guard models use supervised instruction-following to classify inputs and outputs into discrete safety categories.These models operate as external mechanisms for enforcing predefined safety policies.
  • Token-Level Methods: Prior token-level methods typically approximate token labels from sentence-level annotations or use indirect learning methods.These approaches address token-level safety classification without directly obtaining fine-grained labels for incomplete responses.
  • Stream Qwen3Guard: Stream Qwen3Guard samples multiple rollouts and uses Generative Qwen3Guard predictions to estimate labels for incomplete sentences.This procedure is presented as a way to obtain more accurate token-level annotations.

6 Conclusion

Qwen3Guard is a multilingual safety classification series designed for diverse moderation contexts, combining three-tier judgments with real-time token-level detection. Extensive experiments show strong performance across English, Chinese, and multilingual safety benchmarks.

  • Qwen3Guard introduces a controversial category alongside safe and unsafe to support more flexible moderation decisions across regions, platforms, and use cases.
  • Generative Qwen3Guard reformulates safety classification as a generative task, while Stream Qwen3Guard performs token-level detection during incremental text generation.
  • Extensive experiments demonstrate strong performance across multiple safety benchmarks spanning English, Chinese, and multilingual datasets.
  • Qwen3Guard is presented as an enhancement to content moderation rather than a complete solution to the broader, ongoing challenge of AI safety.

Limitations

The paper identifies deployment boundaries involving adversarial robustness, moderation bias, limited generalization, and insufficient adaptation to regional and cultural contexts. It frames these issues as important considerations for real-world use and ongoing research.

  • Qwen3Guard may be vulnerable to adversarial prompt engineering, including paraphrasing, obfuscation, and context manipulation that can bypass safety filters.The paper states that performance may degrade under sophisticated, targeted attacks despite robustness on standard benchmarks.
  • Training annotations may encode cultural and dataset biases, potentially causing disproportionate unsafe or controversial flags for some demographic, linguistic, or cultural groups.
  • Stream-Guard’s partial-context, low-latency design can reduce robustness for ambiguous phrases and increase false-positive or false-negative rates in out-of-distribution or low-resource languages.
  • Qwen3Guard lacks geolocation-aware and policy-adaptive mechanisms, which may restrict effectiveness in localized deployments.
  • The authors describe these limitations as targets for continued research and iterative improvement of safety classifiers.

Authors

The paper lists core contributors separately from additional contributors ordered alphabetically. Together, these passages provide the named author roster.

  • The core contributors are Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
  • The author information distinguishes a core-contributor group from a larger alphabetically ordered contributor group.
  • Additional contributors are listed in alphabetical order, beginning with Baosong Yang and Chen Cheng and extending through Yuxin Zhou.

Appendix

The appendix documents Generative Qwen3Guard’s classification instructions, Stream Qwen3Guard’s category and latency evaluation materials, and an example of safety-reinforced responses. It also includes figures illustrating prompts, confusion matrices, and model outputs.

  • Generative Qwen3Guard: Generative Qwen3Guard instructions evaluate user queries for safety and unsafe-content categories using safe, unsafe, and controversial labels.
  • Generative Qwen3Guard: Response-classification instructions additionally ask whether the assistant response is a refusal, alongside safety and unsafe-content categorization.
  • Stream Qwen3Guard: Stream Qwen3Guard distinguishes different safety categories in most unsafe scenarios, with its confusion matrices documenting prompt and response categorization.
  • Evaluation: Latency evaluation uses sentence-level annotations because token-level annotation is difficult and often has low inter-annotator agreement.
  • Safety reinforcement example: A WildJailbreak example shows Qwen3-4B-SafeRL avoiding harmful advice while attempting to fulfill the request without outright refusal.
Loading 2510.14276v1…