Source-linked AI summary
GuardReasoner: Towards Reasoning-based LLM Safeguards
Yue Liu, Hongcheng Gao, Shengfang Zhai, Yufei He, Jun Xia, Zhengyu Hu, Yulin Chen, Xihong Yang, Jiaheng Zhang, Stan Z. Li, Hui Xiong, Bryan Hooi
TL;DR
Existing guard models have limited reasoning, explainability, and generalization, motivating a safeguard that can reason during moderation. GuardReasoner constructs a reasoning dataset, applies reasoning SFT and hard-sample DPO, and demonstrates effectiveness across 13 benchmarks and 3 guardrail tasks. Its reasoning traces support moderation outputs alongside intermediate reasoning processes, although the explainability evaluation remains limited.
Problem
Existing guard models are limited by straightforward instruction tuning, classifier-only outputs, and reliance on manually designed harmful categories.
Method
GuardReasoner combines the 127K-sample, 460K-step GuardReasonerTrain dataset with reasoning SFT and hard sample DPO to train guard models to reason.
Results
GuardReasoner proves effective across 13 benchmarks for 3 guardrail tasks and improves performance, explainability, and generalization.
Takeaways & Limitations
GuardReasoner provides moderation results together with reasoning processes, supporting a more explainable guard model within the evaluated tasks.
Takeaways & Limitations
The explainability evaluation mainly measures human agreement with the model’s Chain-of-Thought, a crude proxy that may not capture model transparency fully.
Abstract
from arXiv · showhide
As LLMs increasingly impact safety-critical applications, ensuring their safety using guardrails remains a key challenge. This paper proposes GuardReasoner, a new safeguard for LLMs, by guiding the guard model to learn to reason. Concretely, we first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps. Then, we introduce reasoning SFT to unlock the reasoning capability of guard models. In addition, we present hard sample DPO to further strengthen their reasoning ability. In this manner, GuardReasoner achieves better performance, explainability, and generalizability. Extensive experiments and analyses on 13 benchmarks of 3 guardrail tasks demonstrate its superiority. Remarkably, GuardReasoner 8B surpasses GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84% F1 score on average. We release the training data, code, and models with different scales (1B, 3B, 8B) of GuardReasoner : https://github.com/yueliu1999/GuardReasoner/.
1 INTRODUCTION
Existing guard models face limitations in reasoning ability, explainability, and generalization. GuardReasoner addresses these challenges with a reasoning dataset, reasoning SFT, and hard-sample preference optimization.
- Motivation: Existing guard models often rely on straightforward instruction tuning, provide moderation results without explanations, and depend on fixed harmful categories.These design choices limit reasoning ability and performance, weaken explainability, and hinder handling of new harm types.
- Approach: GuardReasoner applies reasoning supervised fine-tuning to unlock basic reasoning capability in base models of different sizes.The paper uses LLaMA 3.2 1B, LLaMA 3.2 3B, and LLaMA 3.1 8B as base models.
- Approach: Hard sample DPO trains on ambiguous samples containing both correct and incorrect outputs, up-weighting samples with more errors.Correct outputs and their reasoning processes are positive items, while incorrect outputs are negative items.
- Contributions: The resulting model is designed to improve performance, explainability, and generalization by producing moderation results together with reasoning processes.The paper reports extensive experiments and analyses demonstrating GuardReasoner’s effectiveness.
- Approach: GuardReasonerTrain contains about 127K samples and 460K detailed reasoning steps synthesized with GPT-4o.The data is created from existing red-teaming datasets using user prompts, target-model responses, and ground truth.
2 GUARDREASONER
GuardReasoner is trained to reason through supervised reasoning data and hard-sample preference optimization, producing moderation decisions alongside reasoning processes. Its pipeline defines guardrail tasks, mines ambiguous samples, and emphasizes difficult examples during training.
- Task Definition: Guardrail evaluation covers prompt harmfulness, response harmfulness, and refusal detection, using F1 score between predicted and ground-truth labels.
- Reasoning Data Synthesis: GuardReasonerTrain contains about 127K samples and 460K detailed reasoning steps synthesized to support reasoning-based guard-model training.
- Reasoning Supervised Fine-tuning: R-SFT trains a base model to generate both intermediate reasoning processes and moderation results from an instruction, prompt, and target-model response.
- Hard Sample Direct Preference Optimization: HS-DPO identifies ambiguous samples near the decision boundary by generating multiple outputs and selecting cases containing both correct and incorrect predictions.
- Hard Sample Direct Preference Optimization: HS-DPO treats correct classifications with their reasoning as positive examples and incorrect outputs as negative examples, while weighting harder samples more heavily.
- Inference with Reasoning: During inference, GuardReasoner outputs moderation results together with intermediate reasoning steps, supporting performance, explainability, and generalizability.
3 EXPERIMENTS
GuardReasoner is evaluated against 22 baselines across 13 benchmarks covering prompt harmfulness, response harmfulness, and refusal detection. It achieves leading or competitive F1 performance across these tasks, with especially strong aggregate and adversarial-prompt results.
- Experimental Setup: 13 benchmarks cover 3 guardrail tasks, evaluated using sample-weighted average F1 across benchmarks.The tasks are prompt harmfulness detection, response harmfulness detection, and refusal detection.
- Prompt Harmfulness Detection: GuardReasoner 8B achieves 81.09% average F1 on prompt harmfulness detection, surpassing the open-source and closed-source runner-ups by 3.10% and 3.09%.Performance improves more on adversarial benchmarks, including a 5.36% increase on ToxicChat.
- Response Harmfulness Detection: GuardReasoner 8B leads response harmfulness detection with 81.22% F1, outperforming the closed-source and open-source runner-ups by 6.77% and 2.55%.GuardReasoner 1B also surpasses the runner-ups MD-Judge 7B and GPT-4o+CoT.
- Refusal Detection: GuardReasoner achieves 89.96% F1 on refusal detection, closely matching GPT-4.The paper characterizes refusal detection as relatively simple, with several models achieving promising performance.
- Overall Performance: GuardReasoner 8B achieves 84.09% average F1 across the three guardrail tasks, exceeding GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84%.The comparison aggregates results from the prompt harmfulness, response harmfulness, and refusal detection tasks.
- Additional Analysis: GuardReasoner also outperforms R2-Guard on AUPRC for BeaverTails and XSTest.This provides an additional comparison using AUPRC rather than F1.
3.3 Ablation Study
The ablation study shows that reasoning-based training improves guard-model performance beyond conventional dataset mixing, with hard-sample preference optimization providing further gains.
- Baseline-Mix achieves comparable performance to Baseline, indicating that mixing conventional training datasets alone does not significantly improve performance.
- R-SFT surpasses Baseline-Mix by 6.30% F1 on 1B models, verifying the effectiveness of GuardReasonerTrain and reasoning supervised fine-tuning.
- HS-DPOensemble outperforms HS-DPOself, indicating that greater hard-sample diversity improves the preference-optimization stage.
- GuardReasoner reasoning steps analyze request content, assess harmfulness, and produce a harmful or unharmful classification.
- HS-DPO further improves R-SFT, with DPO surpassing R-SFT by 6.34% on HarmBenchPrompt and 10.42% on XSTestResponseRefusal.
3.4 Efficiency Experiment
GuardReasoner has similar training memory costs to the baseline but requires more training time because it learns from reasoning data across two stages.
- The efficiency experiment compares GuardReasoner and Baselinemix using equal training-sample amounts and records training and inference costs.
- GuardReasoner has similar GPU memory costs to the baseline during both R-SFT and HS-DPO training stages.For 8B models, GuardReasoner uses 270.86 GB during R-SFT and 273.95 GB during HS-DPO, compared with 270.78 GB for Baselinemix during SFT.
- GuardReasoner increases training time by 40% ∼50% because it learns from reasoning data and uses two training stages.
3.5 Scaling of GuardReasoner
Scaling experiments find that both larger model sizes and more training data improve GuardReasoner’s performance, while its open-ended categories support generalizability beyond fixed label sets.
- Increasing model size and expanding training data both improve performance in the prompt harmfulness task.
- GuardReasoner provides open-ended non-fixed categories in an AegisSafetyTest case, unlike the fixed-category framing described for LLaMA Guard 3.
3.6 Case Study
The case studies examine performance, explainability, and convergence, while the experiments also document high rejection rates for API-based guardrails.
- Performance: GuardReasoner 8B successfully defends both a conventional case and an attack in the case-study evaluation.
- Explainability: GuardReasoner provides explanations that help human annotators correct mislabelled samples.
- Rejection Rate: API-based guardrails have high rejection rates because requests may themselves contain harmful content, affecting practical application as general LLM guardrails.
- Convergence: GuardReasoner’s R-SFT and HS-DPO losses decrease and its performance increases during convergence.
4 IMPLEMENTATIONS
The implementation evaluates GuardReasoner against closed-source APIs and open-source guard models, then details its reasoning-focused training, prompts, and model configurations.
- Baselines: The baselines include 8 closed-source guard APIs and 13 open-source guard models.
- GuardReasoner Training: GuardReasoner uses LLaMA 3.2 1B, LLaMA 3.2 3B, and LLaMA 3.1 8B as base models for R-SFT.R-SFT trains on GuardReasonerTrain, containing 127K samples and 460K reasoning steps.
- GuardReasoner Training: Reasoning-data synthesis prompts GPT-4 to analyze inputs alongside ground truth and produce detailed reasoning steps.
- GuardReasoner Training: R-SFT training examples contain the user prompt, target LLM output, synthesized reasoning steps, and ground-truth result.
- GuardReasoner Training: HS-DPO uses correct reasoning outputs as positive items and incorrect outputs as negative items for ambiguous samples near the decision boundary.
- Evaluation: GPT-4o+CoT and GPT-4+CoT are evaluated with prompts that instruct step-by-step thinking before classification.
- Evaluation: o1-preview and Gemini 1.5 Pro are evaluated without system prompts, with their instructions moved into the user prompt.
- Evaluation: GuardReasoner inference prompts the model to reason before classifying the user prompt and target LLM response.
5 RELATED WORK
Related work covers LLM alignment, guardrail systems, and methods for eliciting or improving reasoning, positioning GuardReasoner within these research directions.
- Evaluation Analyses: The reported label-correction analyses cover prompt harmfulness, response harmfulness, refusal detection, and average performance across three guardrail tasks.
- LLM Alignment: LLM alignment research collects value-oriented data, constructs instructions from benchmarks, and uses advanced LLMs to generate instructions through in-context learning.
- Guard Models: Guard models separately moderate LLM inputs and outputs, complementing safety alignment performed directly on the LLM.
- Guard Models: Existing guardrails include traditional statistical models, closed-source guard APIs, and open-source guard models.
- Reasoning: Reasoning research includes step-by-step prompting, self-correction, self-critique, debate, and plan-and-solve frameworks.
6 LIMITATIONS
The paper acknowledges that its explainability evaluation is limited in scope and does not fully assess model transparency or reasoning-trace alignment.
- Explainability Scope: The explainability study primarily measures whether humans agree with the model’s prediction when shown its Chain-of-Thought.
- Explainability Scope: The authors characterize human agreement with Chain-of-Thought as a crude proxy that may not capture the nuances of model transparency.
- Future Evaluation: A more rigorous evaluation could examine alignment between the Chain-of-Thought and final prediction for possible discrepancies.
- Future Evaluation: Time and human-effort constraints prevented a fully explored explainability study, which the authors defer to future work.
7 CONCLUSION
GuardReasoner combines GuardReasonerTrain, R-SFT, and HS-DPO to improve reasoning-based guard modeling, with effectiveness shown across 13 benchmarks and three tasks. Future work targets efficiency while preserving accuracy and interpretability.
- Conclusion: GuardReasoner combines GuardReasonerTrain with R-SFT and HS-DPO to unlock reasoning capabilities and learn effective reasoning patterns.
- Conclusion: Experiments across 13 benchmarks and three guardrail tasks show that GuardReasoner is effective.
- Conclusion: The authors release GuardReasoner data, code, and models at three scales.
- Conclusion: Future work will reduce unnecessary reasoning steps through model merging and lightweight reasoning strategies.The stated goal is improving efficiency while preserving accuracy and interpretability.
8 IMPACT STATEMENT
The paper aims to improve guard-model performance, explainability, and generalizability by teaching guard models to reason. Its guard model analyzes human–AI interactions through harmfulness and refusal classifications, illustrated with reasoning steps and label-correction examples.
- 8 IMPACT STATEMENT: Learning to reason is intended to improve guard-model performance, explainability, and generalizability for mitigating harmful impacts from LLMs.The authors also suggest that companies could build guard models for commercial use.
- 8 IMPACT STATEMENT: A worked example concludes that an informative definition of vulgar language is unharmful because it does not promote harmful behavior.The example reaches this conclusion through three explicit reasoning steps evaluating content, behavior promotion, and informativeness.
- 8 IMPACT STATEMENT: Figures document harmful-to-unharmful and unharmful-to-harmful label corrections in BeaverTails, alongside prompts and demonstrations for guard-model inference and training.The materials include prompts for closed-source guard APIs and demonstrations for HS-DPO and R-SFT training data.
- 8 IMPACT STATEMENT: The guard model evaluates whether a user request is harmful, whether an assistant response is a refusal or compliance, and whether that response is harmful.These three tasks analyze interactions between human users and AI assistants.