GuardReasoner: Towards Reasoning-based LLM Safeguards

• Existing guard models are described as limited by weak reasoning, classifier-like outputs without explanations, and reliance on fixed harmful categories.These limitations are framed as problems in performance, explainability, and generalization.Source: §1 · p. 1
• GuardReasonerTrain contains about 127K samples with 460K detailed reasoning steps synthesized using GPT-4o from existing red-teaming datasets.Source: §1 · pp. 1–2
• GuardReasoner uses reasoning SFT followed by HS-DPO, which trains on ambiguous samples by treating correct outputs as positive and incorrect outputs as negative.HS-DPO up-weights samples with more incorrect outputs and down-weights those with more correct outputs.Source: §1 · pp. 1–2
• The authors say reasoning enables GuardReasoner to provide explanations and recognize open-ended harm categories without fixed categories, improving explainability and generalization.They also attribute improved performance to unlocking and enhancing the model’s reasoning ability.Source: §1 · p. 2
• The paper reports extensive experiments and analyses demonstrating GuardReasoner’s superiority and effectiveness.Source: §1 · p. 2

2 GUARDREASONER

• GuardReasoner evaluates prompt and response harmfulness and refusal detection using F1, treating harmful or refusal cases as positives.The guard model receives the user prompt and target LLM response, then predicts labels for all three tasks.Source: §2 · p. 2

2.1 Reasoning Supervised Fine-tuning

• 127K samples with 460K reasoning steps form GuardReasonerTrain, synthesized from four red-teaming datasets to add detailed reasoning to classification labels.GPT-4o generates intermediate steps from prompts, target responses, and ground-truth labels; R-SFT trains the base model to produce reasoning and moderation results.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 Hard Sample Direct Preference Optimization

• Hard samples are inputs where high-temperature, top-p outputs from the R-SFT model mix correct and incorrect results; HS-DPO trains preference for a correct result and its reasoning over an incorrect one.The method up-weights samples with more incorrect outputs and down-weights those with more correct outputs.Source: §2.2 · p. 3
• R-SFT models trained on different GuardReasonerTrain subsets add diversity to mined hard samples, which are merged with self-generated samples; the ensemble-trained HS-DPO model is selected as GuardReasoner.The authors state that the ensemble-trained model achieves better performance than the self-generated-data model.Source: §2.2 · p. 3

2.3 Inference with Reasoning

• At inference, GuardReasoner outputs both a moderation result and intermediate reasoning, which the authors say improves performance, explainability, and generalizability.Source: §2.3 · p. 3

3.1 Datasets

• The evaluation uses 13 benchmarks across prompt harmfulness, response harmfulness, and refusal detection, and reports sample-weighted average F1 because benchmark sizes range from 0.1K to 3K.F1 treats the harmful or refusal category as positive; the benchmarks include six prompt-harmfulness, five response-harmfulness, and two refusal-detection benchmarks.Source: §3 · pp. 3–4, Table 3 · p. 4

3 EXPERIMENTS

• Experiments compare GuardReasoner with 22 baselines on 13 benchmarks and use two servers with eight NVIDIA H100 (80GB) GPUs.The setup also specifies four 56-core CPUs, 2T RAM, and LLaMA Factory [22].Source: §3 · p. 3, §3.2 · p. 4

3.2 Performance

• 81.09% average F1: GuardReasoner 8B leads prompt harmfulness detection, exceeding the open-source and closed-source guard-model runners-up by 3.10% and 3.09%, respectively.It improves by 5.36% on ToxicChat, and performance rises from 77.68% for 1B to 81.09% for 8B; GuardReasoner 1B scores 77.68% versus WildGuard 7B at 77.99%.Source: Table 2 · p. 4, §3.2 · p. 5
• 81.22% F1: GuardReasoner 8B leads response harmfulness detection, beating the closed-source guard API runner-up by 6.77% and the open-source guard-model runner-up by 2.55%.GuardReasoner 1B also surpasses the runner-ups MD-Judge 7B and GPT-4o+CoT.Source: §3.2 · pp. 5–6, Table 5 · p. 5
• 89.96% F1: GuardReasoner performs closely to GPT-4 on refusal detection, a task the paper describes as relatively simple.GPT-4, WildGuard, and GuardReasoner all achieve promising performance on this task.Source: §3.2 · p. 6, Table 8 · p. 6
• 84.09% average F1: GuardReasoner 8B achieves the best performance across three tasks, surpassing GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84%.The paper also reports promising performance for the GPT series and relatively limited performance for Claude 3.5 Sonnet and QwQ.Source: §3.2 · p. 6
• GuardReasoner achieves better AUPRC than R2-Guard in the reported comparison on BeaverTails and XSTest.The supplied passage gives no AUPRC values.Source: Table 6 · p. 5, §3.2 · p. 6

3.3 Ablation Study

• Baseline-Mix performs comparably to Baseline, suggesting that mixing conventional training datasets alone does not significantly improve performance.Baseline uses WildGuardTrain; Baseline-Mix combines seed datasets.Source: §3.3 · p. 6
• On 1B models, R-SFT surpasses Baseline-Mix by 6.30% F1, supporting the effectiveness of GuardReasonerTrain and reasoning SFT.Baseline-Mix combines seed datasets, while R-SFT uses the synthesized GuardReasonerTrain reasoning data.Source: §3.3 · p. 6
• On 1B models, HS-DPO surpasses R-SFT by 6.34% on HarmBenchPrompt and 10.42% on XSTestResponseRefusal; HS-DPOself improves R-SFT, and HS-DPOensemble beats HS-DPOself.The ensemble comparison indicates the effectiveness of increasing hard-sample diversity.Source: §3.3 · pp. 6–7

3.4 Efficiency Experiment

• The efficiency experiment compares GuardReasoner with Baselinemix, an ablation guard model trained on a mix of seed datasets, using equal amounts of training samples.Costs are recorded for training and inference stages.Source: §3.4 · p. 7
• For 8B models, GuardReasoner has similar training GPU memory cost to Baselinemix: 270.86 GB at R-SFT and 273.95 GB at HS-DPO, versus 270.78 GB at SFT.Source: §3.4 · p. 7
• GuardReasoner increases training time by 40%–50% because it learns from reasoning data and has two training stages.Source: §3.4 · p. 7
• Inference memory costs are similar for the methods under vLLM with GPU utilization set to 95%.Source: §3.4 · p. 7
• GuardReasoner has higher but described as tolerable inference costs: 13.87 →35.77 ms/query and 17.09 →260.26 token/query.Source: §3.4 · p. 7

3.5 Scaling of GuardReasoner

• Increasing model size and expanding training data both improve prompt-harmfulness performance across tested dataset combinations.The study examines different data combinations across model sizes; results are reported in Table 11.Source: §3.5 · p. 7
• On an AegisSafetyTest [17] case, GuardReasoner provides open-ended, non-fixed categories in comparison with LLaMA Guard 3.The comparison is illustrated in Figure 6.Source: Figure 6 · p. 7

3.6 Case Study

• GuardReasoner 8B successfully defends both a conventional case and an attack in the case studies.These examples are shown in Figures 3 and 4.Source: §3.6 · p. 7
• 93.8% of annotator judgments agreed when relabeling error predictions using GuardReasoner’s explanations.Three LLM- and robustness-expert annotators reviewed samples; majority vote resolved cases without full consensus, and corrected labels improved performance across three guardrail tasks.Source: §3.6 · p. 7
• Compared with LLaMA Guard 3, GuardReasoner supports open-ended, non-fixed harmful categories, enabling generalization to new classes.This comparison is illustrated in Figure 6.Source: §3.6 · p. 7

3.7 Rejection Rate of API-based Guardrail

• API-based guardrails often reject requests that may themselves contain harmful content, and the paper counts rejected samples as harmful/refused for fairness.The authors note that high rejection rates can affect practical use of general LLMs as guardrails; Table 13 is cited, but its values are not supplied here.Source: §3.7 · p. 7

3.8 Convergence

• GuardReasoner’s R-SFT loss converges, while during HS-DPO the loss decreases and performance increases.The convergence behavior is shown in Figure 7.Source: §3.8 · pp. 7–8
• Efficiency experiments train GuardReasoner on 4 NVIDIA H100 (80GB) GPUs and run inference on 1 GPU.Table 9 presents training and inference costs, with its first and second numbers denoting R-SFT and HS-DPO costs, respectively.Source: Table 9 · p. 8

4.1 Baselines

• The evaluation replicates 8 closed-source guard APIs and 13 open-source guard models using their original code.For o1-preview and QwQ-preview 32B, evaluation uses a random 5% sample of at least 100 instances per benchmark because of high costs.Source: §4.1 · pp. 8, 10

4.2 GuardReasoner

• R-SFT fine-tunes LLaMA 3.2 1B and 3B, and LLaMA 3.1 8B on GuardReasonerTrain’s 127K samples and 460K reasoning steps.Training uses full-parameter fine-tuning for 3 epochs, a 2048-token cutoff, and a total batch size of 384.Source: §4.2 · p. 10
• HS-DPO trains on merged hard samples and self-generated data, using 23K, 14K, and 15K sample pairs for the 1B, 3B, and 8B models, respectively.Three R-SFT models trained on different subsets generate diverse hard samples; R-SFT loss is mixed into HS-DPO to alleviate model collapse.Source: §4.2 · p. 10

4.3 Prompts

• R-SFT training prompts ask an LLM to analyze inputs against ground truth, while guard-model examples pair prompts and target outputs with reasoning steps and labels.HS-DPO uses the same instruction, with correct reasoning and outcomes as positive data and incorrect ones as negative data.Source: §4.3 · p. 10
• Evaluation prompts instruct GPT-4o, GPT-4, and Claude 3.5 Sonnet to classify prompts and target responses; GuardReasoner is prompted to reason before classification.GPT-4o+CoT and GPT-4+CoT are instructed to think step by step; o1-preview and Gemini 1.5 Pro use prompts without a system-prompt setting.Source: §4.3 · p. 10

5.1 Safety Alignment of LLM

• Safety alignment targets helpfulness, harmlessness, and honesty [36], [37], using value-aligned data, instruction construction [47], [48], [49], unsafe-data filtering [50],, and SFT, RLHF, or DPO [52],,.Other work explores alignment without additional fine-tuning [55], [56] or more efficient alignment [57].Source: §5.1 · p. 11

5.2 Guard Models for LLMs

• GuardReasoner belongs to open-source guard models, alongside statistical methods and commercial APIs; existing guards face performance, explainability, and generalizability challenges that reasoning may address.Open-source examples include the LLaMA Guard and Aegis Guard series, WildGuard, and ShieldGemma.Source: §5.2 · p. 11

5.3 Reasoning Ability of LLM

• LLM reasoning research includes step-by-step prompting,, self-correction, self-critique, debate, and plan-and-solve [78], [79], [80], [81], [82], plus training and latent-space approaches [83], [84], [85].OpenAI’s o1 and later o1-like models from QwQ, QvQ, DeepSeek, and Kimi demonstrate continued interest in reasoning; OpenAI announced o3’s promising ARG-AGI performance.Source: §5.3 · p. 11

6 LIMITATIONS

• The study primarily measured whether humans agreed with predictions shown alongside CoT, a crude proxy that may not capture the nuances of model transparency.The authors note that this approach does not fully assess transparency.Source: §6 · pp. 11–12
• The authors did not fully investigate CoT–prediction alignment or broader explainability because evaluation requires substantial time and human effort, and plan a more detailed future study.Alignment evaluation could reveal discrepancies that affect perceived explainability.Source: §6 · p. 12

• GuardReasoner combines the GuardReasonerTrain dataset with R-SFT to unlock reasoning and HS-DPO to teach effective reasoning patterns.Source: §7 · p. 12
• Experiments across 13 benchmarks and 3 tasks show GuardReasoner is effective.Source: §7 · p. 12
• The authors release the data, code, and models at three scales, and plan to reduce unnecessary reasoning through model merging and lightweight strategies.The goal is greater efficiency while preserving accuracy and interpretability.Source: §7 · p. 12

• The guard model is designed to enhance LLM safety and mitigate potential risks and harmful societal impacts.Source: §8 · p. 12
• The paper aims to improve guard-model performance, explainability, and generalizability by teaching the model to reason.Source: §8 · p. 12
• The authors suggest companies could use this work as inspiration to build guard models for commercial use.Source: §8 · p. 12