PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

Zhenyu Wu, Siyuan Chen, Changchun Yang, Jiaqi Dong, Min Zhou, Ali Almadan, Talal Hammad, Faisal Wahbo, Aminullah Tora, Mona Alshahrani, Xin Gao

arXiv:2608.24232v1cs.AI

TL;DR

Existing safety benchmarks largely omit unsafe reasoning traces and the evidence supporting safety judgments. TRACE addresses this gap with evidence-grounded annotations across prompts, reasoning traces, and final responses, then evaluates 18 guardrail models. Reasoning-trace safety judgment is harder than prompt or final-response judgment, while evidence attribution remains weak.

  • Problem

    Existing benchmarks focus primarily on prompts and final responses, typically provide binary labels without supporting evidence, and leave reasoning-trace safety largely unexplored.

  • Method

    TRACE annotates prompts, reasoning traces, and final responses with safety labels and source-text evidence, using four LRMs for generation and three additional LRMs for annotation.

  • Results

    18 guardrail models show that reasoning-trace safety judgment is the most challenging task, while evidence attribution remains difficult across all three pipeline components.

  • Takeaways & Limitations

    Guardrail models need to detect unsafe content and precisely localize supporting evidence throughout the LRM inference pipeline.

  • Takeaways & Limitations

    TRACE currently covers prompts in only Chinese and English, and some annotation noise may remain despite human verification.

Abstract

from arXiv · show

Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.

1 Introduction

TRACE addresses the limited evaluation of unsafe reasoning traces and unsupported binary safety judgments by annotating the full LRM inference pipeline with evidence. Evaluation shows that reasoning-trace safety judgment and evidence attribution remain difficult for current guardrail models.

  • Motivation: LRM reasoning traces can contain prohibited strategies even when final responses refuse harmful requests.A harmful airport-security prompt produced unsafe concealment reasoning alongside a safe refusal.
  • Research gap: Existing guardrail models primarily assess prompts and final responses, leaving their effectiveness on unsafe reasoning traces largely unexplored.
  • Research gap: Existing benchmarks mainly provide binary safety labels for prompts or final responses and omit evidence annotations that justify judgments.This prevents evaluation of whether models can identify and attribute the evidence underlying their decisions.
  • TRACE: TRACE provides evidence-grounded safety annotations across prompts, reasoning traces, and final responses for holistic evaluation of judgment correctness and evidence attribution.The benchmark covers safe and unsafe prompts from two datasets, nine risk categories, and ten attack strategies.
  • Findings: 14 out of 18 guardrail models perform best on prompt safety judgment, followed by final-response judgment, while reasoning-trace judgment is most challenging.YuFeng-XGuard-8B scores 88.27%, 86.11%, and 84.26% F1 on prompts, final responses, and reasoning traces, respectively.
  • Findings: Evidence attribution is substantially weaker: YuFeng-XGuard-8B reaches TokenF1 scores of only 11.68%, 13.71%, and 14.88% across prompts, final responses, and reasoning traces.

2 Related Work

Related benchmarks focus mainly on prompts and final responses, whereas TRACE extends safety evaluation to reasoning traces and evidence attribution across the LRM inference pipeline. Its construction combines multi-model generation and annotation with verification of extracted evidence.

  • Benchmark construction: TRACE construction curates safe and unsafe prompts, generates four LRM traces and responses per prompt, and uses three additional LRMs for annotation and evidence extraction.
  • Benchmark construction: Extracted evidence is retained only when majority-aligned annotators provide it and it is verified as a continuous source-text substring; unverifiable samples receive human review.
  • Scope of prior work: Unlike conventional LLMs, LRMs generate detailed reasoning traces alongside final responses, creating an additional safety-evaluation target.Reasoning traces may contain unsafe content even when final responses are safe.
  • Scope of prior work: Existing guardrail models primarily detect unsafe content in prompts or final responses, while reasoning-trace detection remains largely unexplored.
  • TRACE: TRACE evaluates safety judgment correctness and evidence attribution across prompts, reasoning traces, and final responses.

3 The TRACE Benchmark

TRACE constructs an evidence-grounded benchmark spanning prompts, LRM reasoning traces, and final responses, with safety labels and supporting evidence. It combines diverse prompts and LRM outputs to evaluate guardrail judgment correctness and evidence attribution across the inference pipeline.

  • Prompt Curation: TRACE curates safe and unsafe prompts from S-Eval and WildChat, covering nine risk categories, ten attack strategies, and two languages.Stratified sampling across 20 category-language groups yields 1,993 prompts.
  • LRM Generation: Four LRMs generate reasoning traces and final responses for each prompt, combining safety-aligned and abliterated model variants.The four models are Qwen3-8B, Gemma-4-E4B, and their abliterated counterparts.
  • Safety Annotation: Three additional LRMs independently assign safe or unsafe labels and extract verbatim evidence for prompts, reasoning traces, and final responses.The annotation instruction requires labels and supporting excerpts for all three components.
  • Safety Annotation: TRACE retains majority-aligned evidence only when it is verifiable as a continuous substring of the corresponding source text, escalating unverifiable cases to humans.This procedure produces the final evidence-grounded annotations.
  • Dataset Statistics: After removing samples with missing traces or responses, the benchmark contains 5,000 valid prompt-trace-response triples.The generated traces are 54% safe and 46% unsafe, while final responses are 55% safe and 45% unsafe.
  • Evaluation: Guardrail models are evaluated on safety judgment correctness and evidence attribution accuracy for prompts, reasoning traces, and final responses.Safety judgment uses metrics including FPR, FNR, and F1-score; evidence attribution is evaluated separately.

4 Experiments

TRACE evaluates 18 guardrail models across the LRM inference pipeline, finding that reasoning-trace safety judgment is the most challenging stage and evidence attribution remains weak.

  • Experimental setup: 18 guardrail models are evaluated across prompt, reasoning-trace, and final-response safety judgment, with evidence attribution measured separately.The experiments also examine risk-category classification and error trade-offs across attack strategies and pipeline stages.
  • Safety judgment: 84.26% F1-score makes YuFeng-XGuard-8B the strongest reported model for reasoning-trace safety judgment, ahead of LlamaGuard-1-7B and ShieldGemma-9B.It surpasses the two comparison models by 59.92 and 25.24 points, respectively.
  • Safety judgment: 66.31% average F1-score on reasoning-trace safety judgment is lower than 75.75% for prompts and 70.53% for final responses.Fourteen of 18 models perform best on prompt safety judgment, followed by final responses and then reasoning traces.
  • Language effects: All 18 models achieve higher reasoning-trace safety-judgment F1-scores on TRACE-ZH than on TRACE-EN.Seventeen of 18 models also score higher on Chinese prompts and final responses.
  • Evidence attribution: YuFeng-XGuard-8B achieves TokenF1 scores of 11.68%, 14.88%, and 13.71% for evidence attribution on prompts, reasoning traces, and final responses.It outperforms Octopus-14B by 2.15, 0.32, and 0.46 points across those tasks.
  • Error analysis: 13.65% FNR and 15.77% FPR make YuFeng-XGuard-8B the most balanced model for reasoning-trace safety judgment.PolyGuard-8B is most balanced for prompts, with 9.58% FNR and 9.11% FPR; ShieldGemma and LlamaGuard-2 show higher FPR, while several others show higher FNR.
  • Attack robustness: Instruction Encryption attacks cause the most severe degradation in safety-judgment performance across all three pipeline stages.Even YuFeng-XGuard-8B and PolyGuard-8B fail to provide reliable defense under this strategy.

5 Conclusion

TRACE evaluates guardrail safety judgment and evidence attribution across prompts, reasoning traces, and final responses. Its results identify reasoning traces as the hardest stage and show that current models struggle to localize supporting evidence.

  • TRACE evaluates guardrail models for both safety judgment correctness and evidence attribution accuracy across the entire LRM inference pipeline.
  • Reasoning-trace safety judgment is substantially more challenging than prompt or final-response safety judgment.
  • Current guardrail models struggle to accurately extract supporting evidence for their safety judgments.

Limitations

TRACE’s multilingual coverage is limited to Chinese and English, constraining evaluation of guardrail safety judgment across multilingual content.

  • TRACE currently covers prompts in only two languages, Chinese and English.The authors state that future work will expand the benchmark to a broader range of languages.
  • This language scope limits TRACE’s ability to evaluate guardrail safety judgment across multilingual content.

Ethical Considerations

TRACE uses licensed public datasets and abliterated LRMs to construct a broader safety-evaluation benchmark. Because the benchmark contains unsafe content, its use is restricted to safety evaluation research.

  • Data and licensing: TRACE is constructed from S-Eval and WildChat and released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
  • Responsible use: TRACE contains unsafe prompts, reasoning traces, and final responses that could potentially be misused for harmful model training or fine-tuning.
  • Responsible use: TRACE is intended solely for safety evaluation research, and uses facilitating harmful-content generation or dissemination are explicitly prohibited.
  • Benchmark construction: Abliterated variants of safety-aligned LRMs are used to generate broader safety-relevant behaviors for benchmark construction.The paper states that abliteration is used solely to build a more comprehensive evaluation benchmark and does not endorse harmful use.

A Appendix: Additional Experimental Details and Discussions

The appendix details the evaluated guardrail models, their supported safety-judgment capabilities, and the experimental setup used for TRACE evaluation.

  • Evaluated Guardrail Models: The study evaluates 18 representative guardrail models on TRACE.Table 5 records whether models provide explanations and support custom risk categories.
  • Evaluated Guardrail Models: The evaluated models include classifiers built on Llama, Gemma-2, and Qwen3 that judge prompts or LLM-generated responses for safety.The models use predefined or user-defined safety risk categories depending on the model.
  • Evaluation Capabilities: Only Octopus, GPTSafeGuard, and YuFeng-XGuard generate explanations that can be evaluated for evidence identification.The evaluation tests whether explanations correctly identify source-text evidence supporting safety judgments.
  • Evaluation Capabilities: NemotronGuard, PolyGuard, GPTSafeGuard, and YuFeng-XGuard support prompt risk-category classification using user-defined categories.These four models are evaluated on whether they correctly identify prompt risk categories.
  • Experimental Setup: Experiments used one node with 2 × NVIDIA A100 80GB PCIe GPUs, temperature 0.7 for generation, and temperature 0 for annotation and evaluation.Generation allowed up to 6,000 tokens, while annotation and evaluation allowed up to 1,024 tokens.

A.3 Additional Evaluation Metrics

The appendix supplements safety-judgment evaluation with FPR, FNR, F1-score, Precision, and Recall across prompts, reasoning traces, and final responses.

  • Metrics: The evaluation reports False Positive Rate, False Negative Rate, F1-score, Precision, and Recall for guardrail-model safety judgments.These metrics provide a broader assessment than a single safety label accuracy measure.
  • Evaluation Target: For each TRACE instance, a guardrail model predicts a safety label and supporting evidence for the prompt, reasoning trace, and final response.Ground-truth safety labels are used to assess the predictions for each component.
  • Precision: Precision measures the proportion of content classified as unsafe that is truly unsafe, while high Precision indicates few false alarms.This metric reflects the extent of over-blocking safe content.
  • Precision: Precision must be interpreted with Recall because conservative flagging can produce high Precision while detecting only a small subset of unsafe content.The trade-off prevents Precision alone from representing overall unsafe-content detection.
  • Recall: Recall measures the proportion of truly unsafe content correctly classified as unsafe, indicating the effectiveness of unsafe-content detection.Higher Recall reduces the risk of exposing unsafe content to users.

A.4 Additional Experimental Results

Additional results examine base-model choice, scale, training data, precision–recall trade-offs, encrypted attacks, and generator-family effects across TRACE’s inference pipeline.

  • Model Architecture and Scale: Qwen-derived guardrail models generally outperform comparable Llama-, Mistral-, and Gemma-derived models, while larger models within a family improve detection performance.PolyGuard-8B consistently outperforms PolyGuard-0.5B across the pipeline.
  • Training Data Scale: YuFeng-XGuard-8B outperforms Qwen3Guard-8B despite sharing its Qwen3-8B base model, suggesting training-data scale may improve safety-judgment accuracy.The evidence remains suggestive rather than causal because data quality, annotation strategy, and fine-tuning details may also contribute.
  • Precision–Recall Trade-offs: High recall but lower precision causes ShieldGemma and LlamaGuard-2 to over-block safe content, whereas higher-precision models can miss unsafe content.The two trade-offs create different practical risks across evaluation settings.
  • Precision–Recall Trade-offs: YuFeng-XGuard-8B achieves the most balanced performance on reasoning traces with precision 82.26% and recall 86.35%, and on final responses with precision 83.05% and recall 89.40%.For prompts, PolyGuard-8B reaches precision 91.93% and recall 90.42%.
  • Instruction Encryption Attacks: Under TRACE-(IE), LlamaGuard-2 records F1-scores of 47.46%, 54.70%, and 41.12% for prompts, reasoning traces, and final responses, respectively.It outperforms PolyGuard-8B by 42.86, 39.45, and 11.23 percentage points on those stages.
  • Instruction Encryption Attacks: Qwen3Guard-0.6B and Qwen3Guard-4B obtain 0.00% F1-score on reasoning-trace safety judgment under instruction-encryption attacks.The results indicate vulnerability to IE attacks across the LRM inference pipeline.
  • Generator-Family Effects: Guardrail performance varies by whether traces and responses come from Qwen3-series or Gemma-4-series LRMs, although several models differ by no more than two percentage points.YuFeng-XGuard-8B leads both subsets, reaching 84.93% and 86.70% on TRACE-Q and 82.99% and 85.01% on TRACE-G for traces and responses.

A.5 Additional Discussion on Case Studies

TRACE case studies show distinct guardrail failures in safety classification and evidence localization, including errors driven by over-refusal and semantic obfuscation.

  • YuFeng-XGuard-8B labels unsafe prompts, reasoning traces, and final responses as safe and fails to recognize or localize unsafe context.
  • Octopus correctly labels all three unsafe components but fails to identify the supporting evidence for its judgments.
  • GPTSafeGuard labels all three safe components as unsafe, primarily because it over-refuses benign financial-information queries.The model conflates general investment-product overviews with harmful, unqualified financial advice.
  • GPTSafeGuard labels all three unsafe components as safe when semantic obfuscation in creative writing makes crude and sexually suggestive content appear benign.
  • The benchmark organizes unsafe prompts through risk categories and attack strategies, including Ethics and Morality and Cybersecurity examples.The supplied examples include JailBreak-style instruction overriding and requests to design a rapidly spreading information-leakage worm.

D Content Safety Annotation System

TRACE uses a content safety annotation system to validate benchmark annotations through human re-annotation of sampled instances containing prompts, reasoning traces, and final responses.

  • The benchmark randomly selects 1,000 instances for human re-annotation using the content safety annotation system.
  • Each instance presents the user prompt together with the LRM-generated reasoning trace and final response for human annotation.
  • The annotation interface is identified as the Content Safety Annotation System and includes sample review and annotation-statistics controls.
Loading 2608.24232v1…