PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

Zhenyu Wu, Siyuan Chen, Changchun Yang, Jiaqi Dong, Min Zhou, Ali Almadan, Talal Hammad, Faisal Wahbo, Aminullah Tora, Mona Alshahrani, Xin Gao

arXiv:2608.24232v1cs.AI

TL;DR

Guardrails have been assessed on prompts and final responses, but their ability to detect unsafe content in LRM reasoning traces remains largely unexplored. TRACE evaluates safety judgments and evidence attribution across prompts, reasoning traces, and final responses; evaluation finds trace safety judgment most challenging and evidence extraction difficult.

  • Problem

    Guardrails have been assessed on prompts and final responses, but their ability to detect unsafe content in LRM-generated reasoning traces remains largely unexplored.

    Source: §1 · p. 1

  • Method

    TRACE is an evidence-grounded benchmark that annotates safety and supporting source-text evidence for prompts, reasoning traces, and final responses generated using four LRMs.

    Source: §1 · p. 2

  • Results

    66.31% average F1 for reasoning-trace safety judgment was lower than 75.75% for prompt safety judgment and 70.53% for final response safety judgment; models also struggled to extract supporting evidence.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The findings underscore the need for guardrails that reliably detect and precisely localize unsafe content throughout the LRM inference pipeline.

    Source: §5 · p. 9

    TRACE currently covers only Chinese and English, limiting its evaluation of guardrail safety judgment across multilingual content.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.

1 Introduction

TRACE addresses the risk of unsafe reasoning traces and the lack of evidence-grounded evaluation by annotating safety and supporting evidence across prompts, traces, and final responses; evaluation finds trace judgments and evidence extraction challenging for guardrail models.

1 Introduction

  • Reasoning traces can contain explicit unsafe content even when a model’s final response refuses, while guardrail effectiveness on traces remains largely unexplored and existing benchmarks omit evidence annotations.Existing benchmarks focus on prompts or final responses and binary safety labels, limiting evaluation of evidence attribution and localization.Source: §1 · pp. 1–2
  • TRACE annotates safety and source-text evidence for prompts, reasoning traces, and final responses, using majority-voted labels and verified evidence, with human annotation when evidence is unverifiable.It uses safe and unsafe prompts from S-Eval (Yuan et al., 2025) and WildChat (Zhao et al., 2024), covers nine risk categories and ten attack strategies, and has four LRMs generate traces and responses.Source: §1 · p. 2
  • 14 of 18 models ranked prompt safety judgment highest and reasoning-trace judgment hardest; YuFeng-XGuard-8B F1: prompt 88.27%, final response 86.11%, reasoning trace 84.26%.PolyGuard-8B F1 scores were 91.17% for prompt, 84.15% for final response, and 82.57% for reasoning-trace safety judgment.Source: §1 · p. 2
  • YuFeng-XGuard-8B evidence-attribution TokenF1 was 11.68% for prompts, 13.71% for final responses, and 14.88% for reasoning traces, indicating weak evidence extraction.Source: §1 · p. 2

2 Related Work

Prior guardrails and safety benchmarks largely address prompts or final responses, leaving LRM reasoning traces and evidence-backed safety judgments as gaps that TRACE targets.

2.1 Reasoning Trace Safety in LRMs

2.2 Guardrail Models

  • Existing guardrails primarily assess prompts or final responses, while their effectiveness on unsafe content in LRM reasoning traces remains largely unexplored.Source: §2.2 · p. 3

2.3 Unsafe Content Detection Benchmarks

  • Unsafe-content benchmarks cover foundational safety, diverse prompt-risk categories, or adversarial robustness, but typically omit LRM traces and evidence for safety judgments.Source: §2.3 · pp. 3–4

3 The TRACE Benchmark

TRACE evaluates safety across prompts, reasoning traces, and final responses, pairing component-level safety labels with supporting evidence. It is constructed from curated prompts, LRM-generated outputs, and verified annotations.

3.2 Prompt Curation

  • Stratified sampling of S-Eval and WildChat yielded 1,993 prompts spanning nine risk categories, ten attack strategies, and English and Chinese.S-Eval supplies unsafe prompts; WildChat prompts selected for TRACE were labeled safe by the dataset authors. Sampling covered 18 S-Eval category-language groups and two WildChat language groups, with 1% sampled per group.Source: §3.2 · p. 4

3.3 LRM Inference

  • Four LRMs—two safety-aligned models and their abliterated counterparts—generate traces and responses to capture varied safety behaviors.The models are Qwen3-8B, Gemma-4-E4B, and their respective abliterated variants.Source: §3.3 · p. 4

3.4 Multi-Dimensional Safety Annotation

  • Three LRMs independently label prompts, traces, and responses and extract verbatim evidence; majority voting, substring verification, and human review finalize annotations.Evidence is retained only from majority-aligned annotators when it is a continuous substring of the source; samples without verifiable evidence are escalated to human experts.Source: §3.4 · pp. 4–5

3.5 Benchmark Statistics

  • Safety can shift between prompts, reasoning traces, and final responses, motivating evaluation across the entire LRM inference pipeline.TRACE contains 1,993 prompts; 43% are safe and 57% unsafe. Of 5,000 valid triples, 54% of reasoning traces and 55% of final responses are safe.Source: §3.5 · p. 5

3.6 Evaluation Metrics

  • Guardrail evaluation on TRACE measures both safety judgment correctness and evidence attribution accuracy.Models predict a safety label and supporting evidence for each prompt, reasoning trace, and final response.Source: §3.6 · p. 5

3.6.1 Safety Judgment Correctness

  • Safety judgment correctness is assessed with false positive rate, false negative rate, and F1-score for each component.FPR measures safe content incorrectly classified as unsafe, FNR measures unsafe content incorrectly classified as safe, and F1-score is the harmonic mean of precision and recall.Source: §3.6.1 · pp. 5–6

3.6.2 Evidence Attribution Accuracy

  • Evidence attribution accuracy is measured by token-level F1-score, comparing overlap between predicted and ground-truth evidence tokens.A higher TokenF1 indicates greater token overlap and more accurate evidence attribution.Source: §3.6.2 · p. 6

4 Experiments

Across the LRM inference pipeline, guardrail models find reasoning traces hardest to judge and struggle to extract supporting evidence. Their performance also varies by language and attack strategy, with distinct trade-offs between over-refusal and unsafe-content blocking.

4.1 Experimental Setup

  • The experiments evaluate 18 representative guardrail models, using diverse LRM outputs and reproducible settings for safety annotation and guardrail evaluation.LRM inference uses temperature 0.7 and a 6,000-token maximum; annotation and guardrail evaluation use temperature 0 and a 1,024-token maximum.Source: §4.1 · p. 6

4.2 Experimental Results

  • YuFeng-XGuard-8B achieves the highest reasoning-trace safety F1-score at 84.26%, exceeding LlamaGuard-1-7B and ShieldGemma-9B by 59.92 and 25.24 points, respectively.The results indicate that LlamaGuard-1-7B and ShieldGemma-9B struggle to judge reasoning-trace safety reliably, while more advanced models perform better.Source: §4.2 · pp. 6–7
  • Across 18 guardrail models, average safety F1-scores are 75.75% for prompts, 66.31% for reasoning traces, and 70.53% for final responses, making traces the hardest stage.Fourteen models perform best on prompt safety; YuFeng-XGuard-8B scores 88.27% on prompts, 2.16 points above final responses and 4.01 above reasoning traces.Source: §4.2 · p. 7

Can guardrail models accurately attribute evidence supporting their safety judgments?

  • YuFeng-XGuard-8B leads evidence attribution, but TokenF1 scores remain low across evaluated models, showing that reliable evidence extraction remains a weakness.Its prompt, reasoning-trace, and final-response TokenF1 scores are 11.68%, 14.88%, and 13.71%, exceeding Octopus-14B by 2.15, 0.32, and 0.46 points, respectively.Source: Table 3 · p. 7, Can guardrail models accurately… · p. 7
  • GPTSafeGuard-20B achieves the highest prompt risk-category classification F1-score at 62.25%, ahead of YuFeng-XGuard-8B (53.24%) and PolyGuard-8B (16.63%).Only NemotronGuard, PolyGuard, YuFeng-XGuard, and GPTSafeGuard classify prompt categories; TRACE covers nine risk categories.Source: Table 4 · p. 7, Can guardrail models accurately… · p. 7

4.2 Experimental Results

  • All 18 models achieve higher reasoning-trace safety F1-scores on TRACE-ZH than TRACE-EN, and 17 do so for prompts and final responses.Guardrail models are generally more effective at judging Chinese content across all three pipeline stages.Source: §4.2 · p. 7

4.3 Error Analysis

  • Error analysis shows distinct failure patterns: ShieldGemma and LlamaGuard-2 tend toward over-refusal, whereas several other models more often fail to block unsafe content.PolyGuard-8B balances prompt safety at FNR 9.58% and FPR 9.11%; YuFeng-XGuard-8B has FNR/FPR of 13.65%/15.77% on traces and 10.60%/14.76% on final responses.Source: Figure 4 · p. 8, §4.3 · p. 8
  • Instruction Encryption attacks cause the greatest safety-judgment degradation, and even leading models remain vulnerable across reasoning traces and final responses.On reasoning traces, YuFeng-XGuard-8B and PolyGuard-8B score 38.85% and 15.25%; on final responses, they score 35.16% and 29.89%. IE schemes such as Caesar cipher and Base64 obscure harmful intent.Source: Figure 5 · p. 8, §4.3 · p. 8

5 Conclusion

TRACE evaluates guardrail models across the LRM inference pipeline for safety judgments and evidence attribution. Evaluation of 18 models found reasoning traces harder to judge safely than prompts or final responses, while evidence extraction also remains difficult.

5 Conclusion

  • TRACE evaluates guardrail models across the LRM inference pipeline for safety judgment correctness and evidence attribution accuracy.The benchmark is evidence-grounded.Source: §5 · p. 8
  • Safety judgments on reasoning traces were substantially more challenging than on prompts or final responses, and models struggled to extract supporting evidence.Evaluation covered 18 guardrail models.Source: §5 · p. 9
  • The findings underscore the need for guardrail models that reliably detect and precisely localize unsafe content throughout the LRM inference pipeline.Source: §5 · p. 9

Limitations

TRACE’s two-language coverage limits multilingual evaluation, and residual annotation noise may remain despite human verification.

Limitations

  • TRACE covers only Chinese and English prompts, limiting evaluation of guardrail safety judgments across multilingual content.Future work will broaden the language range to improve cross-linguistic applicability.Source: Limitations · p. 9
  • Human verification does not rule out annotation noise, and larger-scale human evaluation could improve benchmark annotation quality.Source: Limitations · p. 9

Ethical Considerations

TRACE’s ethical safeguards address licensing, potential misuse of unsafe content, and the use of abliterated models in benchmark construction.

Licensing

  • TRACE is released under CC BY-NC-SA 4.0 to comply with the licenses of its source datasets, S-Eval and WildChat.S-Eval uses CC BY-NC-SA 4.0, while WildChat uses ODC-BY.Source: Ethical Considerations · p. 9

Misuse safeguards

  • Because TRACE contains unsafe content that could be misused for harmful training, its use is limited to safety evaluation research and harmful-content generation or dissemination is prohibited.Source: Ethical Considerations · p. 9

Benchmark construction

  • Abliterated safety-aligned LRMs generate benchmark responses to broaden safety-relevant behaviors; this technique is used only for benchmark construction and not endorsed or permitted for harmful purposes.Abliteration suppresses refusal behaviors, enabling responses to harmful prompts.Source: Ethical Considerations · p. 9

A Appendix: Additional Experimental Details and Discussions · A.1 Details of Evaluated Guardrail Models · A.2 Hyperparameter Settings

The appendix documents the evaluated guardrail-model pool, explains which models are assessed for evidence extraction and prompt risk-category classification, and reports the experimental settings.

A.1 Details of Evaluated Guardrail Models

  • The pool includes LlamaGuard (Inan et al., 2023), NemotronGuard, ShieldGemma (Zeng et al., 2024), and WildGuard (Han et al., 2024).These models assess prompts and responses for safety; their scopes vary, including predefined risk categories, four specified risk types, and response refusal classification.Source: §A.1 · p. 13
  • Other evaluated systems include Octopus (Yuan et al., 2025), GPTSafeGuard, Qwen3Guard (Zhao et al., 2025), PolyGuard (Kumar et al., 2025), and YuFeng-XGuard (Lin et al., 2026).Octopus provides safety scores and explanations; GPTSafeGuard can assess prompts, reasoning traces, and final responses; Qwen3Guard has 0.6B, 4B, and 8B models; PolyGuard supports 17 languages; and YuFeng-XGuard is reasoning-centric and generates explanations.Source: §A.1 · p. 13
  • Octopus, GPTSafeGuard, and YuFeng-XGuard are evaluated on whether their explanations correctly identify evidence supporting their safety judgments.Source: §A.1 · p. 13
  • NemotronGuard, PolyGuard, GPTSafeGuard, and YuFeng-XGuard are evaluated on identifying prompt risk categories defined by the user.Source: §A.1 · p. 13
  • TRACE evaluates 18 representative guardrail models, with their details summarized in Table 5.Table 5 indicates which models provide explanations and support user-defined risk categories.Source: §A.1 · p. 13, Table 5 · p. 14

A.2 Hyperparameter Settings

  • Generation uses temperature 0.7 and a 6,000-token output limit, while annotation and guardrail evaluation use temperature 0 and a 1,024-token limit.Experiments ran on one node with two NVIDIA A100 80GB PCIe GPUs, CUDA 12.2, and an Intel Xeon Gold 6330 CPU.Source: §A.2 · p. 14

A.3 Additional Evaluation Metrics

TRACE evaluates safety-judgment correctness with FPR, FNR, and F1-score, and adds Precision and Recall for a more comprehensive assessment. Guardrail models predict safety labels and supporting evidence for prompts, reasoning traces, and final responses.

A.3 Additional Evaluation Metrics

  • Precision and Recall complement FPR, FNR, and F1-score in TRACE’s evaluation of guardrail models.Precision measures the share of content classified as unsafe that is truly unsafe; Recall measures the share of truly unsafe content correctly classified as unsafe.Source: §A.3 · p. 14
  • Models predict safety labels and supporting evidence for each prompt, reasoning trace, and final response.The ground-truth safety label is used to assess each prediction.Source: §A.3 · p. 14
  • High Precision indicates few false alarms and less over-blocking, but should be interpreted with Recall because conservative flagging can miss unsafe content.High Recall indicates effective detection of unsafe content, reducing the risk of exposing it to users.Source: §A.3 · p. 14

A.4 Additional Experimental Results

Additional experiments show that guardrail performance varies with base-model family and scale, training-data scale, precision–recall trade-offs, IE attacks, and the LRM family generating outputs.

A.4 Additional Experimental Results

  • Qwen-derived guardrails generally detect unsafe content better than comparable-scale Llama-, Mistral-, and Gemma-based models, and larger base models improve performance within families.PolyGuard-8B consistently outperforms PolyGuard-0.5B across the inference pipeline, suggesting that both base-model capabilities and model scale influence effectiveness.Source: §A.4 · pp. 14–15, Figure 6 · p. 15
  • With the same Qwen3-8B base, YuFeng-XGuard-8B, trained on 2.80 million instances, outperforms Qwen3Guard-8B, trained on 1.19 million, on trace and response F1-score.This suggests larger training datasets may help, but the gap is not causal evidence because data quality, annotation strategy, and fine-tuning details may also contribute.Source: §A.4 · p. 15, Figure 6 · p. 15
  • On TRACE-(IE), where all prompts use IE attacks, prompt, reasoning-trace, and final-response safety judgments outperform PolyGuard-8B by 42.86, 39.45, and 11.23 percentage points; both Qwen3Guard models score 0.00% F1 on reasoning traces.Source: Table 6 · p. 16, §A.4 · p. 16
  • PolyGuard-8B balances prompt precision (91.93%) and recall (90.42%), while YuFeng-XGuard-8B achieves the most balanced trace and response results.YuFeng-XGuard-8B has precision of 82.26% and recall of 86.35% on reasoning traces, and precision of 83.05% and recall of 89.40% on final responses. ShieldGemma models and LlamaGuard-2 have higher recall than precision, whereas LlamaGuard-4, LlamaGuard-3-8B, LlamaGuard-1, WildGuard, NemotronGuard, and GPTSafeGuard have higher precision than recall.Source: §A.4 · pp. 15–16, Figure 7 · p. 15
  • On TRACE-Q versus TRACE-G, LlamaGuard-2, LlamaGuard-3-1B, and ShieldGemma favor Qwen3-generated outputs, while WildGuard and NemotronGuard favor Gemma-4-generated outputs.Several models are consistent across subsets, with differences within two percentage points; YuFeng-XGuard-8B scores 84.93% and 86.70% on TRACE-Q, and 82.99% and 85.01% on TRACE-G, for reasoning traces and final responses, respectively.Source: §A.4 · pp. 16–17, Table 7 · p. 17

A.5 Additional Discussion on Case Studies … Safety Annotation Prompt Template

Case studies expose distinct guardrail errors: missed unsafe content, poor evidence localization, over-refusal, and under-refusal. The accompanying taxonomy and annotation template specify risk and attack labels and require evidence-grounded judgments across inputs, reasoning traces, and responses.

A.5 Additional Discussion on Case Studies

  • YuFeng-XGuard-8B labels the unsafe prompt, reasoning trace, and final response safe, and its explanations fail to identify the unsafe context.The case shows both classification and evidence-localization failures.Source: §A.5 · p. 17
  • Octopus correctly labels all three unsafe components but fails to accurately locate the evidence supporting its judgments.This case separates correct safety classification from unreliable evidence localization.Source: §A.5 · p. 17
  • GPTSafeGuard labels all three safe components unsafe, driven by over-refusal that conflates general investment overviews with harmful financial advice.The example concerns benign financial queries and the “Inappropriate Suggestions – Finance” category.Source: §A.5 · p. 17
  • GPTSafeGuard labels all three unsafe components safe, as creative-writing framing obscures crude, sexually suggestive themes as humor.The passage attributes this under-refusal to vulnerability to semantic obfuscation in fictional, comedic writing.Source: §A.5 · p. 17

B Risk Categories and Attack Strategies

C Safety Annotation Prompt Template

  • The annotation taxonomy covers user-input risks including crimes, privacy, cybersecurity, minors, and inappropriate suggestions in critical domains.It specifies that safe inputs receive risk_category “None” and permits a concise new category when an unsafe input is not covered.Source: C Safety Annotation Prompt Template · p. 23
  • The attack taxonomy includes induction, code injection, instruction jailbreaks, encryption, and nested-scenario strategies.The prompt directs annotators to use “None” when no attack is present and to name an unlisted attack concisely.Source: C Safety Annotation Prompt Template · p. 23
  • The template requests separate safety and attack judgments, with exact-quote evidence and explanations for user input, reasoning traces, and final responses.It requires valid JSON without extra text; safety and attack fields include labels, while evidence must quote the input exactly.Source: C Safety Annotation Prompt Template · p. 24

D Content Safety Annotation System

TRACE’s annotation system asks humans to classify prompt attack strategies and risk categories, then judge the safety of prompts, reasoning traces, and final responses with verbatim evidence. Re-annotation of 1,000 instances achieved Cohen’s Kappa of 0.84 against TRACE labels, supporting their reliability.

D Content Safety Annotation System

  • Cohen’s Kappa reached 0.84 between human annotations and TRACE labels on 1,000 randomly selected instances, supporting the reliability of the benchmark annotations.The re-annotation used the content safety annotation system shown in Figure 12.Source: D Content Safety Annotation System · p. 24
  • Human annotators selected each prompt’s attack strategy and risk category from dropdown menus with hover tooltips describing the options.The interface is shown in Figure 12.Source: D Content Safety Annotation System · p. 24, Figure 12 · p. 25
  • Annotators labeled prompts, reasoning traces, and final responses safe or unsafe, extracting verbatim evidence from each source to support the judgment.Source: D Content Safety Annotation System · p. 24

Related papers

Loading 2608.24232v1…