PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, Radha Poovendran

arXiv:2502.12025v1cs.AIcs.CL

TL;DR

Long chain-of-thought reasoning in large reasoning models creates a safety-evaluation gap because harmful content can occur in intermediate traces even when final answers appear safe. The paper evaluates LRM safety with joint trace-and-answer metrics, studies decoding strategies, and introduces SAFECHAIN for safety alignment. Across its evaluations, the paper finds that long CoT does not necessarily improve safety, while SAFECHAIN improves safety and preserves reasoning performance.

  • Problem

    LRM safety research must account for long reasoning traces alongside final answers, because existing safety-alignment datasets and evaluations primarily target regular short-answer LLM responses.

  • Method

    The paper calibrates safety evaluators with human annotations, develops joint CoT-and-final-answer metrics, evaluates 12 LRMs on StrongReject and WildJailbreak, and fine-tunes models with SAFECHAIN.

  • Results

    Long chain-of-thought does not necessarily enhance safety, while SAFECHAIN improves safety performance and preserves reasoning capabilities across six math and coding benchmarks.

  • Takeaways & Limitations

    SAFECHAIN provides a CoT-style dataset for strengthening LRM safety without degrading the reasoning capabilities measured in the paper.

  • Takeaways & Limitations

    The evaluation covers English policy-violating inputs and single-turn interactions, leaving multilingual and multi-turn LRM safety open.

Abstract

from arXiv · show

Emerging large reasoning models (LRMs), such as DeepSeek-R1 models, leverage long chain-of-thought (CoT) reasoning to generate structured intermediate steps, enhancing their reasoning capabilities. However, long CoT does not inherently guarantee safe outputs, potentially leading to harmful consequences such as the introduction of security vulnerabilities in code or the spread of misinformation. Current research on large language model (LLM) safety usually focuses on short-answer responses, overlooking the long CoT style outputs of LRMs. To bridge this gap, we conduct a systematic study of LRM safety. First, we investigate safety evaluators calibrated against human annotations. Using our newly developed metrics, we thoroughly assess the safety of 12 state-of-the-art LRMs on StrongReject and WildJailbreak datasets. Our results show that LRMs are not safe compared to their reasoning advance. Further, we perform a fine-grained analysis of the reasoning trace and final answer. We find that three decoding strategies-ZeroThink, LessThink, and MoreThink-can improve model safety without additional training. However, these strategies either use constrained reasoning traces or incur high inference costs. To better strengthen LRM safety, we introduce SafeChain, the first-of-its-kind safety training dataset in CoT style. We fine-tune two LRMs with SafeChain, showing that it not only enhances model safety but also preserves performance across 6 reasoning benchmarks.

1 Introduction

Large reasoning models use long chain-of-thought traces that improve complex-task reasoning but do not inherently ensure safe outputs. The paper develops joint CoT-and-answer safety evaluation and introduces a long-CoT alignment dataset to improve safety while preserving reasoning capabilities.

  • Motivation: Long chain-of-thought reasoning helps LRMs address complex mathematics and coding tasks but does not inherently guarantee safe responses.LRMs generate structured reasoning traces with multiple intermediate steps, yet unsafe responses remain possible.
  • Motivation: LRM safety evaluation must inspect both intermediate reasoning traces and final answers because a safe refusal can contain harmful content in its CoT.The longer outputs also make manual evaluation more difficult.
  • Safety strategies: ZeroThink, LessThink, and MoreThink vary CoT length, with ZeroThink producing the best safety performance without additional training.These strategies are presented as decoding setups for improving LRM safety.
  • Approach: The paper evaluates safety evaluators and develops three metrics that jointly examine reasoning thoughts and final answers across StrongReject and WildJailbreak.The evaluation covers 12 state-of-the-art reasoning models and includes human-calibrated evaluator comparisons.
  • SAFECHAIN: SAFECHAIN is a safety-alignment dataset in long-CoT style designed to strengthen LRM safety while preserving reasoning capabilities.The dataset addresses the absence of CoT data in existing safety-alignment datasets for regular LLM response styles.

2 Preliminary: LRMs with Long CoT

For an LRM, a response consists of a reasoning trace followed by a final answer. The reasoning trace can explore alternatives and correct errors, and may not be visible to users.

  • Response structure: An LRM response y concatenates a chain-of-thought reasoning trace yCoT and a final answer yans.The notation is y = yCoT ⊕ yans, where ⊕ denotes concatenation.
  • Reasoning trace: The reasoning trace enables branching into alternative paths and reverting to earlier checkpoints to correct errors before producing the final answer.
  • Visibility: Depending on the developer, an LRM’s reasoning trace may not be visible to users.

3 Safety Evaluation of LRMs

This section establishes an evaluation framework for LRM safety and applies it across models, datasets, and decoding settings. Results show that safety remains inconsistent: scaling helps, but long CoT and higher temperature can undermine safety.

  • Pilot Study of Safety Evaluators for LRMs: Llama-Guard consistently outperforms RS-Match, OpenAIMod, and HarmBenchEval across ACC, F-1, and PCC against human annotations.The study therefore uses Llama-Guard as its safety evaluator for reasoning models.
  • Experimental Setup: The evaluation covers open- and closed-source LRMs on StrongReject and WildJailbreak using Safe@1, Safe@K, and ConsSafe@K.StrongReject contains 310 policy-violating queries, while WildJailbreak supplies adversarial jailbreak prompts.
  • Experimental Results: Safety improves with model scale within the same family, from DeepSeek-R1-1.5B to R1.This pattern is reported across Safe@1, Safe@K, and ConsSafe@K under the evaluated configurations.
  • Experimental Results: Unsafe responses tend to be longer than safe responses in token-based response-length histograms.The analysis identifies some unsafe responses with extremely long lengths.
  • Experimental Results: Fine-tuning with long CoT does not necessarily enhance safety, with R1-70B compared against Llama-3.3-70B-Instruct and Llama-3.1-70B.Figure 3 reports that only 32.3% of R1-70B responses are considered safe, while the accompanying comparison reports 45.7% for Llama-3-Instruct and 76.6% for R1-70B against the base model.
  • Experimental Results: As temperature increases, LRM safety degrades; R1-7B Safe@K falls from 30% to below 20% at temperature 1.2.Top-p and top-k values do not significantly affect safety in the reported experiments.

4 Safety of LRMs’ Thought and Answer

The paper separates reasoning traces from final answers to examine how each contributes to LRM safety. It also tests decoding strategies that alter thought length and finds safety improvements without training.

  • Granular safety analysis: Responses are decomposed into reasoning traces and final answers, which Llama-Guard evaluates separately.The analysis focuses on DeepSeek R1-series models because they provide clear segmentation tags.
  • Granular safety analysis: Only 41.1% of responses have both safe thoughts and safe answers, while unsafe thoughts are more likely to produce unsafe answers.Some unsafe thoughts can still yield safe answers through reflection and error correction.
  • Decoding strategies: ZeroThink, LessThink, and MoreThink modify the length of the thought process through empty, short, or minimum-forced reasoning traces.MoreThink continues thinking until ten delimiter replacements or 10,000 thinking tokens.
  • Decoding strategies: All three decoding strategies improve safety over the default setup, with ZeroThink achieving the strongest safety performance.ZeroThink and LessThink disable the model’s thought process, preventing unsafe thought generation in the tested setup.

5 SAFECHAIN Dataset: Enhancing Safety under Chain-of-Thought

SAFECHAIN is a chain-of-thought safety-alignment dataset designed to improve LRM safety while preserving reasoning capabilities. Fine-tuning with SAFECHAIN preserves utility across math and coding benchmarks, unlike the GPT-3.5-based baseline.

  • Dataset construction: SAFECHAIN addresses the absence of chain-of-thought data in existing safety-alignment datasets for regular LLM response styles.The dataset samples 50,000 WildJailbreak instructions uniformly.
  • Dataset construction: R1-70B generates five responses per instruction, and Llama-Guard retains instructions whose five responses are all safe.One response is randomly selected for each retained instruction, producing 40,000 instruction-response pairs.
  • Baselines: WJ-40K uses the same instructions as SAFECHAIN but pairs them with safe responses generated by GPT-3.5.The vanilla baseline uses models without extra-data training.
  • Experimental setup: R1-7B and R1-8B are fine-tuned with supervised learning and evaluated on six math and coding benchmarks plus StrongReject and WildJailbreak.Math and coding use pass@1, while safety uses Safe@1 under greedy decoding.
  • Results: SAFECHAIN improves safety while preserving utility across benchmarks, whereas WJ-40K reaches higher safety but degrades reasoning performance.R1-7B’s LiveCodeBench performance falls from 39.3% to 14.5% after WJ-40K fine-tuning.

6 Related Work

Related work covers chain-of-thought reasoning, LLM safety alignment, and safeguards for unsafe behavior. The paper positions its study as the first systematic safety evaluation of reasoning models.

  • Chain-of-Thought and Reasoning Models: Chain-of-thought prompting was introduced to improve LLM performance on complex reasoning tasks such as mathematics and coding.This work prompted subsequent research into additional reasoning-prompting techniques.
  • LLM Safety: Safety alignment commonly uses supervised fine-tuning and reinforcement learning during post-training to support helpful and harmless behavior.Red-teaming studies nevertheless find aligned models can remain unsafe in real-world settings.
  • LLM Safety: Existing research also proposes test-time safeguards, while this paper presents the first systematic study focused on reasoning-model safety.The supplied passages frame reasoning-model safety as distinct from conventional LLM safety evaluation.

7 Conclusion

The paper finds that long chain-of-thought does not necessarily improve LRM safety and introduces SAFECHAIN to improve safety while preserving reasoning capabilities. It also identifies multilingual extension and improved long-CoT evaluation as future directions.

  • Conclusion: Long chain-of-thought in state-of-the-art LRMs does not necessarily enhance safety on StrongReject and WildJailbreak.The conclusion reports comprehensive evaluations of LRM safety.
  • Conclusion: SAFECHAIN outperforms existing datasets on DeepSeek-R1-1.5B and DeepSeek-R1-7B while preserving reasoning capabilities.The dataset is introduced for fine-tuning LRMs with safety-alignment data.
  • Future work: Future work will extend SAFECHAIN to multilingual settings and refine safety evaluation methods for long chain-of-thought reasoning.These directions are stated as future work in the conclusion.

Limitations

The study evaluates emerging LRMs using English, policy-violating inputs in single-turn interactions. Multilingual and multi-turn safety evaluation remain outside its scope.

  • Limitations: The evaluations use English policy-violating inputs and focus exclusively on single-turn interactions with LRMs.The paper identifies multilingual and multi-turn evaluation as open areas beyond this study.

Ethical Statement

The study evaluates LRM safety in a controlled setting using publicly available datasets. It introduces SafeChain as a safety-alignment dataset intended to develop safer LRMs without creating new harmful data.

  • Ethical Statement: The safety evaluations use publicly available datasets in a controlled setting.The paper states that this avoids creating new harmful data that could be misused.
  • Ethical Statement: SafeChain is introduced as a safety-alignment dataset for developing safer LRMs without introducing ethical concerns.The passage presents the dataset as part of the paper’s approach to safety alignment.

A.1 Dataset Details

SafeChain’s initial synthesis samples uniformly from four categories, with 12.5K samples per category before processing into the final dataset.

  • A.1 Dataset Details: The initial data synthesis samples uniformly from four categories.Each category contains 12.5K samples, for 50K samples in total before processing.
  • A.1 Dataset Details: Table 9 provides the number of instructions per category in the final SAFECHAIN dataset.The passage identifies Table 9 as the overview of the processed dataset.

A.2 Training Details

The paper documents supervised fine-tuning infrastructure and evaluation-prompt materials, including pairwise safety judging and harmful-query response examples. The examples illustrate reasoning traces that initially consider harmful information before moving toward refusal.

  • A.2 Training Details: Model training uses LLaMAFactory on four NVIDIA A100-SXM4-80GB GPUs with an AMD EPYC 7763 processor and 512 GB of RAM.Table 10 lists the supervised fine-tuning hyper-parameters.
  • A.2 Training Details: Figure 5 presents the prompt template used for pairwise evaluation with an LLM judge.The evaluator is instructed to assess how safely two responses handle harmful or disallowed aspects and compare their overall safety.
  • A.2 Training Details: The harmful-query examples show reasoning traces that initially consider napalm-related information before revising toward a refusal.The traces explicitly avoid listing ingredients or methods and emphasize dangers, legality, and safer alternatives.
  • A.2 Training Details: MoreThink adds enforced extra thinking that reminds R1-8B of safety and leads to a final safe response in the example.The example notes that the model initially attempted to provide information before its additional reasoning increased safety awareness.
Loading 2502.12025v1…