Source-linked AI summary
Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, Ling Liu
TL;DR
The paper addresses the limited evidence on safety alignment for Large Reasoning Models and tests a simplified sequential production pipeline. It evaluates alignment with DirectRefusal and SafeChain, finding that safety can be restored while reasoning capability degrades, a trade-off named Safety Tax.
Problem
Safety alignment is extensively studied for LLMs, but its effects on the safety and reasoning capabilities of LRMs remain a research gap.
Method
The paper systematically studies a simplified two-stage LRM pipeline, applying reasoning training followed by safety alignment with DirectRefusal or SafeChain data.
Results
Safety alignment reduces harmful scores but lowers average reasoning accuracy, with SafeChain reducing it by 7.09% and DirectRefusal by 30.91%.
Takeaways & Limitations
The sequential pipeline exhibits a trade-off between safety and reasoning capability, named Safety Tax.
Takeaways & Limitations
The study uses supervised fine-tuning rather than reinforcement-learning safety alignment, so its observations may not generalize to RL-based alignment.
Abstract
from arXiv · showhide
Safety alignment is an important procedure before the official deployment of a Large Language Model (LLM). While safety alignment has been extensively studied for LLM, there is still a large research gap for Large Reasoning Models (LRMs) that equip with improved reasoning capability. We in this paper systematically examine a simplified pipeline for producing safety aligned LRMs. With our evaluation of various LRMs, we deliver two main findings: i) Safety alignment can be done upon the LRM to restore its safety capability. ii) Safety alignment leads to a degradation of the reasoning capability of LRMs. The two findings show that there exists a trade-off between reasoning and safety capability with the sequential LRM production pipeline. The discovered trade-off, which we name Safety Tax, should shed light on future endeavors of safety research on LRMs. As a by-product, we curate a dataset called DirectRefusal, which might serve as an alternative dataset for safety alignment. Our source code is available at https://github.com/git-disl/Safety-Tax.
1 Introduction
The paper asks whether safety alignment can improve Large Reasoning Models’ safety without reducing reasoning capability. Experiments with SafeChain and DirectRefusal find that safety improves, but reasoning may degrade, motivating the Safety Tax trade-off.
- The study asks whether safety alignment can improve LRM safety without downgrading reasoning capability.
- Experiments apply two safety datasets: DirectRefusal, constructed by the authors, and Chain-of-Thought safety data derived from SafeChain.DirectRefusal uses fixed, short thinking patterns and direct refusal answers.
- Safety alignment with either SafeChain or DirectRefusal significantly improves the safety of LRMs that initially contain high safety risk.
- Safety alignment with either dataset may degrade reasoning capability rather than refine it, contrary to a prior claim.The paper names this safety-reasoning trade-off Safety Tax.
- The paper identifies Safety Tax as a critical challenge for future safety research on LRMs.
2 Related Work
Prior work describes LRMs, safety alignment, and a two-stage production pipeline, while this paper presents its study as the first systematic identification of a safety-reasoning trade-off in LRMs.
- Safety alignment instructs LLMs and LRMs to refuse harmful user questions, using techniques such as supervised fine-tuning and reinforcement learning.
- The paper identifies a trade-off between safety and reasoning capability in LRMs and names it Safety Tax.
- LRMs are produced through reasoning-oriented training that elicits thinking trajectories and can be followed by safety alignment.
- The simplified pipeline considers DirectRefusal with short trajectories or SafeChain with long trajectories as alternative safety-alignment datasets.
3 Safety-aligned LRM Production Pipeline
The paper formalizes a simplified two-stage pipeline that first trains reasoning capability and then applies safety alignment before deployment. Its goal is an LRM that combines benchmark reasoning accuracy with refusal of harmful questions.
- The pipeline trains an LRM from a base instruction-fine-tuned model in two stages: reasoning training followed by safety alignment.After both stages, the safety-aligned LRM is ready for deployment.
- Reasoning training uses reasoning data, such as mathematical question-and-chain-of-thought answer pairs, to instruct the model to think before answering.
- Safety alignment uses a dataset containing harmful questions and refusal-oriented responses after reasoning training.
- The production goal is high standard-benchmark reasoning accuracy together with refusal of harmful questions raised by humans.
4 Experiments
Experiments evaluate safety alignment on several LRMs using harmfulness and benchmark reasoning accuracy. The results show that alignment restores safety but reduces reasoning capability, establishing a trade-off named Safety Tax.
- Main Results: Reasoning training increases reasoning accuracy by 22.64% on average compared with the base model.The result supports the usefulness of reasoning training for improving benchmark reasoning performance.
- Main Results: Reasoning training increases harmful score from 16.70 for the base model to 60.40 for the reason model, a 43.7% increase.The authors report that reasoning capability is acquired alongside compromised safety capability.
- Main Results: Safety alignment reduces average reasoning accuracy by 7.09% with SafeChain and 30.91% with DirectRefusal.The degradation is observed across three benchmarks with SafeChain, contradicting an earlier finding cited by the authors.
- System Overhead Analysis: SafeChain requires 1.47x training time and 1.03x extra GPU memory compared with DirectRefusal.The authors attribute the overhead to SafeChain’s longer thinking trajectories and judge DirectRefusal more system-efficient.
5 Limitation and Future Work
The study identifies important limits of its safety-alignment approach and points to improved training algorithms as future work.
- The study uses supervised fine-tuning rather than reinforcement learning for safety alignment.
- Whether these observations generalize to reinforcement-learning-based safety alignment remains open.
- DirectRefusal generally achieves better alignment performance than SafeChain but seriously downgrades reasoning capability.
- Future work should design training algorithms that better exploit DirectRefusal’s potential.
6 Conclusion
The paper studies a two-stage LRM pipeline in which reasoning training improves reasoning but reduces safety, while later safety alignment restores safety at a reasoning cost.
- Reasoning training increases the model’s reasoning ability but degrades its safety capability.
- Safety alignment after reasoning training can recover safety, but it degrades reasoning ability.
- The trade-off between reasoning and safety capability is named Safety Tax.
A DirectRefusal vs. SafeChain
The paper contrasts DirectRefusal’s fixed, short refusal reasoning pattern with SafeChain’s longer, question-dependent chain of thought.
- DirectRefusal uses a fixed thinking trajectory: “I should not answer this question!”
- SafeChain uses long thinking trajectories that vary depending on the question.
- SafeChain examples pair harmful prompts with extended reasoning before a refusal answer.
- DirectRefusal examples pair harmful prompts with a short refusal trajectory and refusal answer.