Source-linked AI summary
STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Yanqing Liu, Jieru Mei, Brian R. Bartoldson, Bhavya Kailkhura, Cihang Xie
TL;DR
Large reasoning models face safety risks from harmful prompts and amplified unsafe outputs, while alignment training can degrade general performance. STAR-1 constructs a diverse 1K dataset with policy-grounded deliberative reasoning and rigorous filtering; fine-tuning improves safety substantially while minimally affecting reasoning.
Problem
Large reasoning models face harmful-prompt vulnerabilities and amplified unsafe outputs, while alignment training can degrade overall performance.
Method
STAR-1 combines diverse safety data, policy-grounded deliberative reasoning, and GPT-4o-based quality selection to create a 1K-sized dataset.
Results
40.0% average safety improvement across five R1-distilled models accompanies only a 1.1% decline in general reasoning ability.
Takeaways & Limitations
STAR-1 strengthens LRM safety while largely preserving general reasoning performance, with gains reported across five safety benchmarks.
Takeaways & Limitations
STAR-1 fine-tuned models show signs of overrefusal on XStest’s borderline safety queries.
Abstract
from arXiv · showhide
This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1.
1 Introduction
Large reasoning models offer stronger reasoning but face distinct safety risks, including vulnerability to harmful prompts and amplified unsafe outputs. STAR-1 addresses the resulting alignment trade-off with a small, policy-grounded dataset and reports strong safety gains with minimal reasoning degradation.
- LRMs’ extended chain-of-thought reasoning improves capabilities but creates new safety challenges, including vulnerability to harmful prompts and amplified unsafe outputs.
- Alignment training can degrade overall performance, motivating a stronger balance between safety alignment and general reasoning capabilities.
- STAR-1 is a 1K-sized dataset with SafeTy Aligned Reasoning processes designed to improve safety alignment in large reasoning models.
- STAR-1’s data-generation pipeline combines diversity, deliberative reasoning grounded in safety policies, and high-quality data selection.
- 40.0% average safety improvement across five R1-distilled models accompanies a 1.1% decline in general reasoning ability after STAR-1 training.The reported training setup uses five epochs and requires about 45 minutes on 8×A5000 GPUs for an 8B model.
2 STAR-1 Dataset
STAR-1 is constructed by collecting diverse safety data, grounding reasoning in category-specific policies, and filtering samples for quality and balanced representation. The pipeline reduces a broad corpus to 1,000 policy-aligned examples while preserving safety-category diversity.
- 41K safety training data are collected, structured with deliberative reasoning, and filtered to produce the final 1K-sample STAR-1 dataset.
- Diversity: Diversity is promoted across safety categories and data content, using human-written, machine-generated, and template-augmented samples.
- Diversity: 529,816 harmful instructions from 18 sources spanning eight safety categories are initially collected before duplicate and near-duplicate removal.
- Deliberative Reasoning Paradigm: Deliberative reasoning pairs each categorized harmful instruction with a tailored policy specifying its objective and rules for responses.
- High-Quality Data Selection: GPT-4o scores reasoning traces and answers for safety compliance, policy relevancy, and reasoning accuracy, retaining only samples rated 10 on all criteria.This accuracy filter reduces the dataset to 2,368 samples before diversity-based selection.
- High-Quality Data Selection: A discard probability based on source and category proportions is applied iteratively until 1,000 samples remain, while the reported filtering process preserves category diversity.
3 Experiment
The experiments evaluate STAR-1 by fine-tuning diverse DeepSeek-R1-distilled models and comparing safety and reasoning against instruction and original distilled baselines. Across models and benchmarks, STAR-1 consistently improves safety while generally preserving reasoning performance, with gains varying by model size.
- Experimental Setup: Safety is measured on four benchmarks covering refusal of disallowed content and robustness to adversarial jailbreaks.The evaluation uses StrongReject, JBB-Behaviors, WildChat, and WildJailbreak, with safety rate computed from binary response indicators.
- Experimental Setup: STAR-1 is evaluated on five DeepSeek-R1-distilled models spanning diverse model families and sizes from 1.5B to 32B.The comparisons include corresponding safety-trained instruction models and the original distilled models.
- Safety Results: All evaluated LRMs increase their safety rates after STAR-1 fine-tuning across the five reported safety benchmarks.On the harder WildChat and WildJailbreak benchmarks, average safety improvements are 21.4% and 35.4%, respectively.
- Safety Results: Safety improvement declines with model size, from 54.5% at 1.5B to 27.7% at 32B, while remaining positive at every tested scale.The authors associate the diminishing return with larger models’ stronger pretraining and alignment strategies.
- Comparison with Instruction Models: For Qwen2.5 32B, STAR-1 reaches a 96.1% average safety rate, exceeding the corresponding instruction model by 8.1%.This comparison indicates that the STAR-1-trained LRM can outperform a comprehensively safety-trained instruction counterpart on the reported safety measure.
- Reasoning Results: STAR-1 fine-tuning causes only a 1.1%–3.0% decrease in reasoning performance, while Qwen2.5 32B improves by 1.3%.Reasoning is evaluated across five benchmarks, and the largest tested model shows an average improvement rather than degradation.
4 A Closer Look at the Data Paradigm
STAR-1’s safety gains arise from explicit, policy-grounded reasoning and high-confidence data selection, but reasoning-based safety data affects LRMs and standard LLMs differently. Adding benign examples also reduces overrefusal with modest safety and reasoning trade-offs.
- Two Hidden Keys of Less is More in LM Safety Training: STAR-1’s deliberative reasoning process outperforms randomly selected SafeChain reasoning data for safety alignment.Its process explicitly grounds reasoning in safety policies, whereas SafeChain’s reasoning is relatively coarse-grained.
- Two Hidden Keys of Less is More in LM Safety Training: 11.9% lower safety rate occurs with STAR-1 Med 1K than with STAR-1 High 1K.STAR-1 High 1K also surpasses the full 40K SafeChain dataset on safety rate.
- The Role of Safety Reasoning in LRMs and LLMs: 18.5% lower safety performance results when LRMs are trained on STAR-1 w/o think instead of the original STAR-1.The gap narrows with model size, from 36.2% for 1.5B models to 5.1% for 8B models.
- The Role of Safety Reasoning in LRMs and LLMs: Standard LLMs improve safety by 10.7% with STAR-1 but by 14.3% with STAR-1 w/o think.The authors suggest STAR-1’s reasoning style may disrupt internalized safety priors in standard LLMs.
- A Mitigation for the Overrefusal Behaviour: Adding 915 benign variants raises average not overrefusal rate from 68.9% to 78.1% across two models.Average safety rate decreases 3.7%, from 94.0% to 90.3%, while reasoning ability increases from 56.3% to 57.2%.
5 Related Work
Related work covers safety training through human supervision and explicit safety principles, alongside evidence that small, carefully curated datasets can substantially improve language-model performance.
- LLM Safety Training: Standard LLM safety training commonly uses supervised fine-tuning on high-quality human annotations to mitigate harmful outputs.Other work aligns reasoning processes with explicit safety rules, including human-written principles and AI-driven self-critiques.
- High-quality LLM Training Data: Small, high-quality datasets can produce strong language-model performance with far fewer examples than conventional training corpora.Examples include LIMA’s 1K-example fine-tuning and LIMO’s 817-example mathematical reasoning dataset.
6 Conlusion
The paper introduces STAR-1 as a high-quality 1K-scale safety dataset for LRMs, reporting substantial safety gains with minimal reasoning-performance compromise across model families and scales.
- Conclusion: STAR-1 delivers up to an average 40% enhancement on key safety benchmarks with only a minimal compromise in reasoning performance.The conclusion reports this pattern across multiple model families and parameter scales.
Ethics Statement
STAR-1 is intended to support safer and more robust reasoning, but its sourced content and potential misuse require responsible research use. Qwen3 experiments also report safety gains with minimal reasoning compromise.
- Ethics Statement: STAR-1’s sourced data may contain harmful, biased, or sensitive content, and misuse of aligned models remains possible.The dataset and code are released for non-commercial research purposes.
- Ethics Statement: 18.7% average safety improvement is reported for Qwen3-4B and Qwen3-8B, with a 0.4% compromise in reasoning ability.The comparison is 94.0% versus 75.3% for safety and 69.3% versus 69.7% for reasoning ability.
A.2 Ablation on Dataset Size
Under the same training budget, STAR-1’s 1K dataset matches larger-data safety performance while improving general abilities in the reported Qwen-7B and LLaMA-8B comparisons.
- 0.1% drop and 1.7% gain in safety performance make 5K-trained Qwen-7B and LLaMA-8B comparable to 1K-trained models.
- 0.2% and 3.1% improvements in general abilities favor the 1K-trained models over their 5K-trained counterparts.
- The comparison supports STAR-1’s claim that curated 1K training data can match or exceed larger-scale training under equal step budgets.
- The broader evaluation includes original and STAR-1-trained Qwen3 models on safety and reasoning tasks.
B.3 Analysis of STAR-1 Data Distribution
STAR-1 filtering produces a more balanced distribution across eight safety categories and 18 data sources, but quality prioritization prevents perfect uniformity.
- The filtered STAR-1 distribution is significantly more balanced than the original 41K dataset across safety categories and data sources.
- Iterative removal of samples with high discard probabilities improves coverage across safety concerns and data origins.
- STAR-1 is not perfectly uniform because the pipeline prioritizes data quality and selects high-scoring samples.
- Categories or sources with few high-quality or high-confidence samples cannot necessarily contribute equal numbers while preserving quality.
C.1 Data deduplication process details in STAR-1 generation pipeline
STAR-1 generation combines deduplication, test-set decontamination, safety classification, scoring, and iterative filtering to reduce a large collection to 1K samples.
- Three similarity filters—n-gram, TF-IDF cosine, and sentence-embedding matching—remove duplicate or semantically redundant samples.
- The pipeline also removes training samples resembling test questions to reduce contamination and data leakage.
- 40,961 high-quality samples remain after progressively applying the deduplication and similarity-filtering steps.
- GPT-4o classifies harmful instructions into eight safety categories, assigning unclassifiable cases to Other.
D.3 Tested Models and Evaluation Datasets
The evaluation spans multiple model families, safety benchmarks, reasoning tasks, data-generation checks, and policy resources, with WildChat receiving additional error analysis.
- The tested models span five model sizes and two model families.
- Safety evaluation uses StrongReject, JBB-Behaviors, and WildChat, covering policy violations, misuse behaviors, and toxic conversations.
- Reasoning evaluation covers mathematics, code, science, and knowledge-intensive tasks through AIME 2024, Math500, HumanEval, GPQA Diamond, and MMLU-Pro.
- The overrefusal analysis generates 1,000 benign variants, processes them with DeepSeek-R1, and retains 915 samples after GPT-4o evaluation.
- XSTest scoring combines full compliance and partial refusal using the Not Overrefusal rate formula.
- WildChat analysis found 43 failures among 370 samples, including 32 involving sexual/adult content, reviewed by humans and GPT-4o.