Source-linked AI summary
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, Yaodong Yang
TL;DR
LLM safety alignment needs scalable preference data and fine-grained safety labels because models can produce harmful or deceptive behavior and annotation is costly. PKU-SafeRLHF provides graded labels and decoupled or single preferences, then applies them to moderation and RLHF. The paper reports finer-grained moderation and effective alignment across these settings, while noting dataset size and taxonomy limitations.
Problem
LLMs can generate harmful or deceptive behavior, while preference datasets and safety meta-label classification are costly to annotate at scale.
Method
PKU-SafeRLHF collects 265k graded safety-labeled Q-A pairs and 166.8k single- and dual-preference examples, using them for moderation and RLHF.
Results
The dataset supports severity-sensitive moderation with finer-grained safety filtering and RLHF in both single-preference and dual-preference settings.
Takeaways & Limitations
PKU-SafeRLHF provides an open research platform for safety alignment with graded harm labels and separate helpfulness and harmlessness preferences.
Takeaways & Limitations
The dataset is relatively small compared with large-scale human preference datasets, and its 19 harm categories may omit harms or overlap substantially.
Abstract
from arXiv · showhide
In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF.
1 Introduction
PKU-SafeRLHF addresses the safety risks and annotation costs involved in aligning LLMs by providing large-scale safety labels and preference data. It supports moderation and RLHF research through graded harm annotations and decoupled helpfulness and harmlessness preferences.
- LLMs can generate offensive content, leak privacy, spread misinformation, or exhibit deceptive alignment, motivating alignment with human intentions and values.
- Preference datasets and meta-label classification are costly, limiting large-scale safety alignment efforts.
- 265k Q-A pairs from 44.6k prompts receive safety meta-labels covering 19 harm categories and three severity levels.
- 166.8k Q-A-B pairs provide dual preferences that decouple helpfulness and harmlessness and single preferences that condense multi-metric judgments into one comparison.
- The dataset is applied to severity-sensitive moderation, RLHF with dual or single preferences, and evaluation of open-source or API-based models.
2 Related Work
Related work frames safety alignment as a response to undesirable LLM behavior and describes RLHF as optimizing human-rated quality while avoiding low-rated outputs. Existing approaches include moderation, safety alignment, and methods addressing the tension between helpfulness and harmlessness.
- LLM alignment seeks to ensure that language models adhere to human intentions and values despite undesirable behaviors such as untruthful answers.
- RLHF uses human feedback to construct a reward function and optimize language models with reinforcement learning methods such as PPO.
- Safety methods include data filtering, safety alignment, red teaming, and post-hoc moderation of model inputs or outputs.
- Safe RLHF methods address the tension between helpfulness and harmlessness, while other approaches bypass reward modeling or use generative reward models.
3 Dataset
PKU-SafeRLHF constructs diverse prompts and multiple responses, then jointly annotates safety across harm categories and severity levels. The resulting dataset uses human-AI annotation and structured filtering to support preference and safety research.
- Dataset overview: The dataset overview combines prompt and response generation with safety annotation across harm categories, severity levels, and model sources.
- Model selection: Alpaca3-70B generates prompts, while Alpaca-family models generate corresponding responses after supervised fine-tuning on Alpaca 52K.
- Model selection: Open-source SFT data is used to improve transparency about behavioral changes from pre-training through SFT and RLHF.
- Model selection: RLHF resource constraints motivate using 7B/8B models that can be trained on a single machine with 8 × A800 or A100 80GB GPUs.
- Response generation: The pipeline generates one default response and 10 higher-temperature responses per prompt, then filters garbled text and selects diverse responses by textual similarity.
- Response generation: PKU-SafeRLHF responses have 32% less garbled and semantically unclear content than BeaverTails.
- Annotation: Joint human-AI annotation with over 28 annotators assesses Q-A pairs, while the harm taxonomy contains 19 categories informed by prior datasets and safety investigations.
- Harm classification: Category analysis finds strong overlaps for several harm pairs but low or negative correlations for most remaining categories, while annotator variance causes some inconsistent labels.
4 Application
PKU-SAFERLHF supports severity-sensitive moderation and safety alignment by combining fine-grained safety labels with decoupled helpfulness and harmlessness preferences. Its applications report strong moderation performance, improved aligned-model results, and preference-model agreement with human evaluations.
- Severity-Sensitive Moderation for Risk Control of LLMs: 85% accuracy shows that severity-sensitive moderation can identify the severity of unsafe question-answer pairs.The model uses all severity-level meta-labels to distinguish minor-to-severe harmful behavior.
- Severity-Sensitive Moderation for Risk Control of LLMs: 93% accuracy and 71.3% exact-match accuracy demonstrate strong binary unsafe-content detection and multi-class harm-category identification.The binary classifier also maintains a low false-positive rate, while category performance is imperfect for some overlapping or bias-sensitive subcategories.
- Severity-Sensitive Moderation for Risk Control of LLMs: Severity-sensitive moderation distinguishes harmful conversations across severity levels, supporting flexible risk control beyond binary safety judgments.The supported range runs from minor to severe harm.
- Safe RLHF Pipeline: Decoupling helpfulness and harmlessness in dual-preference data with direct Lagrangian optimization significantly outperformed direct single-preference alignment on Alpaca models.Performance based on PKU-SAFERLHF also exceeded the improvements obtained with BeaverTails relative to the original Alpaca models.
- Safe RLHF Pipeline: PKU-SAFERLHF-trained aligned models achieved win rates above 80% against models trained on other datasets across helpfulness and harmlessness.The direct comparison further supports the quality of the dataset for safety alignment.
- Reward and Cost Models for Evaluation: Reward and cost models showed high consistency with human evaluations, while the cost-model score was calibrated to human safety grades.The cost-model threshold at zero aligned with the boundary between human safety grades 3 and 4.
5 Conclusion
PKU-SAFERLHF provides large-scale safety preference and meta-label data for LLM alignment, and the authors report gains in moderation accuracy, helpfulness, and harmlessness.
- 166.8k preference examples and 265k safety-labeled Q-A pairs support both single- and dual-preference alignment experiments.The dataset is based on Llama-series models and includes graded safety labels spanning 19 harm categories.
- The severity-sensitive moderator offers finer-grained safety filtering and significantly surpasses existing regulation methods in accuracy.
- Models trained with PKU-SAFERLHF show significant improvements in helpfulness and harmlessness compared with models trained with BEAVERTAILS.
6 Limitations
The dataset is smaller than commercial preference datasets, its harm taxonomy has coverage and overlap issues, and its applicability is limited across domains and languages.
- PKU-SAFERLHF is relatively small compared with large-scale human preference datasets constructed by commercial organizations.The authors identify open-source availability and research-focused design as compensating advantages.
- The 19 harm categories may omit some harms, while overlap between categories can affect severity-sensitive moderation.The authors plan to improve classification, enrich underrepresented categories, and balance category distributions.
- High-risk legal, medical, and financial applications require additional domain-specific annotations because specialized risks are not fully captured by the current taxonomy.
- The dataset focuses on English-language alignment, leaving non-English coverage limited and potential cultural biases unresolved.Regional differences in sensitivity to harmful content can affect interpretation of alignment preferences.
7 Fair and Ethical Labor
The paper describes annotation labor, dataset governance, and the harm taxonomy used to structure safety labels across categories and severity levels.
- 28 full-time crowdsourced workers with text-annotation expertise supported the dataset’s annotation process.Their estimated average hourly wage was USD 8.02 to USD 9.07, above the cited local minimum wage of USD 3.69.
- The project received Institutional Review Board approval and makes the dataset available under a CC BY-NC 4.0 license.
- The authors acknowledge that the dataset could theoretically be misused to train harmful or malicious assistants and condemn malicious use.
- The taxonomy contains 19 harmful-event categories, including violence, privacy violation, cybercrime, psychological harm, and animal abuse.
- Three severity levels classify harmful events according to their impact and the response measures they require.The authors state that LLM security is not a simple binary right-or-wrong problem.
B Data Collection
The data-collection pipeline combines manually designed harm examples and guidelines with model-generated prompts, contextual enrichment, and filtering for diversity and quality.
- 77.6% of prompts are model-generated, with 30 selected examples per harm category spanning three severity levels.Manual examples are written first to improve diversity and quality, then used alongside safety guidelines for generation.
- Prompts are generated separately for each harm category using severity rules, examples, and safety guidelines as model inputs.Alpaca3-70B and WizardLM-30B-Uncensored generate three prompts per category across minor, moderate, and severe levels.
- Alpaca3-70B enriches prompts by adding background information and reasons while mimicking the user’s tone.
- The generation prompt requires harmful questions to be ordered by increasing harm level and to violate specified trust-and-safety guidelines.
B.1.2 Prompt Analysis
PKU-SAFERLHF prompts retain the basic characteristics of BEAVERTAILS while exhibiting a more diverse distribution. Context supplementation also produces a more uniform token-length distribution.
- PKU-SAFERLHF prompts preserve BEAVERTAILS’ basic characteristics but show a more diverse distribution after UMAP visualization.
- Context-supplemented prompts have a more uniform token-length distribution than BEAVERTAILS prompts.
B.2 Response Generation (may contain harmful examples)
Response generation produces diverse candidate answers through temperature variation, similarity filtering, and quality checks. Annotation then combines AI recommendations, human judgments, and staged quality control to construct reliable data.
- Response Generation: Alpaca-(1, 2, 3) models generate an initial response and 10 candidate responses for each prompt using varied temperatures.The initial response is A1, while the candidates form A2s for subsequent filtering and selection.
- Response Generation: Sentence-BERT selects the three A2 candidates with the lowest cosine similarity to A1.
- Response Generation: Regular expressions and NLTK tools remove garbled candidates before one is randomly paired with A1 and the others are added to the Q-A dataset.
- Human-AI Annotation: GPT-4 provides recommended harm-category and severity labels with analyses, which annotators refine using their own judgments.
- Human-AI Annotation: Annotation proceeds through harm-category labeling, independent helpfulness and harmlessness judgments, and quality sampling by GPT-4, human reviewers, and researchers.The process covers 19 harmful categories and uses staged review thresholds for acceptance or re-annotation.
- Motivation: Solely crowdsourced annotation offers high quality but is extremely time-consuming, costly, and difficult to scale.
- Motivation: The joint human-AI strategy improves annotation efficiency and scalability while retaining human quality control for safety-related tasks.Safety-related prompts can cause existing models to refuse answers, so human oversight remains essential for data integrity.
C.3.2 Annotation Guidelines — Helpful Evaluation
Helpful evaluation prioritizes accuracy while also considering informativeness, conciseness, and instruction following. Safety is evaluated independently because safe responses can conflict with correctness or helpfulness on dangerous queries.
- Accuracy: Accuracy is the highest-priority helpfulness criterion when comparing responses.If both responses fail, they are marked as invalid data.
- Information: Helpful responses answer the question directly, provide relevant detail, and minimize extraneous information.A top-down structure that answers first and then expands is generally preferred under limited context length.
- Instruction Following: Instruction following requires directly answering the user’s question rather than merely supplying related information.The Andy Lau example fails because it gives relevant biography details without stating the requested year.
- Safety: Safety is an independent evaluation dimension that may conflict with correctness, informativeness, and instruction following for unsafe queries.Evaluation considers the overall safety of both the query and response.
- Harm and Severity: Harm annotation assigns categories and severity levels to each Q-A pair, while allowing multiple categories when applicable.Severity levels distinguish minor, moderate, and severe harms, and safety preferences can still partially order pairs at the same level.
- Safety Preference: Safety preference favors refusals and educational redirection over specific instructions that could enable dangerous behavior.In the violence example, Response B is safer because it refuses direct help and recommends peaceful communication.
D Training details
Training uses PPO-based objectives for RLHF and SafeRLHF, with clipped policy updates and an additional cost term in the safety-oriented formulation.
- RLHF: PPO training uses a surrogate clipped loss as the RLHF fine-tuning objective.
- SafeRLHF: SafeRLHF fine-tuning uses PPO-Lagrange with a corresponding surrogate clipped loss.
- Objective Components: The importance-sampling weight compares current and previous policy probabilities, while the clip ratio constrains policy updates.The advantage is estimated using generalized advantage estimation.
D.2 Training Parameter
Training uses a multi-GPU server and AdamW optimization, with detailed hyperparameters provided in Table 9 and training loss shown in Figure 9.
- Experiments run on an Intel Xeon Platinum 8378A CPU server with 64 cores and eight NVIDIA A800-SXM4-80GB GPUs.
- Models are trained with AdamW, using β1 = 0.9 and β2 = 0.95.The passage directs readers to Table 9 for additional training parameters.
- Figure 9 presents the loss curve during SafeRLHF training.
- Table 9 is labeled as the source of hyperparameters for model fine-tuning.