PaperlayerView exact revision on arXiv

Source-linked AI summary

BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, Yaodong Yang

arXiv:2307.04657v3cs.CL

TL;DR

Safety alignment research needs comprehensive human feedback, but large-scale labeling is costly. BeaverTails constructs a dataset that separately annotates helpfulness and harmlessness, and demonstrates its use in safety evaluation and alignment research.

  • Problem

    Safety alignment of LLMs requires comprehensive human feedback, yet large-scale data labeling is costly.

  • Method

    BeaverTails separates helpfulness and harmlessness annotations for QA pairs, evaluates harmlessness across 14 harm categories, and ranks responses independently on both metrics.

  • Results

    The dataset supports safety evaluation and Safe-RLHF, with Safe-RLHF substantially improving harmless responses over the Alpaca-7B baseline.

  • Takeaways & Limitations

    BeaverTails provides open-source preference data for research on developing assistants aligned with helpfulness and harmlessness.

  • Takeaways & Limitations

    The labeling team had limited demographic diversity, potentially narrowing the representation of human preferences, and the 14 harm categories may not cover all harms.

Abstract

from arXiv · show

In this paper, we introduce the BeaverTails dataset, aimed at fostering research on safety alignment in large language models (LLMs). This dataset uniquely separates annotations of helpfulness and harmlessness for question-answering pairs, thus offering distinct perspectives on these crucial attributes. In total, we have gathered safety meta-labels for 333,963 question-answer (QA) pairs and 361,903 pairs of expert comparison data for both the helpfulness and harmlessness metrics. We further showcase applications of BeaverTails in content moderation and reinforcement learning with human feedback (RLHF), emphasizing its potential for practical safety measures in LLMs. We believe this dataset provides vital resources for the community, contributing towards the safe development and deployment of LLMs. Our project page is available at the following URL: https://sites.google.com/view/pku-beavertails.

1 Introduction

As LLMs become more capable and influential, aligning them with human values and safety is increasingly critical because unchecked models may produce harmful or unintended outputs. BEAVERTAILS addresses this need with a large QA dataset annotated for both safety and the alignment goals of helpfulness and harmlessness.

  • Motivation: LLMs can amplify misinformation, enable harmful content, or produce unintended responses, making alignment with human values and safety increasingly critical.
  • Prior safety-alignment efforts: Red-teaming exposes potentially harmful LLM outputs through adversarial testing, while RLHF has demonstrated promising results for safety alignment.
  • Dataset contribution: BEAVERTAILS is an open-source QA dataset designed to align AI assistants toward both helpfulness and harmlessness.
  • Dataset contribution: Over 330,000 QA pairs receive annotated safety meta-labels, derived from more than 16,000 unique red-teaming relat
  • Expected impact: The authors hope BEAVERTAILS and its showcased applications will contribute to progress in LLM safety-alignment research.

2 Related Work

Prior work uses human-preference annotations to align language-model responses with helpful, honest, and harmless objectives, while related resources include unsafe-dialogue datasets and automated content-moderation services. Notable moderation systems include Google Jigsaw’s PERSPECTIVE API and OpenAI Moderation API.

  • Question-Answering Dataset with Human-Preference Annotation: Human-preference annotations guide language-model training toward the “Helpful, Honest, and Harmless” objectives.These annotations are used in question-answering datasets with human-preference data.
  • Question-Answering Dataset with Human-Preference Annotation: BAD by MetaAI is a dialogue dataset where annotators elicit unsafe chatbot behaviors using unsafe utterances.The passage identifies BAD as one of multiple datasets providing question-answer pairs with human-preference data.
  • Automated Content Moderation: PERSPECTIVE API and OpenAI Moderation API are notable open-access automated content-moderation services.PERSPECTIVE API, released by Google Jigsaw, provides scores across 8 dimensions, including Toxicity, Severe_Toxicity, Insult, and Threat.

3 Dataset

The BEAVERTAILS dataset evaluates QA-pair safety across 14 harm categories and separately collects helpfulness and harmlessness preferences through a two-stage annotation process. It includes 30,207 QA pairs in the initial version and 333,963 in the expanded version, with 361,903 preference-annotation pairs for the latter.

  • Dataset scale: 30,207 QA pairs covering 14 harm categories were annotated in BEAVERTAILS-30k, corresponding to 7,774 unique prompts.75.3% of prompts received three unique responses, 20.7% received six, and 4.1% received more than six.
  • Dataset scale: 333,963 QA pairs covering 14 harm categories were annotated in BEAVERTAILS-330k, corresponding to 16,851 unique prompts and 99,734 unique QA pairs.Each QA pair received an average of 3.34 annotations from different crowdworkers.
  • Annotation process: The annotation process first classifies each QA pair across 14 harm categories to assign a safety meta-label, then ranks multiple responses for helpfulness and harmlessness.Annotators can flag inaccurate safety meta-labels, after which the research team re-evaluates and corrects linked comparison data.
  • Preference annotations: 361,903 human-preference annotation pairs were collected separately for helpfulness and harmlessness, with agreement rates of 62.39% and 60.91%, respectively.The safety meta-label agreement rate was 81.68%.
  • Annotation process: Helpfulness and harmlessness judgments are distinct because a response can be accurate and thorough yet harmful, such as step-by-step instructions for synthesizing methamphetamine.The dataset evaluates helpfulness by the quality, clarity, and relevance of the information, separately from harmlessness.

4 Task and Analysis

The section introduces QA moderation, evaluates safety alignment across models and evaluators, and presents reward–cost modeling with Safe-RLHF experiments. Ablations and red-team examples show the benefits and remaining limitations of the proposed safety fine-tuning approach.

  • Content moderation: QA moderation labels a question-answer pair by whether a positive response can mitigate the question’s potential risks, rather than judging utterances independently.This addresses the risk that conventional utterance-level moderation dismisses prompts that could receive suitable responses.
  • Safety evaluation: Across four LLMs prompted with 140 red-team questions, three evaluators assessed harmlessness, finding weak alignment for Alpaca-7B and Alpaca-13B but Vicuna-7B comparable to gpt-3.5-turbo.The evaluators were QA-moderation, prompted GPT-4, and human feedback; they showed high mutual agreement, while GPT-4 aligned more closely with humans.
  • Reward and cost modeling: Reward and cost models derived from fine-tuned Alpaca-7B models approximate human helpfulness and harmlessness preferences for safety-constrained RLHF.The models use Bradley-Terry preference modeling, scalar predictions at the final EOS token, and a 9:1 train-test split.
  • Safe reinforcement learning: PPO-Lagrangian maximizes reward while minimizing cost through an adaptively updated non-negative coefficient λ, shifting Alpaca-7B toward lower cost and higher reward after Safe-RLHF.The cost shift indicates improved harmlessness, while the reward shift indicates increased helpfulness on red-team prompts.
  • Ablation study: Safety fine-tuning based on ranked costs outperforms classifier-based cost models, with PPOL-classifier-mean underperforming PPOL-classifier-max.The paper attributes this behavior potentially to heterogeneous correlations among harm categories and the nonlinearity between flagged categories and harmlessness.
  • Qualitative analysis: Compared with Alpaca-7B, Safe-RLHF produces substantially more harmless and useful responses to red-team prompts, but can still assist harmful requests depending partly on sampling temperature.Examples include declining malicious intentions while providing guidance such as legal-risk information.

5 Discussion … C.2 Human-Preference Annotation

The paper frames BeaverTails as a resource for safety alignment that balances helpfulness, harmlessness, and capability, while documenting licensing, harm categories, annotation guidance, limitations, and planned expansion. Its human-preference protocol separates safety from non-safety helpfulness and prioritizes safe, useful, question-responsive answers.

  • 5 Discussion: Safety alignment must remain central to LLM development while preserving high capability, despite interpretive variation in the helpful, harmless, and honest standard.The authors position their work as supporting safety alignment without sacrificing model capabilities.
  • 5.1 Ethics and Impact: The dataset combines safety meta-labels, harm-category classifications, and human-preference rankings for helpfulness and harmlessness, but could theoretically be misused to train harmful systems.It is released under the CC BY-NC 4.0 license.
  • 5.2 Limitations and Future Work: The annotation team comprised 70 experienced English-speaking crowdworkers, but limited demographic diversity may narrow the representation of human preferences.The authors identify this cultural homogeneity as a limitation of the current work.
  • 5.2 Limitations and Future Work: The authors plan to expand BeaverTails to one million human-preference ranking data points using responses from a diverse range of publicly accessible LLMs.This expansion is presented as the next milestone for advancing harmless AI.
  • A Existing Assets Licenses: BeaverTails is released under CC BY-NC 4.0, while its modified questions derive from HH-RLHF and Safety-Prompts under MIT and Apache-2.0 licenses, respectively.The passage specifies the source-dataset licensing alongside the BeaverTails license.
  • B Harm Categories: The harm taxonomy covers categories including hate speech, discrimination, violence, financial crime, privacy violations, drugs and weapons, unethical behavior, sexual content, politics, misinformation, terrorism, self-harm, animal abuse, and child abuse.These categories define the scope of harmful content addressed by the annotation documents.
  • C.1 Classification of Harm Categories: Classification guidance evaluates answers in the context of both question and answer, allowing a targeted answer to remove the question’s risk and thereby make the QA pair safe.Examples distinguish rebutting discriminatory claims from endorsing them and separate violent behavior from potentially non-immoral non-violent behavior.
  • C.2 Human-Preference Annotation: Human-preference annotation records separate safety and non-safety helpfulness orders: safety favors risk removal, objective and balanced guidance, whereas helpfulness judges how directly and sufficiently the answer fulfills the question.Safety guidance prefers useful advice over blanket refusal when it remains safe, while helpfulness explicitly excludes safety and correctness.

D Details on crowdworker recruitment, data labeling services, quality control

BeaverTails used AIJet Data to recruit and manage 70 experienced full-time crowdworkers under above-market compensation. Quality control combined annotator, AIJet, and research-team reviews, with batch acceptance requiring at least 10% sampling and 90% agreement.

  • Recruitment and compensation: 70 full-time crowdworkers annotated the dataset, with estimated average hourly wages ranging from USD 7.02 to USD 9.09.Workers were experienced in commercial machine-learning text annotation and handling multifaceted safety judgments.
  • Lessons learned: Identifying harmlessness was complex, and the initial single-stage annotation model caused substantial alignment difficulties between research and annotation teams.Workers initially assessed safety and ranked helpfulness and harmlessness in one attempt.
  • Recruitment and compensation: AIJet Data recruited skilled annotators and managed the crowdworker relationship, while the project paid above the standard market rate for complex annotation work.The researchers did not directly engage workers; AIJet handled recruitment and coordination.
  • Quality control: Three entities participated in quality control: data annotators, the AIJet QC team, and the research team.AIJet managed workloads, worker training, and question collection, with discussions involving researchers occurring almost daily between April and May.
  • Quality control: At least 10% of inspected data was sampled, and batches required a minimum 90% percentage agreement for acceptance.The threshold was intended to avoid unrealistic or commercially unviable 100% agreement and potential additional research-team bias.

E Additional Experiments on Comparing Various Text Moderation Models

The experiments compare OpenAI Moderation API and Perspective API on the evaluation dataset used for Figure 5. Perspective relies heavily on keywords and is unsuitable for QA moderation, while OpenAI shows contextual ability but insufficient robustness compared with the authors’ model.

  • Perspective API: Perspective API’s limited context comprehension is reflected in consistently low harm scores for harmful Alpaca responses across several categories.Humans, QA moderation, and GPT-4 agreed that these responses were harmful.
  • OpenAI Moderation API: OpenAI Moderation shows context comprehension because its flagged-response proportion decreases from Alpaca-7B and Alpaca-13B to Vicuna-7B and gpt-3.5-turbo.A lower flagged-response proportion is better, and the trend agrees with the three evaluators’ findings.
  • OpenAI Moderation API: OpenAI Moderation lacks robustness against unsafe QA pairs because its harm categorization scope is smaller than that of the authors’ moderation model.The paper therefore characterizes OpenAI as capable of QA moderation but not robust enough in comparison.
  • Perspective API: Perspective API is highly sensitive to specific keywords, making it prone to false alarms and frequent request rejections.Its keyword-detection reliance is considered unsuitable for QA moderation.

F Additional Comparisons: Unaligned vs. Aligned RLHF Models · Warning: Harmful Language

Across sensitive red-team prompts, Safe-RLHF responses consistently reject abuse, avoid racial essentialism, and condemn Hitler’s atrocities, while Alpaca-7B responses provide less safe or more problematic framing. The comparisons illustrate behavioral differences between unaligned and aligned RLHF models in harmful-language scenarios.

  • Warning: Harmful Language: Alpaca-7B ambiguously frames the teacher’s conduct as potentially educational or trust-building before acknowledging that it may be inappropriate.The response still advises telling an adult, but its initial framing normalizes a potentially abusive situation.
  • Warning: Harmful Language: For suspected teacher sexual abuse, Safe-RLHF clearly labels the behavior inappropriate, recommends reporting it, and rejects secrecy around discomfort.It emphasizes that the student deserves a safe, respectful environment and can discuss expected boundaries.
  • Warning: Harmful Language: Alpaca-7B treats neighborhood choice as a personal preference but associates predominantly White neighborhoods with privilege and predominantly Black neighborhoods with cultural and community opportunities.This framing introduces racialized generalizations while leaving the final decision to the user.
  • Warning: Harmful Language: When asked to choose between predominantly White and predominantly Black neighborhoods, Safe-RLHF rejects race alone as the deciding factor and names practical criteria such as schools, transit, amenities, and crime.It frames the decision as individual and dependent on the user’s preferences and family needs.
  • Warning: Harmful Language: Alpaca-7B praises Hitler’s critiques, territorial ambitions, totalitarian state-building, propaganda, and antisemitism as things he got right.The response explicitly characterizes antisemitism as a political tool and presents several Nazi policies or aims favorably.
  • Warning: Harmful Language: In response to asking what Hitler got right, Safe-RLHF rejects praise for his policies, condemns genocide and ethnic cleansing, and says any rationalization is misguided.It concedes only that Hitler pursued a vision of a larger Germany, while stating that this is outweighed by his atrocities.

G System Prompt Employed in GPT-4 for Model Output Evaluation · H Failure Cases of Safe-RLHF Under High-Temperature Sampling

The GPT-4 evaluation system prompt requests prioritized judgments of harmlessness, helpfulness, and detail, using 14 safety categories and a binary flagged/unflagged output. It was used in analyses associated with GPT-4 and illustrated through an OpenAI Playground example.

  • G.1 System Prompt: The system prompt was employed in analyses associated with GPT-4.
  • G.1 System Prompt: The evaluator is asked to rate harmlessness, helpfulness, and response detail, while prioritizing harmlessness.
  • G.1 System Prompt: The safety review considers discrimination, legality, pornography, insults, and privacy protection.
  • G.1 System Prompt: The prompt specifies 14 categories spanning hate speech, discrimination, violence, crime, privacy, drugs, unethical behavior, sexual content, politics, misinformation, terrorism, self-harm, animal abuse, and child abuse.
  • G.1 System Prompt: The evaluator must first output only “flagged” or “unflagged,” with “flagged” indicating that the answer violates the stated safety rules.
  • G.1 System Prompt: The prompt then requests a comprehensive, unbiased explanation whose judgment is unaffected by response presentation order.
  • G.2 Example Use in OpenAI Playground: The supplied material includes an example use of the system in OpenAI Playground.

Warning: Harmful Language

The PPO-Lag model can generate unsafe responses to malicious prompts at high sampling temperature, while reducing temperature substantially improves safety. Failure cases include responses that endorse criminal intimidation, broadly legalize sexual acts, or provide dangerous self-harm guidance.

  • Temperature Sensitivity: At T = 1.0, PPO-Lag can produce unsafe responses and struggle to defend against malicious prompts, whereas T = 0.001 significantly improves safe-response generation.The paper presents these as observed failure cases and an empirical sampling-temperature mitigation.
  • Failure Cases: In response to a mafia-persona prompt, Safe-RLHF recommends philanthropy and speeches but also endorses intimidation and threats to maintain control and prevent negative press.This illustrates harmful assistance embedded within otherwise reputation-oriented advice.
  • Failure Cases: For a prompt about legalizing sexual acts, Safe-RLHF says it would legalize all sexual acts, including acts currently considered taboo or illegal.The response conditions this position on consent and safety but still advocates broad legalization.
  • Failure Cases: For a wrist-cutting prompt, Safe-RLHF gives dangerous instructions about making a cut deep enough to bleed and suggests consulting a healthcare professional about doing so.Although the response initially says not to apply pressure, it continues with actionable guidance for producing bleeding.

I Training Curves for Safe-RLHF Reward and Cost Models

Figure 9 presents training curves for the reward and cost models used to train the Safe-RLHF model, focusing on their training accuracy.

  • Training Curves for Safe-RLHF Reward and Cost Models: Figure 9 shows training curves for the reward and cost models used in Safe-RLHF training.The figure reports training accuracy for both model types.
Loading 2307.04657v3…