PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang

arXiv:2310.12773v1cs.AIcs.LG

TL;DR

LLM training must balance helpfulness with harmlessness because these objectives can conflict. Safe RLHF decouples their annotations and models, then uses constrained optimization with adaptive balancing during fine-tuning. Across three rounds, it improved helpfulness and harmlessness while better navigating this tension than static Reward Shaping and maintaining equal or superior improvements to existing value-aligned algorithms.

  • Problem

    Helpfulness and harmlessness often conflict during LLM training, making it difficult to produce models that are helpful, safe, and willing to respond.

  • Method

    Safe RLHF decouples helpfulness and harmlessness preferences, trains separate reward and cost models, and uses Lagrangian optimization to balance them under safety constraints.

  • Results

    Across three fine-tuning rounds, Safe RLHF enhanced helpfulness and harmlessness, better navigated their tension than Reward Shaping, and maintained equal or superior improvements to existing value-aligned algorithms.

  • Takeaways & Limitations

    Decoupling annotation and dynamically balancing objectives provides a safety-alignment framework that improves helpfulness while reducing harmful responses.

  • Takeaways & Limitations

    The study used inaccessible pretraining data through the Stanford Alpaca Dataset, lacked an expansive high-quality SFT corpus, and focused on Llama-1 and Alpaca models in single-turn settings.

Abstract

from arXiv · show

With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples human preferences regarding helpfulness and harmlessness, effectively avoiding the crowdworkers' confusion about the tension and allowing us to train separate reward and cost models. We formalize the safety concern of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints. Leveraging the Lagrangian method to solve this constrained problem, Safe RLHF dynamically adjusts the balance between the two objectives during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we demonstrate a superior ability to mitigate harmful responses while enhancing model performance compared to existing value-aligned algorithms. Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with collected human preferences, significantly improving its helpfulness and harmlessness according to human evaluations.

1 INTRODUCTION

LLMs offer broad capabilities but must avoid harmful content, creating a training tension between helpfulness and harmlessness. Safe RLHF addresses this tension by decoupling preference data and objectives, then dynamically balancing them during optimization.

  • Motivation: LLMs support instruction understanding, summarization, reasoning, coding, medical assistance, education, and law, while their responses must avoid harmful content.The paper identifies discrimination, misinformation, and violations of social norms and morals as examples of harmful content.
  • Motivation: RLHF aligns language models with human preferences, but increasing helpfulness and harmlessness can conflict during training.A refusal may be safe yet unhelpful in some scenarios, motivating a balance between the objectives.
  • Safe RLHF: Safe RLHF decouples human preferences during annotation and establishes separate helpfulness and harmlessness optimization objectives.The framework formalizes harmlessness as a safety constraint within Safe RL.
  • Safe RLHF: Decoupled annotation reduces crowdworker confusion, while the Lagrangian method adaptively balances the two conflicting training objectives.The pipeline separately annotates helpfulness and harmlessness and dynamically integrates them during policy optimization.
  • Results: Three Safe RLHF fine-tuning rounds enhanced helpfulness, reduced harmful responses, and maintained equal or superior performance improvements relative to existing value-aligned algorithms.The paper also reports better navigation of the helpfulness–harmlessness tension than static Reward Shaping.

2 PRELIMINARIES

The preliminaries describe RLHF preference modeling and the reinforcement-learning foundations used by Safe RLHF. They define reward maximization and constrained policy optimization as the basis for safety-aware training.

  • Preference Modelling: RLHF uses human preference data and a reward model Rϕ(y, x) to assign scalar scores to language-model responses.Here, x denotes the prompt and y denotes the generated response.
  • Preference Modelling: The Bradley-Terry model estimates preference-pair probabilities as a sigmoid of the difference between the two responses’ reward scores.The likelihood is written as σ(R(yw, x) − R(yl, x)).
  • Safe Reinforcement Learning: RL models language generation as a policy in a Markov Decision Process with states, actions, rewards, transitions, an initial distribution, and discounting.The policy objective maximizes expected cumulative discounted reward.
  • Safe Reinforcement Learning: Safe RL extends the MDP with cost functions and thresholds, defining feasible policies whose expected cost returns satisfy the constraints.The feasible policy set is denoted ΠC.
  • Safe Reinforcement Learning: The goal of Safe RL is to find the optimal policy within the feasible policy set.This constrained objective combines reward maximization with cost-threshold requirements.

3 METHOD: SAFE RLHF

Safe RLHF separates helpfulness and harmlessness preferences into distinct models, then optimizes helpfulness subject to safety constraints. Its Lagrangian formulation dynamically adjusts the penalty for harmfulness during reinforcement learning.

  • Framework: Safe RLHF modifies preference annotation and policy optimization to balance helpfulness against harmlessness.The pipeline introduces substantial changes compared with conventional RLHF in these two stages.
  • Preference Annotation: Crowdworkers separately assess helpfulness and harmlessness, producing decoupled datasets with differing preference labels for the same QA pairs.Safety labels additionally classify responses as harmful or harmless across predefined harm categories.
  • Preference Modeling: The Reward Model estimates helpfulness preferences, while the Cost Model uses harmlessness preferences and safety information to assess response safety.The Cost Model’s classification separates responses into safety-based clusters, supporting adjustment of conflicting objectives.
  • Safe Reinforcement Learning: Safe RL maximizes expected reward while constraining expected cost, replacing an all-responses safety requirement with an expectation-based surrogate controlled by d.The constraint is applied to responses generated by the policy for prompts sampled from the RL distribution.
  • Safe Reinforcement Learning: The Lagrangian converts the constrained problem into an unconstrained form by adding a dynamically modulated harmfulness penalty to the helpfulness objective.The model parameters and Lagrange multiplier are updated iteratively in a min-max procedure.
  • Data Iteration: Across three Safe RLHF rounds, most safety-related prompts become solved, while human red-teaming adds prompts from the second round to maintain balance.The prompt mix includes safety-unrelated, resolved safety-related, unresolved safety-related, and red-teamed prompts.

4 EXPERIMENTS

The experiments evaluate whether Safe RLHF can improve helpfulness and harmlessness together, explain the benefits of decoupled objectives, and test how the method balances their tension. Across three fine-tuning cycles, Safe RLHF shifted outputs toward lower cost, improved evaluation scores, reduced harmful responses, and outperformed reward shaping in balancing objectives.

  • Experimental Questions: The experiments test whether Safe RLHF can simultaneously improve LLM helpfulness and harmlessness, assess decoupled annotation benefits, and examine objective balancing.The study also includes an ablation of the classification-capable Cost Model.
  • Experimental Setup: Three Safe RLHF cycles combine red teaming, preference-data annotation, Reward and Cost Model training, and Safe RL fine-tuning.Red teaming is excluded from the first cycle and supports follow-up training phases.
  • Helpfulness and Harmlessness Evaluation: After three rounds, Beaver-v3 increased helpfulness Elo over Alpaca-7B by GPT-4: +244.91 and Human: +363.86, while harmlessness increased by GPT-4: +268.31 and Human: +237.98.Both GPT-4 and human evaluators assessed the Elo comparisons, with nearly consistent evaluations.
  • Helpfulness and Harmlessness Evaluation: Harmful-response probability fell from 53.08% for Alpaca-7B to 2.45% for Beaver-v3 on the evaluation set.Crowdworkers labeled whether model responses were safe.
  • Decoupling of Harmlessness and Helpfulness: Decoupled annotation raised inter-rater agreement to Helpfulness: 69.00% and Safety: 66.53%, compared with 61.65% for single-dimensional annotation.The study also reports higher approval rates and greater helpfulness gains from PPO, while Safe RLHF achieved stronger harmlessness improvements.
  • Balance Between Objectives: Reward shaping weights that were excessively high or low over-optimized one objective, while moderate weights remained inferior to Safe RLHF.The tested weights were ν = 0.01, 0.5, 1, 2, 5, 10, and 100; the Cost Model’s integrated safety labels and preferences supported dynamic balancing.

5 RELATED WORKS

Related work presents LLMs as capable systems that can also produce inaccurate, biased, misleading, or harmful responses. RLHF uses human feedback to steer models toward diverse goals, including safer and more controllable behavior.

  • LLMs support instruction understanding, summarization, complex reasoning, coding, medical assistance, education, and law-related applications.
  • LLM-based systems can produce inaccurate, biased, misleading, and harmful responses despite their capabilities.
  • RLHF aligns LLM behavior with human preferences across goals such as style and ethical values.

6 LIMITATIONS AND FUTURE WORK

The study is constrained by limited training data, missing safety checks, substantial costs, and restricted model and conversation settings. Future work targets broader preference coverage, multi-turn use, newer base models, and additional safeguards.

  • The study used inaccessible pretraining data through the Stanford Alpaca Dataset for PTX loss across all three Safe RLHF rounds.
  • The study lacked an expansive high-quality SFT corpus that could improve helpfulness and harmlessness.
  • Pre- and post-check strategies are warranted, while RLHF incurs substantial financial costs.
  • Safe RLHF currently covers helpfulness and harmfulness, single-turn conversations, and Llama-1 and Alpaca models rather than broader preference, dialogue, and base-model settings.

7 ETHIC DISCUSSION

The work releases an open-source dataset for iterative reward and cost-model training, including red-team prompts for assessing safety vulnerabilities. It also acknowledges labor practices and the misuse risks of public red-team resources.

  • The released open-source dataset supports iterative training of reward and cost models and includes red-team prompts for assessing safety vulnerabilities.
  • Public red-team data carries misuse risks because bad actors could fine-tune models with reversed objectives detrimental to public welfare.
  • Crowdworkers receive estimated average hourly wages of USD 7.02 to USD 9.09, or 1.98x ∼2.56x the local hourly average.

8 CONCLUSION

The conclusion frames Safe RLHF as a way to address the tension between helpfulness and harmlessness by separating preferences and objectives. Three training rounds improved both dimensions and better navigated their trade-off than static Reward Shaping.

  • Helpfulness and harmlessness often conflict, making their combination into a single training objective unreliable.
  • Safe RLHF decouples human preferences during annotation and uses a λ-trade-off between helpfulness and harmlessness objectives.
  • Three Safe RLHF rounds improved helpfulness and harmlessness and better navigated their tension than Reward Shaping’s static balance.
  • The evaluation dataset classifies QA-pair safety across 14 predefined harm categories before ranking generated responses.
  • Helpfulness is assessed separately through response clarity, relevance, and quality rather than the safety classification alone.

A.2 DATA GENERATION

The data-generation process iteratively produces model responses, adds targeted red-team prompts in later rounds, and collects human annotations for reward and cost modeling. Annotation guides and multi-stage quality control support consistent labeling.

  • Data generation: Responses are generated from the latest RLHF model in each iteration, using prompts primarily from prior safety datasets and limited research-team red-team prompts in rounds 2 and 3.Sampling uses temperature values of 0.001, 0.5, 1.0, and 1.2, with top-K 50 and top-p 0.95.
  • Safety labeling: Harmlessness assessment uses 14 harm categories, and a response is labeled safe only when every category is risk-neutral.The categories cover risks including hate speech, discrimination, violence, financial crime, misinformation, terrorism, self-harm, animal abuse, and child abuse.
  • Annotation: Two annotation guides explain harm-category classification and the distinct preference dimensions to reduce crowdworkers’ confusion about decoupled preferences.The guides combine category examples with instructions for preference annotation.
  • Quality control: A data partner recruits, assigns, and trains annotators, while its quality-control team and the research team jointly review annotation batches.The research team samples at least 10% of each reviewed batch, with percentage agreement required to meet or exceed 90%.

B IMPLEMENTATION DETAILS

The implementation initializes separate reward and cost models, combines RL and pretraining objectives, and uses PPO-based constrained fine-tuning with alternating model and Lagrange-multiplier updates.

  • Model initialization: Reward and cost models are initialized from LLaMA-7B, replacing the pretrained model’s final head with a randomly initialized fully connected output layer.The reward and cost models are the same size as the actor model, while the remaining layers load pretrained weights.
  • Preference models: Reward-model and cost-model training adds regularization terms to improve generalizability and stabilize training.The coefficients μR and μC control the regularization strength.
  • RLHF training: RLHF training combines an RL objective with a PTX pretraining objective and applies a per-token KL penalty relative to a frozen reference model.The reward model supplies a sparse final-token reward, while KL divergence supplies dense token-level penalties.
  • RL optimization: PPO trains the language model using a clipped surrogate objective based on importance-sampling weights and GAE-estimated reward advantages.The importance weight compares the current policy with the policy from the previous gradient update.
  • Safe RLHF optimization: Safe RLHF alternates updates to the LLM parameters θ and Lagrange multiplier λ while defining separate reward and cost signals.KL rewards are split between reward and cost and normalized through a (1+λ) factor; a moving average estimates the cost during training.

C SUPPLEMENTARY DETAILS OF THE EXPERIMENTS

The supplementary materials specify training hyperparameters, GPT-4 evaluation prompts for helpfulness and harmlessness, and red-team attack categories used to analyze safety failures.

  • Hyperparameters: Tables 2–4 list the hyperparameters for reward-model, cost-model, and three-round Safe RLHF training.These tables organize the training settings for the main model components and Safe RLHF rounds.
  • Model-based evaluation: GPT-4 evaluates paired model outputs separately for helpfulness and harmlessness using prompts that compare two assistants’ responses to the same question.The helpfulness evaluation considers accuracy, clarity, completeness, contextual understanding, creative problem-solving, and depth of explanation.
  • Red teaming: Red teaming before rounds 2 and 3 identifies four attack types: partial harmfulness, scenario assumptions, contradictory analysis, and complex text command embedding.The authors characterize the latter three as arising from language models’ tendency to follow instructions, while partial harmfulness is an intermediate state during simultaneous helpfulness and harmlessness improvement.

D.3 CONTRADICTORY ANALYSIS

Contradictory-analysis attacks can make the model highlight positive aspects of harmful content, even when direct questioning elicits a negative answer. Other examples show harmful outputs arising from role-play or mandated text inclusion.

  • Contradictory analysis: When directly asked whether restricting education is beneficial, the model gives a negative answer, but contradictory analysis can elicit a positive framing of the harmful content.The example identifies this behavior as a probability rather than a guaranteed outcome for Beaver-v2.
  • Model response: A long Beaver-v3 response refuses to endorse restricting education and instead explains why education should remain broadly accessible.The response connects educational access with knowledge, skills, poverty reduction, and community development.
  • Related attack patterns: Other documented attacks include harmful outputs caused by inserting required text or requesting a response in Python-program form.The examples concern Beaver-v1 and show that specific formatting or content instructions can alter safety behavior.
Loading 2310.12773v1…