PaperlayerView exact revision on arXiv

Source-linked AI summary

Rule Based Rewards for Language Model Safety

Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng

arXiv:2411.01111v1cs.AI

TL;DR

Safety alignment needs precise, updateable behavior specifications because human safety data can be costly, outdated, or produce overcautious and undesirable responses. The paper introduces Rule-Based Rewards, which uses modular LLM-graded rules directly as reinforcement-learning rewards and achieves stronger safety–usefulness performance than a human-feedback baseline. The method is demonstrated for settings where desired behaviors can be separated into explicit, easy-to-judge rules, while reducing human supervision may amplify model biases.

  • Problem

    Human safety-feedback data can be costly to maintain, become outdated, and inadequately specify desired response behavior.

  • Method

    Rule-Based Rewards uses fine-grained, composable rules and LLM grading as an additional reward directly during reinforcement-learning training.

  • Results

    RBRs achieve a 97.1 safety–usefulness F1 score versus 91.7 for the human-feedback baseline and improve safety while minimizing over-refusals.

  • Takeaways & Limitations

    RBRs offer scalable, flexible control of safety behavior with minimal human data and easier updates when desired behavior changes.

  • Takeaways & Limitations

    RBRs are applied where desired behaviors can be separated into explicit, easy-to-judge propositions, and their reduced human supervision may magnify LLM biases.

Abstract

from arXiv · show

Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a costly need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that utilizes AI feedback and only requires a small amount of human data. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. refusals should not be judgmental) along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, achieving an F1 score of 97.1, compared to a human-feedback baseline of 91.7, resulting in much higher safety-behavior accuracy through better balancing usefulness and safety.

1 Introduction

The paper proposes Rule-Based Rewards (RBRs) to specify safety behavior through detailed, modular rules and direct AI feedback during reinforcement learning. RBRs achieve higher safety–usefulness F1 than the human-feedback baseline while reducing over-refusals.

  • RBRs provide fine-grained control by expressing desired and undesired response behaviors as specific, composable rules graded by an LLM.
  • 97.1 F1 for safety and usefulness, versus 91.7 for the human-feedback baseline and 95.8 for the helpful baseline.
  • RBRs substantially reduce over-refusals while maintaining comparable safety performance to human-feedback baselines.
  • RBRs improve safety behavior across reward models with both overcautious tendencies and occasional preferences for unsafe outputs.
  • The experiments include ablations of the safety-prompt set’s amount and composition.

2 Related Works

Prior AI-feedback methods commonly synthesize comparison data for supervised fine-tuning or reward-model training. RBRs instead apply fine-grained, composable LLM feedback directly during reinforcement learning, distinguishing the method from related approaches.

  • Sparrow uses rule-conditioned reward modeling with human feedback, whereas RBRs focus on automated AI feedback.
  • RBRs incorporate LLM feedback directly into reinforcement learning rather than generating synthetic comparison datasets for reward-model training.
  • Other related methods improve final outputs or design rewards, but RBRs target safety behavior through reinforcement-learning training.

3 Setting and Terminology

The experimental setting assumes a standard SFT–RLHF pipeline, helpful-only resources, moderation-based safety prompts, and explicit content and behavior policies. The behavior policy maps content categories to response types while including boundary cases to limit over-refusals.

  • The setup starts with supervised fine-tuning, reward-model training from preference data, and reinforcement learning against the reward model.
  • Helpful-only resources include demonstrations, preference data, and reinforcement-learning prompts without unsafe requests.
  • Safety-relevant prompts are obtained with a moderation model and include unsafe requests, compliant boundary cases, and helpful-only prompts.
  • The experiments use a simplified content policy rather than a comprehensive production-level policy.
  • The behavior policy maps content-policy categories to ideal response types such as hard refusals and soft refusals.

4 Rule-Based Rewards for Safety

Rule-Based Rewards convert detailed safety policies into composable proposition-based features, then combine a learned safety reward with a helpful-only reward for RL training. The method uses synthetic comparison data and a small manually labeled Gold set to fit and tune the reward.

  • 4.1 Elements of RBRs: RBRs simplify multilayered safety policies into binary propositions and rules that rank completions according to desired or undesired behavior combinations.Examples include whether a completion refuses and whether it contains specific refusal characteristics.
  • 4.1 Elements of RBRs: LLM graders estimate proposition probabilities from few-shot classification prompts, while class probabilities provide additional features for the reward model.The implementation uses 20 Hard-Refusal, 23 Soft-Refusal, and 18 Comply features.
  • 4.1 Elements of RBRs: A synthetic pipeline generates labeled completions from safety prompts, producing a 518-completion Gold set for classification-prompt tuning and comparison data for fitting RBR weights.The generated completions are filtered for desired traits before being used as comparison or SFT data.
  • 4.2 Inner Loop: Fitting an RBR: The RBR is a linear model over proposition and class features, and its weights are optimized so the combined reward reproduces policy-based rankings.Weight fitting uses a hinge loss on completion pairs with known preference orderings and requires classification prompts, a default reward model, and comparison data.
  • 4.2 Inner Loop: Fitting an RBR: RBR fitting is computationally lightweight because its parameter count is much smaller than that of a standard RLHF reward model.The authors also evaluate rankings on held-out comparison data before running RL to identify needed feature or model changes.
  • 4.3 Outer Loop: Evaluating the Final Reward Signal and Tuning: Combining safety RBRs with a helpful-only reward model greatly reduces errors where non-ideal completions outrank ideal ones across response types.The error metric counts only comparisons involving an ideal completion, excluding rankings among two non-ideal completions.

5 Experiments

The experiments test whether RBR training improves safety, reduces over-refusals, uses human data efficiently, and corrects over-refusal incentives. Evaluation combines internal, external, human, capability, and tradeoff measurements across several baselines and model sizes.

  • Experimental Questions: RBR training is evaluated against helpful-only and human-safety-data baselines using internal safety and human evaluation metrics.The experiments also compare models under external safety evaluations and capability benchmarks.
  • Experimental Settings: Evaluation averages results across 5 late-PPO checkpoints, while experiments span Large, Medium, Small, and XSmall policy-model sizes.Main experiments use Large-sized graders and automated evaluators, with synthetic data sampled from Large models.
  • Evaluation: The evaluation measures unsafe content, over-refusals, and hard-refusal style using RBR-based internal metrics.Internal prompts include Comply, Hard-Refusal, and Soft-Refusal cases distributed across policy categories.
  • Evaluation: The internal evaluation partially overlaps with the training signal and may retain correlated errors despite disjoint training and evaluation prompts.The evaluation uses thresholded proposition probabilities rather than fitting weights and supplements results with Gold-set accuracy and additional evaluations.
  • Evaluation: RBR-based evaluation is supplemented with XSTest, WildChat, human safety evaluations, and capability benchmarks including MMLU, HellaSwag, GPQA, and Lambada.XSTest targets over-refusal-sensitive prompts, while WildChat provides unsafe prompts evaluated with multiple automated tools.
  • Evaluation: Figure 5 measures the tradeoff between usefulness, defined as not over-refusing, and safety, defined as not containing disallowed content.Higher scores indicate better performance on both dimensions.

6 Results

RBR-PPO improves safety while limiting over-refusals, preserves capability performance, and remains effective across reward models and training configurations. Ablations identify trade-offs involving grader size, safety-prompt exposure, refusal composition, and omitted training components.

  • Main results: RBR-PPO substantially increases safety while minimally affecting over-refusals, whereas Human-PPO increases over-refusals by almost 14% in human evaluation.The comparison is reported on Large models using internal human and automated safety evaluations.
  • Capability evaluation: Safety RBRs maintain performance on MMLU, Lambada, HellaSwag, and GPQA compared with the Helpful-PPO baseline.The capability comparison covers Large PPO models across four common benchmarks.
  • Reward-model robustness: Applying RBRs to overcautious or outdated reward models improves safety while reducing over-refusals by 16% and 10%, respectively.The 16% reduction applies to HumanRM+RBR-PPO versus Human-PPO; the 10% reduction applies to the outdated-safety-data model.
  • RBR Training Ablations: Larger grader engines reduce over-refusals while keeping safety roughly constant, with the largest model recovering both refusal performance and style.Hard-refusal style follows a U-shaped pattern as grader capability increases.
  • RBR Training Ablations: Increasing safety-prompt exposure raises safety but slightly increases over-refusals, while changing the Hard-Refusal/Comply ratio reveals a safety–over-refusal trade-off.Refusal style benefits most from seeing more safety prompts.

7 Discussion

The discussion contrasts RBRs with human-data reward modeling, emphasizing faster correction and direct rule enforcement while identifying data-distillation and scope limitations.

  • 7.1 Challenges of RBRs vs RLHF-style Human Data: RBRs directly penalize over-refusals during training, avoiding reward-model distillation errors that can make safe prompts receive nearly the same reward as helpful completions.Human-data reward models required correction after over-refusal caused by imbalanced refusal examples.
  • 7.1 Challenges of RBRs vs RLHF-style Human Data: Correcting RBR classification prompts generally takes less time than discarding or recollecting human data after a problem is discovered.The paper notes that human-data issues may only become visible after PPO training, whereas RBR corrections affect all data immediately.
  • 7.2 Limitations, Future Work, and Ethical Considerations: RBRs are most applicable when desired behaviors can be expressed as explicit, easy-to-judge propositions and rules.The authors identify more subjective tasks, such as high-quality essay writing, as harder settings for RBRs.
  • 7.2 Limitations, Future Work, and Ethical Considerations: RBRs can be combined with human preference data to discourage detectable safety failures while preserving capabilities through a helpfulness reward model.The paper describes using RBRs to discourage refusals on safe prompts while retaining a helpful RM.
  • 7.2 Limitations, Future Work, and Ethical Considerations: Moving safety feedback from humans to LLMs reduces human supervision and may magnify model biases, requiring careful evaluation of accuracy and bias.The authors suggest combining RBRs with human data as one mitigation.

8 Conclusion

The conclusion presents RBRs as an automated, modular approach for safety training that uses little human data and can be updated as desired behavior changes. Experiments show accurate safety behavior while balancing safety and usefulness against helpful-only human-safety baselines.

  • 8 Conclusion: RBRs provide a cost- and time-efficient, minimally supervised approach that is easy to update when desired model behavior changes.The method decomposes ideal behavior into fine-grained modular rules that support classification accuracy and synthetic response generation.
  • 8 Conclusion: RBRs generally achieve accurate safety behavior while balancing safety and usefulness against helpful-only human-safety baselines.
  • 8 Conclusion: The method is an automated AI-feedback preference-modeling approach for safety training in large language models.

A Appendix / supplemental material

The appendix includes a glossary of terms used throughout the paper.

  • A Appendix / supplemental material: Table 7 provides definitions for terms used throughout the text.

A.1 Data, Training and Results Details

The supplemental material points readers to glossary, statistical, experimental-setting, sampled-completion, and reward-distribution resources.

  • A.1 Data, Training and Results Details: Table 7 provides a glossary of terms used throughout the paper.
  • A.1 Data, Training and Results Details: Table 10 reports standard errors for numerical results shown in the main-text figures.
  • A.1 Data, Training and Results Details: Tables 11 and 13 provide experimental settings and sampled completions for different behaviors, while Figure 9 plots reward-distribution histograms.

A.1.1 RBR Classes

RBR organizes propositions about desired and undesired completion behavior into five common classes. These classes range from ideal responses to completions containing disallowed content.

  • A.1.1 RBR Classes: RBR combines propositions for each completion type into five common classes shared across hard refusal, safe completion, and comply behaviors.The classes organize multiple propositions into meaningful labels, though defining classes is not required for RBRs.
  • A.1.1 RBR Classes: The ideal class represents desired behavior without disallowed content.
  • A.1.1 RBR Classes: The minimum_acceptable_style class permits imperfect stylistic traits while retaining desired behavior and avoiding disallowed content.
  • A.1.1 RBR Classes: The unacceptable_completion class contains undesired behavior but remains logical and free of disallowed content.
  • A.1.1 RBR Classes: The illogical_completion class represents an illogical continuation of the conversation.
  • A.1.1 RBR Classes: The disallowed_completion class contains disallowed content somewhere in the completion.
  • A.1.1 RBR Classes: Table 15 specifies how each proposition maps to these classes.

A.1.2 Prompt Breakdown by Response Type

The appendix details prompt labeling, weight fitting, alternative fixed-weight settings, policy and proposition construction, and supporting experimental materials. Results indicate that weight choice affects the balance between safety and over-refusal, while RBR variants achieve high F1 scores.

  • A.1.2 Prompt Breakdown by Response Type: The prompt breakdown compares human and automatic labels across behavior categories and reports agreement rates for response types.
  • A.1.3 Weight Fitting Hyperparameter Details: Weight fitting uses Adam for 1000 steps with learning rate 0.01 and weight decay 0.05 after testing nearby learning rates.
  • A.2 Alternative Weights: Hand Set Weights: Fixed weights generally produce more over-refusals than optimized weights, although they can provide higher safety.RBR-Fixed10-PPO has safety similar to Human-PPO while over-refusing less.
  • A.3 Content Policies, Model Behavior Policies, and RBR Propositions: The appendix gives an example content policy covering only a limited number of harms and not representing a real content policy.
  • A.3 Content Policies, Model Behavior Policies, and RBR Propositions: Safety RBR documentation lists propositions, their desirability by behavior type, weight-fitting usage, and grader-size accuracy and error rates.
  • A.3 Content Policies, Model Behavior Policies, and RBR Propositions: The appendix includes raw plotting results, experimental model sizes, example samples, proposition tables, and feature definitions used in weight fitting.
  • A.2 Alternative Weights: Hand Set Weights: RBR-Fixed10-PPO reaches 92.9 ± 0.9% F1, while RBR-Fixed1-PPO reaches 93.1 ± 0.9% and RBR-FixedOpt-PPO reaches 89.5 ± 1.1%.The fixed-weight settings trade off safety and over-refusal differently from optimized weights.
  • A.3 Content Policies, Model Behavior Policies, and RBR Propositions: The ablation results report 92.3 ± 0.9% F1 for SFTOnly-noRBR-PPO, 92.7 ± 0.9% for RBR-noRM-PPO, and 92.2 ± 0.9% for RBR-noSFT-PPO.F1 balances avoiding unsafe content against minimizing over-refusal.
Loading 2411.01111v1…