Source-linked AI summary
Secrets of RLHF in Large Language Models Part II: Reward Modeling
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
TL;DR
Reward models in RLHF must handle noisy preference labels and generalize beyond the distributions on which they were trained. This report addresses these issues with preference-strength modeling, contrastive learning, and meta-learning, reporting improved alignment performance and consistent language-model improvement across 3 to 4 rounds on two datasets.
Problem
Reward models face incorrect or ambiguous preference labels and poor generalization from specific training distributions to out-of-distribution examples.
Method
The report uses multi-reward-model voting to measure preference strength, label correction and smoothing, adaptive margins, contrastive learning, and meta-learning.
Results
The proposed methods improve final alignment performance, transfer reward modeling to out-of-distribution data, and produce consistent language-model improvement in 3 to 4 rounds on Anthropic’s HH-RLHF and OpenAI’s summarization datasets.
Takeaways & Limitations
Reward-model training can be improved by addressing preference quality and shifted output distributions, supporting more stable reinforcement learning and iterative RLHF.
Takeaways & Limitations
The report acknowledges incomplete and less rigorous evaluation, fixed model sizes, and the absence of new preference data.
Abstract
from arXiv · showhide
Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.
1 Introduction
The report targets two practical weaknesses of reward models in RLHF: noisy or ambiguous preference data and poor generalization beyond the training distribution. It proposes preference-quality methods plus contrastive and meta-learning approaches to improve reward modeling and iterative alignment.
- Motivation: Reward models serve as proxies for human preferences in RLHF, which trains them from preference data before optimizing the language model with reinforcement learning.RLHF first trains a reward model and then optimizes the language model to maximize its reward.
- Challenges: Low annotator agreement introduces incorrect and ambiguous preferences, while distribution-specific training can cause poor performance on out-of-distribution examples.The reported agreement among annotators is about 0.6 to 0.7.
- Data quality: A multi-reward-model voting metric identifies preference strength and supports correcting wrong labels, smoothing ambiguous labels, and applying adaptive margins.These methods are intended to reduce overfitting to low-quality data and improve discrimination between similar responses.
- Generalization: Contrastive learning improves the reward model’s ability to distinguish subtle preference differences, while meta-learning adapts it to shifted output distributions.The combined approach is designed to support transfer to out-of-distribution data and iterative RLHF.
2 How Data Impacts the Modeling of Human Preference?
Preference data varies in quality: incorrect, ambiguous, and strong pairs affect reward-model training differently. The paper measures preference strength with multiple reward models and uses label correction, smoothing, and adaptive margins to improve robustness across validation sets and PPO training.
- Preference data and reward modeling: RLHF reward modeling trains on chosen-versus-rejected comparisons, but preference labels can be noisy because annotator agreement is limited.The pipeline samples two outputs, obtains human preferences, and fits a reward function using pairwise comparisons.
- Measuring preference strength: An ensemble-based preference-strength metric uses reward-score means and standard deviations to identify potentially incorrect, ambiguous, and strong preferences.The method trains multiple reward models and measures agreement in their preference differences for each pair.
- Measuring preference strength: The 500 highest-strength validation pairs matched GPT-4 annotations at 0.956, versus 0.164 for the 500 lowest-strength pairs and 0.544 near zero.Higher preference strength therefore coincided with greater consistency between original and GPT-4 labels.
- Impacts of data quality: The lowest-strength 20% harmed validation performance, near-zero-strength data performed around 0.5 accuracy, and most remaining data improved performance.The highest-strength 10% did not produce the best performance when trained alone, distinguishing useful strong data from sufficient data coverage.
- Mitigating data-quality effects: Label flipping, soft labels, and adaptive margins mitigate noisy or overfit preference signals, with adaptive margins improving preference modeling across data.Soft labels prevent rapid loss convergence and encourage robust features, while denoising methods are stable across three validation sets.
- Implications for RL training: Denoising reward models produced more stable PPO behavior, whereas baseline and margin methods showed late-stage KL divergence growth, fluctuations, or overfitting to noise.The denoising methods showed stable performance across validation sets and more linear KL-divergence increases during PPO.
3 Preference Generalization and Iterated RLHF
This section improves reward-model discrimination and generalization using contrastive learning and MetaRM, which adapts training to policy-induced distribution shifts. Experiments report more stable training, stronger response discrimination, and continued out-of-distribution performance, while gains eventually decline across some in-distribution rounds.
- 3 Preference Generalization and Iterated RLHF: Contrastive learning addresses similar features between chosen and rejected responses by learning representations that distinguish subtle preference differences.The approach uses preference-pair or preference-difference samples and can combine contrastive loss with the original reward-model loss.
- 3 Preference Generalization and Iterated RLHF: The total reward-model objective combines the original RM loss with a weighted contrastive-learning loss, using methods such as SwAV or SimCSE.The contrastive component is intended to improve recognition of subtle variations and similarities in the data.
- 3 Preference Generalization and Iterated RLHF: Contrastive-learning reward models produce more stable training rewards and returns, and their downstream RLHF models are evaluated for harmlessness and helpfulness against baselines.The cited results describe greater training stability but do not provide the corresponding numerical evaluation values.
- 3 Preference Generalization and Iterated RLHF: MetaRM trains the reward model to fit original preference pairs while increasing differentiation among responses sampled from the shifted policy distribution.Its meta-process applies difference-loss ascent on shifted-distribution samples before the vanilla reward-model update.
- 3 Preference Generalization and Iterated RLHF: Each MetaRM round markedly outperforms SFT, but win rates decline in later rounds for dialogue generation and summarization, indicating task-dependent limits.The report also states that MetaRM outperforms other baselines and that GPT-4 and human assessments align closely.
- 3 Preference Generalization and Iterated RLHF: MetaRM continues to outperform baselines on out-of-distribution tasks, although its win rate is slightly lower than in-distribution evaluation.The report attributes this decline to query distribution shift and states that OOD alignment can avoid cost-intensive preference labeling for a new domain.
4 Related Work
Related work frames reward models as the mechanism that incorporates human preferences into RLHF while emphasizing noisy feedback and dataset-specific generalization as persistent challenges. The report presents these issues as important for advancing the reliable and ethical application of RLHF.
- 4 Related Work: Reward models act as reward functions that guide AI-system optimization toward objectives aligned with human preferences.They are described as a core component of RLHF for integrating human feedback into learning.
- 4 Related Work: Human feedback can be noisy, ambiguous, conflicting, biased, or misaligned with the intended objectives, reducing reward-model accuracy and effectiveness.The cited discussion identifies evaluator goals and perspectives as potential sources of this uncertainty.
- 4 Related Work: Reward models trained on specific datasets may perform well in-distribution but struggle to preserve their learned preferences on unseen tasks, contexts, or environments.This limitation is especially relevant when reward learning and policy training are separated across environments.
- 4 Related Work: Addressing noise, ambiguity, bias, and generalization limits is presented as important for the advancement and ethical application of RLHF systems.The conclusion connects these challenges to the broader development and application of RLHF.
Discussion
The report focuses on improving reward models for RLHF and presents training analyses, while acknowledging evaluation and data limitations. It also reports applications to translation, code, and reasoning.
- Discussion: RLHF was also explored for translation, while outcome-based rewards were investigated as an approximation to process supervision in code and reasoning.
- Discussion: The report targets more robust reward models to better align language models with human intentions.Its stated motivation is that robust reward modeling remains important but comparatively underexplored.
- Discussion: The authors emphasize practical analysis with straightforward methods and detailed reward-model and PPO training processes.They state that methodological innovation is not the primary focus and that training details are valuable.
- Discussion: The report has incomplete and less rigorous evaluation, fixed model sizes, and no new preference data.
A.1 Reward Inflation during Training
Reward inflation can occur during reward-model training: scores increase and loss decreases without meaningful improvement in distinguishing chosen from rejected responses.
- A.1 Reward Inflation during Training: Decreasing training loss and increasing reward scores do not necessarily improve differentiation between chosen and rejected samples.The reported inflationary phenomenon appears during prolonged training.
- A.1 Reward Inflation during Training: At 5000 and 10000 steps, reward scores show noticeable amplification while the reward-score difference remains largely unchanged.
- A.1 Reward Inflation during Training: The unchanged reward-score difference indicates that the baseline reward model’s performance does not significantly improve despite lower training loss.
B Experiment Details
Experiments use Llama 2 with 7 billion parameters as the foundational model, primarily on general dialogue tasks, with additional meta-learning experiments for summarization.
- B Experiment Details: Llama 2 with 7 billion parameters is used as the foundational model across all experiments.
- B Experiment Details: The primary experiments evaluate general dialogue tasks.
- B Experiment Details: Additional meta-learning experiments are conducted on the summarization task.
B.1 Dataset
The experiments use dialogue, summarization, and out-of-distribution prompt sources to evaluate reward modeling across datasets and domains.
- B.1 Dataset: The dialogue setup uses 96k filtered ShareGPT conversations spanning mathematics, knowledge querying, and coding.
- B.1 Dataset: Anthropic-RLHF-HH supplies 170k helpfulness and harmlessness comparisons for AI-assistant responses.Ten percent is reserved for validation.
- B.1 Dataset: The summarization setup uses 123,169 Reddit posts paired with human-authored summaries.
- B.1 Dataset: For out-of-distribution evaluation, helpfulness prompts come from Oasst1 and harmlessness prompts from PKU-SafeRLHF.Oasst1 contains over 10k human-annotated assistant-style conversations.
B.2 Implementation Details
The implementation specifies training settings for SFT, reward modeling, and PPO-based RL fine-tuning, including distribution, precision, optimization, and sampling choices.
- Infrastructure: Training used 8 A100-SXM-80GB GPUs with data parallelism, bfloat16 automatic mixed precision, and DeepSpeed ZeRO.
- SFT Phase: SFT used a global batch size of 32, learning rate 2e−5, one epoch, and 10% warm-up before decay.
- RM Training: Reward modeling used learning rate 5e−6, with batch size 16 for contrastive methods and 32 for other methods.Contrastive methods applied dropout-based augmentation at rate 0.05.
- RL Fine-tuning: PPO used actor and critic learning rates of 5e−7 and 1.5e−6, respectively, for 2000 iterations with batch size 32.Each query generated four rollout samples per GPU using nucleus sampling.
- RL Fine-tuning: The critic initialized from reward-model weights, with λ=0.95, γ=1, and meta-learning settings using KL coefficient β=0.05 and reward clipping 0.8.
- Baselines: Baselines included the SFT model and a PPO model trained with the vanilla reward model.
B.4 Evaluation
The evaluation compares models on helpfulness and harmlessness using human judgments, supplemented by GPT-4-based pairwise assessment with prompts designed to reduce evaluation bias.
- Human Evaluation: Human evaluation used 100 randomly selected test prompts for helpfulness and 100 red-teaming prompts for harmlessness.Evaluators judged which response was higher quality, more useful, and harmless.
- GPT-4 Evaluation: GPT-4 evaluation was added because prior studies reported close alignment with human judgments, alongside greater efficiency and consistency.
- GPT-4 Evaluation: The evaluation used a dedicated GPT-4 prompt instructing an impartial judge to select the more helpful and harmless assistant response.
- GPT-4 Evaluation: The GPT-4 instructions allowed a tie response and required the final word to be the selected choice.
- Prompt Format: Responses were presented between explicit start and end markers for assistants A and B.
- Prompt Format: The evaluation materials included a summarization GPT-4 win-rate prompt and a separate harmlessness comparison instruction.
C.1 Data Selection
Data-selection experiments retrained reward models on progressively expanding subsets to examine how incorrect preferences affect performance.
- Data Selection: Each point represented a reward model retrained from scratch on a progressively expanding data subset using the base model’s hyperparameters.The experiments show that incorrect preferences have a detrimental impact on reward-model training.
C.2 Supplementary experiments regarding margin and soft labels
Supplementary experiments examine how preference strength, label correction, soft labels, and adaptive margins affect reward-model learning, while examples extend RLHF preference alignment to translation and programming.
- Margin and Soft Labels: Combining soft labels with an adaptive margin outperformed using either soft labels or margin alone on the lowest-preference-strength data.The lowest 10% was treated as mostly incorrect and its labels were flipped before testing these methods.
- Preference Strength: Increasing the share of lowest-strength data changes reward-model performance, and incorrect preferences require substantial high-quality data to offset their negative impact.
- Preference Strength: When erroneous preferences were excluded, validation accuracy did not decrease as training steps increased while the strongest-preference data proportion rose.
- Margin and Soft Labels: Adaptive margins slightly improved differentiation of similar responses, whereas soft labels alone did not improve it.This motivated using an adaptive margin in the reward loss for all data.
- Label Correction: For the lowest 10% preference-strength data, label flipping and soft-label correction both mitigated the impact of incorrect preferences.
- Training Dynamics: Label flipping made probability distributions concentrate near 0 and 1 around 5k steps, compared with about 10k steps for the baseline.
- Validation Analysis: Performance differences between training methods mainly arose from ambiguous and potentially incorrect preferences in the validation set.
- Validation Analysis: Label flipping and soft labels reduced learning of incorrect and ambiguous preferences while improving performance on normal data.