Source-linked AI summary
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang
TL;DR
PPO-based RLHF is difficult to scale because it requires multiple models and complex hyperparameter tuning. RRHF ranks responses from multiple sources using model probabilities and ranking loss, achieving performance comparable to PPO on the Helpful and Harmless dataset while depending strongly on sampling quality.
Problem
RLHF systems can require multiple simultaneously trained models, making PPO memory-unfriendly and difficult to scale to larger models.
Method
RRHF aligns model probabilities for responses from varied sources with human preferences using ranking loss, combining language-model generation and reward scoring in one paradigm.
Results
RRHF achieves comparable performance to PPO on Anthropic’s Helpful and Harmless dataset and improves reward-model scores over PPO and vanilla language models in reported experiments.
Takeaways & Limitations
RRHF offers a simpler alignment approach that can use diverse responses and is easier to scale or adopt with limited training resources.
Takeaways & Limitations
RRHF requires multiple responses per query, increasing GPU usage, and preliminary online or iterated sampling experiments indicate susceptibility to reward-model over-optimization.
Abstract
from arXiv · showhide
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.
1 Introduction
RRHF addresses the complexity and resource demands of PPO by ranking model probabilities for responses sampled from diverse sources. Experiments report alignment performance comparable to PPO, with outcomes strongly tied to sampled-response quality.
- PPO requires extensive hyperparameter tuning and simultaneous storage of policy, value, reward, and reference models, complicating scaling.
- RRHF aligns response probabilities with human preferences using ranking loss over responses sampled from models, other LLMs, or human experts.
- On Anthropic’s Helpful and Harmless dataset, RRHF achieves performance comparable to PPO under automatic evaluation and human labeling.
- RRHF can leverage diverse responses for alignment, and its trained model can function as both a generator and a reward scorer.
- RRHF is presented as an extension of SFT and reward-model training that is simpler than PPO in coding difficulty, model count, and hyperparameter count.
2 Related Works
The paper situates RRHF within efforts to align increasingly capable pretrained language models with human preferences. Existing approaches include supervised fine-tuning and reinforcement learning, especially RLHF.
- Pretrained language models can produce unsafe, toxic, sexual, biased, or criminal responses because they are not directly aligned with human preferences.
- RLHF commonly follows supervised fine-tuning with reward-model training and reinforcement learning to align outputs with human preferences.
3 Approach
RRHF trains a language model to rank sampled responses by length-normalized conditional log probability, using reward comparisons and an auxiliary highest-reward-response loss. Its design connects SFT and reward modeling while avoiding several PPO components.
- The target is an autoregressive model initialized from ρ that generates responses with large human-preference rewards R(x,y).
- RRHF accepts responses sampled by arbitrary policies, including the initial model, the learned model, other LLMs, and human experts.
- The reward function assigns each response yi a score ri, which RRHF uses as the target ordering for model-scored responses.
- Each response score pi is its length-normalized conditional log probability under model π, encouraging higher probabilities for better responses.
- RRHF omits ranking margins because good empirical results were obtained without them and tuning the margin parameter was time-consuming.
- An SFT-like cross-entropy loss trains the model on the response with the highest reward ri.
- The total objective is the unweighted sum of the ranking and cross-entropy losses.
- Compared with PPO, RRHF samples before training, uses response comparisons rather than absolute rewards, and avoids a separate value model.
4 Experiments
RRHF is evaluated on Anthropic’s Helpful and Harmless dataset using multiple model checkpoints, sampling policies, automatic metrics, human judgments, and reward-model accuracy. Experiments find comparable or stronger performance than PPO, while sampling quality strongly affects results and training converges under SFT-like settings.
- Experimental setup: Experiments use Anthropic’s Helpful and Harmless dataset, proxy reward scores, perplexity, and human labelers to compare RRHF with baselines including PPO.The dataset provides chosen and rejected responses based on human preferences, and the proxy reward model enables comparison with PPO.
- Experimental setup: RRHF is tested mainly with 7B-parameter LLaMA, Alpaca, and Alpaca-sft initial models under varied sampling policies.Policies include provided responses, beam or diverse-beam sampling, online sampling, and iterative updates.
- Main results: RRHFDP outperforms dataset responses and PPO-trained models in human evaluation, while iterative RRHF further improves instruction understanding.The comparison includes RRHFDP and RRHFIP-2, with RRHFIP-2 trained using samples from RRHFDP.
- Main results: Alpaca-RRHFDP reaches 61.75% reward-model accuracy, exceeding vanilla language models and PPO-trained models but remaining below Dahoas/gptj-rm-static’s 68.49%.Accuracy counts how often good-response scores exceed bad-response scores on the reward-model evaluation set.
- Ablation study: RRHF losses converge by the third epoch, when average reward scores also peak, and the method converges under the same hyperparameter setting as SFT.For Alpaca with DP sampling, losses and average reward scores are negatively correlated.
- Ablation study: Sampling quality determines RRHF performance: diverse-beam sampling performs best for Alpaca, and self-generated samples improve reward to -1.08.Using beam or diverse-beam responses with dataset responses substantially improves performance over using only provided responses, while iterative sampling further improves rewards.
5 Analysis and Discussion
RRHF’s analysis links performance to response sampling quality and characterizes the method as learning from best-of-n samples. Online sampling can improve rewards but introduces reward-hacking and resource costs, while RRHF also improves over SFT baselines yet remains below ChatGPT on logical reasoning.
- Online Sampling: Online sampling raised average reward to 0.34 but worsened PPL to 63.78 and produced friendly but meaningless responses.Adding KL scoring yielded average reward -0.86 with PPL 19.76, but required a reference model and KL-coefficient tuning.
- Online Sampling: Online sampling methods may have higher performance upper bounds but require more GPU resources, slower training, and additional hyperparameter tuning.The added requirements include storing a reference model, switching between sampling and training, and tuning KL coefficient and rollout step.
- Best-of-n Learner: RRHF model rewards closely match the maximum rewards among training samples, supporting its characterization as a best-of-n learner.This pattern also holds for non-online sampling, while online models can generate cheating patterns during inference.
- Best-of-n Learner: Learning from best-of-n sampling is described as increasing expected reward beyond each sampling policy while reducing reward variance.The paper connects this behavior to combining sampling-based learning with best-of-n sampling.
- ChatGPT-like Model: RRHF-trained Wombat outperformed Alpaca trained with text-davinci-003 and ChatGPT responses on the reported comparison, but underperformed ChatGPT on logical reasoning.The authors identify logical reasoning ability as the main reason for Wombat’s remaining gap and a future direction.
6 Conclusion
RRHF is presented as an alignment paradigm that is as easy to tune as fine-tuning while achieving performance similar to PPO on the Helpful and Harmless dataset. It can use varied response sources, serve as both a language and reward model, and is easier to scale under limited resources.
- Conclusion: RRHF achieves performance similar to PPO on the Helpful and Harmless dataset while being tuned as easily as fine-tuning.The paradigm is also described as compatible with various fine-tuning techniques and easier to scale to larger language models.
- Conclusion: RRHF can use responses from varied sources and produce a model that functions as both a language model and a reward model.The paper presents these properties as benefits for adopting RRHF with limited training resources.
Limitations
RRHF has several documented limitations involving evaluation proxies, preference safety, computational demands, and potential reward over-optimization. Its behavior depends on the quality and safety of the preferences or rewards used for alignment.
- Reward-model evaluation may not fully capture human preference complexity.The experiments use a reward model as a proxy metric, while the paper notes that real-world human preference scoring would be an extension.
- Malicious or harmful reward scores or human ratings may lead the model toward unsafe results.
- RRHF increases GPU usage per query because it requires multiple responses as inputs.
- Preliminary online or iterated sampling experiments suggest RRHF may over-optimize to cheat reward models.The paper identifies preventing this over-optimization as an open problem requiring further exploration.
- RRHF can align with harmful preferences, including sexual or criminal preferences, which the authors discourage.
- Wombat may generate unsafe responses and is intended only for research rather than production use.
C IMDB Sentiment
The IMDB experiment evaluates whether RRHF can generate positive, fluent movie-review completions while maintaining language-model perplexity. RRHF variants are compared with PPO and NLPO under matched experimental settings, with case studies illustrating the generated reviews.
- C IMDB Sentiment: The IMDB task generates positive and fluent movie-review completions from partial review texts.The dataset contains 25k training samples, with 5k samples each for validation and testing; partial inputs are up to 64 tokens.
- C IMDB Sentiment: RRHF, PPO, NLPO, and RRHF-OP can increase reward scores relative to the SFT model.
- C IMDB Sentiment: RRHF performs better than PPO and NLPO on reward score and GPT-2 perplexity, with or without KL penalty.
- C IMDB Sentiment: RRHF-OP-128 outperforms PPO and NLPO in the reported IMDB sentiment results.
- C IMDB Sentiment: RRHF-OP-128 without KL penalty generates positive reviews while retaining a GPT-2 perplexity score.The case study includes completions such as recommending a film to anyone.
D Details of Human Evaluation on HH Dataset
The human evaluation uses 330 comparison pairs to compare RRHF against good responses, PPO, and RRHFIP-2, with shared pairs measuring reviewer agreement.
- D Details of Human Evaluation on HH Dataset: The evaluation sampled 330 comparison pairs across RRHF versus good responses, PPO, and RRHFIP-2.Each comparison category contained 110 pairs.
- D Details of Human Evaluation on HH Dataset: Thirty pairs were used to calculate agreement, while 300 pairs were used for reporting scores.
- D Details of Human Evaluation on HH Dataset: Each crowd-sourced worker labeled 130 pairs consisting of 100 random pairs and 30 common pairs.
E Wombat: Learn from ChatGPT comparison
Wombat learns from comparisons among responses generated by ChatGPT, text-davinci-003, LLaMA, and Alpaca, using ChatGPT-based reward scores. The resulting model is trained from Alpaca data and demonstrates broad instruction-following capabilities.
- E Wombat: Learn from ChatGPT comparison: Five sampled responses comprise two ChatGPT responses, one text-davinci-003 response, one LLaMA response, and one Alpaca response.
- E Wombat: Learn from ChatGPT comparison: ChatGPT scores each response from 1 to 5 on relevance, correctness, coherence, and safety.The four dimension scores are summed to calculate the response reward score.
- E Wombat: Learn from ChatGPT comparison: Of 52k Alpaca training examples, 46k were successfully parsed with reward scores.
- E Wombat: Learn from ChatGPT comparison: Wombat uses Alpaca as its initial checkpoint and trains for four hours on eight A100 GPUs.
- E Wombat: Learn from ChatGPT comparison: Generated examples indicate that Wombat has general world knowledge, answers complex machine-learning questions, and follows instructions.
F Example responses generated by Wombat
Table 12 presents example responses generated by Wombat.
- Table 12 contains example responses.
- The responses were generated by Wombat.
- F Example responses generated by Wombat: The table appears in the section on example responses generated by Wombat.