Source-linked AI summary
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash
TL;DR
RLHF aligns language models effectively, but high-quality human preference labels are expensive, motivating evaluation of AI-generated feedback. The paper compares RLAIF with RLHF across three tasks, studies self-improvement and direct reward generation, and finds comparable or better performance for RLAIF. It also identifies prompting and labeler-scaling techniques that affect alignment with human preferences.
Problem
RLHF depends on expensive high-quality human preference labels, and prior RLAIF work did not directly establish whether AI feedback could replace human feedback.
Method
The paper compares RLAIF and RLHF across three text-generation tasks, studies same-size and same-checkpoint AI labelers, introduces d-RLAIF, and evaluates techniques for aligning AI preferences with human preferences.
Results
RLAIF performs comparably to RLHF across three tasks, improves over SFT with same-size or identical-checkpoint labelers, and d-RLAIF matches or outperforms canonical RLAIF.
Takeaways & Limitations
RLAIF is presented as a viable alternative to RLHF that does not depend on human annotation and offers appealing scaling properties.
Takeaways & Limitations
AI-generated feedback may transfer and amplify off-the-shelf LLM biases, so human experts remain the gold standard in high-stakes domains.
Abstract
from arXiv · showhide
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.
1. Introduction
The paper compares RLAIF with RLHF across summarization, helpful dialogue, and harmless dialogue, finding comparable or better human-evaluated performance. It also studies self-improvement, direct reward generation, and methods for aligning AI preferences with human preferences.
- 71% and 73% of evaluators preferred RLAIF and RLHF over SFT for summarization, while 63% and 64% did so for helpful dialogue generation.
- For harmless dialogue generation, RLAIF, RLHF, and SFT achieved harmless rates of 88%, 76%, and 64%, respectively.The task uses independent harmlessness ratings rather than pairwise win rates.
- RLAIF can improve an SFT policy when the AI labeler matches the policy size or is the exact same checkpoint as the initial policy.The exact-checkpoint setting demonstrates strict LLM self-improvement.
- Direct-RLAIF obtains rewards directly from an off-the-shelf LLM during reinforcement learning and matches or outperforms canonical RLAIF.This technique circumvents separate reward-model training.
- Chain-of-thought reasoning consistently improves alignment between AI-generated and human preferences, while detailed preambles and few-shot prompts help only on certain tasks.The paper also studies trade-offs between labeler size and alignment with human preferences.
- RLAIF achieves comparable performance to RLHF across summarization, helpful dialogue generation, and harmless dialogue generation.Human evaluations compare both methods across all three tasks.
2. Methodology
The paper uses off-the-shelf LLMs to generate preference distributions, trains RLAIF reward models from those labels, and also evaluates direct-RLAIF, which supplies LLM-derived rewards during reinforcement learning. Its labeling pipeline addresses position bias and can elicit chain-of-thought reasoning before scoring candidates.
- Preference labeling with LLMs: LLMs label candidate responses by producing a preference distribution over “1” and “2” from the tokens’ log-probabilities.The prompt includes instructions, optional few-shot exemplars, a sample to annotate, and an ending that elicits the preference.
- Preference labeling with LLMs: Two inferences reverse candidate order and average their outputs to mitigate position bias, which is especially prevalent in smaller LLM labelers.
- Preference labeling with LLMs: Chain-of-thought labeling first elicits an explanation of candidate quality, then appends that response to the original prompt before computing preferences.The procedure can use zero-shot or few-shot reasoning exemplars.
- RLAIF: Canonical RLAIF trains a reward model on soft LLM preference labels and uses that model to assign rewards during reinforcement learning.The reward model is trained with cross-entropy on the softmax of its scores, making the procedure a form of model distillation.
- Direct-RLAIF: d-RLAIF directly uses an off-the-shelf LLM’s normalized quality score as the reward, avoiding reward-model training and its staleness issue.The LLM scores generations from 1 to 10; the likelihood-weighted score is normalized to [−1, 1] before reinforcement learning.
- Evaluation: The evaluation uses AI Labeler Alignment, human-evaluated Win Rate, and Harmless Rate.Harmless Rate is used for harmless dialogue because many responses are equally safe and difficult to rank relatively.
3. Experimental Details
Experiments use downsampled preference datasets and PaLM 2 models, with reward models trained from PaLM 2 XS and policies initialized from supervised fine-tuned models. Human evaluators assess held-out generations, while one explored dataset showed no meaningful improvement after correcting for length bias.
- Datasets: The experiments cover Reddit TL;DR summarization and Anthropic helpful and harmless dialogue preference datasets.The datasets contain posts with summaries or conversations with paired assistant responses and preference information.
- Scope boundary: On the Stanford Human Preferences dataset, neither RLHF nor RLAIF meaningfully improved over SFT after correcting for length bias.
- Datasets: After downsampling each training split and filtering summarization examples for high-confidence human preferences, each task retained 3–4k examples.AI labeler alignment was calculated on these downsampled datasets.
- Models: PaLM 2 models provide AI labels, with PaLM 2 Large used by default and instruction tuning but no prior reinforcement learning.
- Models: Supervised fine-tuned models start from PaLM 2 Extra-Small, while reward models are trained from PaLM 2 XS checkpoints.Summarization uses task-specific Reddit TL;DR fine-tuning; other tasks use instruction-tuned models.
- Training: Reward models are fine-tuned on full preference-dataset training splits using AI labels for RLAIF and human labels for RLHF.
- Training and evaluation: Policies use modified REINFORCE with a baseline, and both policy and value models are initialized from the supervised fine-tuned model.Evaluation contexts come from test splits not used for training or other evaluation.
4. Results
Across the evaluated tasks, RLAIF performs on par with or better than RLHF, while same-size labelers and direct reward querying support improvement over SFT. Prompting and labeler-scale experiments further identify ways to improve AI–human preference alignment.
- RLAIF vs. RLHF: RLAIF matches or outperforms RLHF across summarization, helpful dialogue, and harmless dialogue generation.
- RLAIF vs. RLHF: 88% harmless rate puts RLAIF above RLHF at 76% and SFT at 64% for harmless dialogue generation.
- Towards Self-Improvement: 68% of human annotators preferred same-size RLAIF over SFT, compared with 71% for RLAIF using a larger labeler.
- Direct-RLAIF: 74% of annotators preferred d-RLAIF over SFT, and d-RLAIF beat same-size RLAIF at 60%.
- Direct-RLAIF: 66% win rate over SFT established strict LLM self-improvement when d-RLAIF used the same checkpoint for feedback and the starting policy.
- Prompting Techniques: Chain-of-thought reasoning generally improves alignment, while detailed preambles and few-shot prompting have task-dependent effects.
- Size of LLM Labeler: AI-labeler alignment falls 4% when replacing PaLM 2 L with PaLM 2 S and another 11% with PaLM 2 XS.
5. Qualitative Observations
RLAIF and RLHF often produce similar summaries, but qualitative inspection reveals distinct failure patterns in hallucination and fluency. A small evaluation found no statistically significant difference, leaving broader analysis for future work.
- RLHF sometimes hallucinated details absent from the source text, whereas RLAIF did not in the inspected examples.
- RLAIF sometimes produced less fluent summaries, including run-on sentences and repeated phrases.
- 70-example blind ranking found no statistically significant difference between RLHF and RLAIF on accuracy, coverage, or coherence.
6. Related Work
Prior work established RLHF for aligning language models with human preferences and RLAIF for combining AI- and human-labeled preferences. The paper addresses the unresolved direct comparison between AI and human feedback.
- RLHF trains a reward model on pairwise human comparisons and has been applied to summarization, instruction following, dialogue, and question answering.
- Earlier RLAIF combined LLM- and human-labeled preferences to optimize helpfulness and harmlessness jointly.
7. Conclusion
The paper finds that RLAIF can match RLHF across three text-generation tasks, support self-improvement, and benefit from direct AI-provided rewards. It also identifies open questions about extending RLAIF to model-based RL and granular credit assignment.
- RLAIF achieves comparable improvements to RLHF on three text-generation tasks, with similar human preference rates in head-to-head comparisons.
- RLAIF remains effective when the AI labeler matches the policy’s size or is the exact same checkpoint as the initial policy.
- Direct-RLAIF obtains rewards directly from an LLM during reinforcement learning and outperforms the canonical setup that first trains a separate reward model.
- Open questions include adapting RLAIF to model-based reinforcement learning and using AI feedback for granular credit assignment.
Impact Statement
The paper presents AI feedback as a faster and more affordable route to model alignment, while emphasizing risks from bias transfer and malicious use. It also reviews the RLHF pipeline that RLAIF modifies.
- AI-generated preferences are more affordable and faster to produce than human labels, potentially making model alignment more accessible to developers.
- AI feedback may transfer biases from off-the-shelf LLMs into preferences, potentially amplifying bias and misaligning policies.
- The paper urges extreme caution when deploying such models in high-stakes domains including medicine, law, and employment.
- Lowering alignment barriers could facilitate misuse, including generating convincing misinformation or hateful and abusive content.
- RLHF comprises supervised fine-tuning, reward-model training, and reinforcement learning, with the reward model serving as a proxy for human preferences.
- A KL-divergence term can penalize deviation from the original supervised policy to reduce reward hacking and unnatural language.
B. Position Bias in LLM Labelers
The position-bias analysis tests whether LLM labelers preserve their preferred candidate position after the candidates are swapped. Smaller PaLM 2 labelers show substantially more position bias.
- Experimental setup: The analysis uses the “Detailed + CoT 0-shot” prompt on the summarization task and reports results by model size.
- Position-bias measure: Position bias is measured as the percentage of inference pairs where an LLM prefers the same position after candidates are swapped.
- Position-bias results: PaLM 2 L, S, and XS prefer the same position 18%, 21%, and 56% of the time, respectively, indicating more bias in smaller models.
- Position-bias results: Among same-position preferences, PaLM 2 L favors the first candidate 94% of the time, whereas PaLM 2 S and XS favor the second 91% and 99% of the time.
- Experimental setup: The broader evaluation covers summarization, helpful dialogue, and harmless dialogue, with AI labelers rating which of two candidate responses is preferred.
- Reward-model accuracy: Human-feedback reward models outperform AI-feedback reward models on holdout human preferences, but RM accuracy does not consistently predict final RL effectiveness.
I. Human Evaluation Details
Human evaluation compares responses from multiple policies across roughly 2,000 rating instances. Three independent raters assess each instance, with reasonable inter-annotator agreement.
- Human evaluation generated approximately 2k rating instances containing multiple distinct model responses, including SFT, RLAIF, and RLHF outputs.
- These instances produced approximately 6k context-response pairs and 18k context-response-rating tuples after assessment by three independent raters.
- Kendall’s W ranged from 0.6 to 0.7 across evaluation sessions, indicating reasonable agreement among raters.
J. Controlling for Response Length
The paper examines whether response length affects human preference comparisons and reports length-adjusted results for summarization and helpful dialogue. After adjustment, both RL policies continue to outperform SFT by similar margins, supporting RLAIF’s comparability to RLHF.
- 164, 161, and 132 average characters were generated by RLAIF, RLHF, and SFT summarization policies, respectively.
- Method: A logistic regression estimates length-controlled win rates from the ratio of policy-response length to comparison-summary length.The model is trained on binary human-preference labels and then used to estimate a controlled win rate.
- Summarization: 59% and 61% are the length-adjusted win rates for RLAIF and RLHF versus SFT on summarization.The authors state that both RL policies continue to outperform SFT by similar margins.
- Other tasks: The same conclusions are reported for helpful dialogue generation and several related experiments, including same-size RLAIF and direct RLAIF.
- Limitation: The post-hoc adjustment is imperfect because it assumes logistic regression accurately learns the relationship between summary length and human preference.The authors suggest encouraging similar output lengths during training as a more principled alternative.
K. Combining Human and AI Feedback
The paper studies combining human and AI feedback and reports no improvement over RLHF alone in the tested summarization setup. It also documents AI-labeling costs, prompt-based preference scoring, and reduced alignment from self-consistency sampling.
- Combining feedback: RLHF + RLAIF and RLHF achieve 71% and 74% win rates over SFT, respectively, with no statistically significant difference.Raters prefer both policies equally in head-to-head comparison.
- Combining feedback: The experiment does not show positive results from combining RLAIF and RLHF, although alternative curricula and feedback ratios remain possible.The authors leave these alternative setups for future work.
- Cost analysis: AI preference labeling is estimated at $0.06 per example, compared with $0.67 for human labeling in the analyzed summarization setting.The cost analysis excludes training, annotator expertise, and LLM-labeling setup costs.
- Self-consistency: Self-consistency decreases AI-labeler alignment across tested temperatures, with the largest drop exceeding -5% at T = 1.0.Using 4 versus 16 samples does not affect alignment in these experiments.
N. End-to-end Sensitivity to AI Labeler Alignment
The paper tests whether better AI-labeler alignment improves downstream RLAIF policies by comparing Base 0-shot and Detailed CoT 0-shot prompting. Human evaluators prefer the policy trained with the more aligned labeler, while the authors caution that further experiments are needed.
- Experiment: 76.1% and 78.0% are the AI-labeler alignment rates for Base 0-shot and Detailed CoT 0-shot prompting, respectively.
- Results: 59% of the time, human evaluators prefer summaries from Detailed CoT 0-shot over Base 0-shot.The result is statistically significantly different from 50% according to a binomial test.
- Limitation: The authors state that this study is limited and that further experiments are required for generalizable conclusions.
- Prompting: For chain-of-thought prompts, the decoded rationale is concatenated with the original prompt before scoring the preferred summary.The resulting preference distribution is obtained through the paper’s scoring procedure.
- Evaluation: The labeler evaluates summaries using coherence, accuracy, coverage, and overall quality.The prompt asks the LLM to explain which of two summaries better satisfies these axes.
- Example: An example rationale favors Summary 2 because it is judged more coherent, accurate, and complete than Summary 1.