PaperlayerView exact revision on arXiv

Source-linked AI summary

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, Bruno Castro da Silva

arXiv:2404.08555v2cs.LGcs.AIcs.CL

TL;DR

RLHF aims to align increasingly capable language models with human objectives, but its foundations remain tied to assumptions about reward modeling and feedback. The paper analyzes RLHF through reinforcement-learning principles, focuses on reward models, and reviews the literature. It identifies incorrect generalization, misspecification, feedback sparsity, and their effects on language-model performance, while proposing oracular rewards as an ideal reference.

  • Problem

    LLMs must behave predictably, safely, and trustworthily for human tasks, yet understanding RLHF remains entangled with the design choices that popularized it.

  • Method

    The paper analyzes RLHF’s assumptions, reward-modeling choices, function-approximation caveats, and training methods, supplemented by a comprehensive literature review.

  • Results

    The analysis characterizes incorrect reward generalization, model misspecification, and feedback sparsity, and examines their impact on language-model performance.

  • Takeaways & Limitations

    Oracular rewards provide an ideal reference for evaluating reward models, while reducing feedback reliance and improving reward encoding remain future directions.

  • Takeaways & Limitations

    Limited feedback combined with function approximation can cause inaccurate out-of-distribution reward generalization and arbitrary misalignment with human preferences.

Abstract

from arXiv · show

State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHF -- the reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts.

1 Introduction

RLHF addresses the challenge of aligning broadly capable language models with human objectives, but its reward-model assumptions create important data and generalization limitations. The paper analyzes these foundations and surveys the surrounding literature.

  • RLHF is presented as a method for making widely deployed LLMs more predictable, safe, trustworthy, and aligned with human objectives.
  • RLHF’s success in games and later LLM fine-tuning motivates examining how feedback-based reinforcement learning transfers to language-model alignment.
  • The paper centers its analysis on the reward function, using a Bayesian perspective to examine how reward-model design shapes RLHF.
  • The analysis identifies extensive feedback requirements and misgeneralization from limited data combined with function approximation.
  • Reward sparsity, reward-model misspecification, and imperfect rewards are analyzed for their effects on language-model performance, alongside the concept of an oracular reward.
  • The literature review covers feedback types, RLHF training methods, and alignment approaches that avoid reinforcement learning or reward models.

2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models

Pre-trained language models optimize language statistics rather than necessarily fulfilling users’ goals, motivating RLHF as an approach that infers and optimizes a latent reward. The paper frames RLHF through Bayesian reward modeling and questions its foundational choices.

  • Pre-trained language models face an objective mismatch because modeling language statistics does not necessarily fulfill a user’s specific goals.
  • RLHF differs from supervised fine-tuning by inferring an underlying user objective and optimizing it, assuming that objective can be represented and approximated as a reward function.
  • Bayesian Interpretation of RLHF: The Bayesian interpretation treats reward-model parameters as hypotheses and human-feedback data as evidence for inferring the latent reward.
  • Bayesian Interpretation of RLHF: Reward-model learning combines feedback likelihood with constraints on the hypothesis space through explicit and implicit regularization.
  • The resulting RLHF recipe uses pairwise ranking with a Bradley-Terry likelihood and PPO for reinforcement-learning optimization, while the paper encourages reconsidering these choices.

3 Formulation: Text Generation as Sequential Decision-Making

The paper models autoregressive text generation as a deterministic sequential decision process: contexts initialize states, tokens are actions, and the language model is the policy. This formulation exposes unusually large spaces and demanding generalization requirements.

  • Text generation is formulated as a sequential decision-making process to provide the foundation for reinforcement-learning algorithms.
  • An MDP consists of states, actions, transitions, rewards, and an initial-state distribution, with policies selecting actions from states over time.
  • The autoregressive process feeds the previous token together with the context into the policy to generate the next token, while a reward function scores the output.
  • In language generation, the context is the initial state, the vocabulary is the action space, each generated token is an action, and concatenation produces the next deterministic state.
  • Properties and Challenges: The formulation has an extremely large action space, exemplified by a 32K-token vocabulary, which expands the reinforcement-learning search space.
  • Properties and Challenges: Text-generation policies must generalize across an enormous support of possible contexts, and pre-training is necessary to obtain an initially useful state representation.
  • Properties and Challenges: Training and evaluation may use different reward functions because the humans providing evaluation feedback can differ from those used to train the reward model.
  • Because reinforcement learning maximizes accumulated reward, RLHF updates the language model toward more rewarding outputs, making reward definition central to fine-tuning.

4 The Role of Reward

Reward models convert human feedback into an optimizable signal that represents both task information and output preferences. Their usefulness depends on assumptions about human rewards, feedback collection, generalization, and whether estimated rewards accurately measure alignment.

  • RLHF reward learning converts human feedback into an optimizable reward that encodes task information and preferences over outputs.
  • RLHF commonly assumes a deterministic oracular reward mapping each context-output pair to a scalar preference value.
  • Human feedback can be provided as ratings or pairwise preferences, with preference feedback requiring a learned reward model before reinforcement-learning optimization.
  • Ranking N outputs yields N(N −1)/2 pairwise comparisons, while preferences avoid requiring humans to assign precise absolute scores.
  • Deterministic rewards average across feedback providers and cannot represent variability arising from personalization, dialogue adaptation, or diverse-output settings.
  • Reward models both make reinforcement-learning objectives available and reduce the cost of online feedback collection, although current systems require thousands to millions of feedback points.
  • Reward Modeling: Preference-based reward modeling uses a likelihood over human comparisons, while the paper generalizes feedback and likelihood choices beyond specific implementations.
  • Measuring Alignment: Alignment performance is measured using rewards over generated outputs, but practical estimates can be misleading when the learned reward is inaccurate on evaluated context-output pairs.

5 Inferring the Reward from Human Feedback

RLHF infers rewards from limited human-feedback data through an encoding mechanism and function approximation, but the resulting reward model is underdetermined and can misgeneralize. These limitations affect optimization through inaccurate rewards, sparse delayed feedback, and deterministic averaging of diverse preferences.

  • Reward inference: Pairwise preference and other feedback are encoded into rewards by Ω, and the general formulation covers RLHF-style methods that use reward models.The paper describes Ω as mapping feedback and text to a reward value, including common pairwise-preference encoding.
  • Reward inference: Human-feedback data cover only subsets of all possible contexts and outputs, so reward-model learning becomes an underdetermined regression problem.Multiple reward functions can perfectly fit the collected data, while almost all fail to represent the oracular reward accurately.
  • Misgeneralization: Small coverage ratios ρ and κ, together with limited function-approximator generalization, cause incorrect rewards on unseen context-output pairs.The analysis identifies both output and context coverage as determinants of generalization performance and notes that these ratios are extremely small in practice.
  • Misgeneralization: Inaccurate out-of-distribution reward extrapolation can arbitrarily misalign optimized language-model behavior with human preferences and contributes to reward hacking and hallucinations.Reinforcement-learning benefits depend on access to an accurate reward function and accurate out-of-distribution generalization.
  • Feedback limitations: Human feedback is sparse and delayed because annotators usually evaluate complete generations, increasing RL optimization difficulty and sample complexity.Feedback on partial sentences is difficult to obtain, limiting direct use of methods designed for sparse-feedback trajectories.
  • Feedback limitations: A deterministic reward that averages annotator preferences can represent no individual annotator and cannot capture uncertainty or variability across preferences.The paper identifies this as a case of reward-model misspecification.
  • Implications: Because the reward model is central to RLHF performance, its shortcomings motivate safety measures when deploying language models fine-tuned with RLHF.The paper connects reward-model imperfections directly to the need for safeguards during use.

6 Reinforcement Learning with Imperfect Rewards

This section analyzes policy-gradient RLHF with imperfect reward models, showing how reward errors on unseen contexts can cause misalignment and how KL regularization trades off robustness against performance.

  • 6.1 Policy Gradient Algorithms: PPO is an actor-critic policy-gradient algorithm that restricts parameter updates to prevent drastic changes in the policy distribution.Its clipped objective uses estimated advantages, while the update restriction improves training stability.
  • 6.2 Misalignment due to Imperfect Rewards: The KL penalty acts as a regularizer or prior that limits overoptimization of an imperfect reward model.Its weight is denoted by β and is added to the PPO training objective.
  • 6.2 Misalignment due to Imperfect Rewards: Reward-model errors on out-of-distribution contexts can make the RLHF policy arbitrarily misaligned with human preferences.The analysis compares performance on evaluation contexts with the policy trained using an oracular reward.
  • 6.2 Misalignment due to Imperfect Rewards: The performance gap increases with discrepancy from the oracular reward and is weighted by the likelihood and oracular reward of each context-output pair.This identifies where reward-model imperfections contribute most strongly to misalignment.
  • 6.2 Misalignment due to Imperfect Rewards: Increasing β can mitigate unseen-context misalignment, but it also moves the policy toward the pretrained model and lowers performance.The mitigation follows the 1/β dependence in the policy exponent, creating a direct robustness-performance trade-off.
  • 6.3 Why use Reinforcement Learning Algorithms?: Unlike SFT, reinforcement learning explores the output space and uses positive, negative, and neutral samples weighted by their rewards.These richer gradient updates are reported to improve generalization and sample efficiency.

7 Review of Reinforcement Learning from Human Feedback for Language Models

This section reviews how RLHF adapts reinforcement learning to language models, from pretrained models and human feedback to reward modeling and iterative policy updates. It also surveys feedback types, representative systems, and alternatives to reward-model-based RL.

  • 7.1 Pretrained Language Models: Pretraining gives language models broad capabilities, but its simple objective is insufficient for reliably following human intentions and instructions.Pretrained models acquire semantic representations, commonsense knowledge, and pattern-following abilities from large text corpora.
  • 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation: Supervised fine-tuning adapts pretrained models using high-quality human demonstrations that reflect intended tasks and behavior.Instruction-following models can be trained on annotated prompts or reformatted public datasets.
  • 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation: Human feedback is important for aligning models with user intentions, but collecting high-quality demonstrations or output evaluations is costly and impractical at scale.RLHF addresses this by using offline feedback to train a reward model that supplies surrogate reward signals during language-model training.
  • 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation: The review covers pretrained models, feedback types, RLHF training methods, representative systems, and approaches that avoid reinforcement learning or reward models.It positions RLHF within a broader literature spanning language-model training and alternative alignment methods.
  • 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation: RLHF treats language generation as sequential decision-making in which the model generates tokens and receives evaluative rewards.The language model represents the policy, and reinforcement learning trains it from scalar feedback.
  • 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation: The standard RLHF workflow collects feedback on model outputs, trains a scalar reward model, and updates the language model with reinforcement learning, often iteratively.Pretraining and optional supervised fine-tuning precede this loop.

7.3 Human Feedback

Human feedback is needed because many language tasks are ambiguous and automated metrics often fail to reflect human judgment. RLHF therefore uses several feedback formats, but collecting feedback that is informative, reliable, and sufficiently broad remains difficult.

  • Nature and objectives: Ambiguous tasks such as summarization require human judgment because BLEU and ROUGE often poorly correlate with perceived quality.Feedback is typically collected on model outputs or input-output pairs to guide training.
  • Nature and objectives: RLHF feedback can target helpfulness, honesty, and harmlessness, covering task completion, accuracy, uncertainty, safety, and bias-related behavior.These broad objectives contain narrower goals such as coherence, creativity, factuality, and avoiding harmful advice.
  • Feedback types: Feedback formats include binary or scalar ratings, rankings, free-form language, segment-level labels, token-level labels, and combinations of these.Preference feedback is widely used, while language feedback provides more detailed information about errors and desired improvements.
  • Feedback types: Free-form language feedback can clarify why an output is inappropriate and help the model generalize from a single example without requiring more examples.It is denser and more precise than a sparse judgment that merely marks an output as inappropriate.
  • Feedback types: Language feedback has been used across dialogue, summarization, question-answering, and code generation, but incorporating it into RLHF remains difficult.In summarization, one study found it more effective than preference feedback, while labelers took 3x as long to provide it.
  • Feedback collection: Feedback quality and quantity strongly affect outcomes: noisy feedback can harm performance, while insufficient feedback may prevent the intended objective from appearing.Clear labeling instructions and appropriate screening help reduce low-quality or systematically biased feedback.

7.4 Supervised Fine-Tuning: Limitations and Role

Supervised fine-tuning uses human demonstrations or preferred outputs to improve instruction following, but it cannot fully learn from negative feedback or prevent unwanted behavior. Despite these limitations, it provides the strong initial policy that RLHF generally needs for subsequent reinforcement learning.

  • Role: Supervised fine-tuning trains a language model on expert human demonstrations that serve as positive target outputs for prompts.Existing NLP examples can also be converted into instruction-tuning demonstrations.
  • Limitations: Fine-tuning directly on preferred outputs does not account for negative feedback, limiting robustness against adversarial situations and errors.Methods incorporating both positive and negative feedback address this limitation.
  • Limitations: Supervised fine-tuning does not penalize hallucinations or enable learning from neutral and negative feedback, which can leave harmful behaviors easier to elicit.Behavior cloning may also reproduce desirable surface behavior without fully capturing the underlying objective.
  • Role: PPO and related RL algorithms can struggle when initialized poorly, especially for text generation with a large action space.A KL penalty also keeps the RL-trained model close to its initial model, making initial policy quality important.
  • Empirical role: High-quality human demonstrations produce significant improvements over starting RLHF from pretrained language models.InstructGPT fine-tuned on customer- and labeler-written prompts and outputs before beginning RLHF training.
  • Role: Supervised fine-tuning remains an integral RLHF step because it supplies a robust initial policy for later reinforcement learning, despite its independent limitations.Prompted models can be alternatives, but available evidence does not establish equivalence to fine-tuning on human demonstrations.

7.5 Reward Modeling

Reward modeling replaces inaccessible human objectives with a learned proxy trained from feedback, making it central to RLHF. Its effectiveness depends on modeling choices, scale, initialization, generalization, calibration, and robustness, all of which remain imperfect.

  • Reward-model role: A reward model emulates human feedback by assigning scores to model outputs and serving as a proxy for the true human reward.The proxy is especially imperfect for out-of-domain inputs.
  • Training: Preference-based reward modeling fine-tunes a language model with a scalar projection to predict which of two trajectories humans prefer.Bradley-Terry-style modeling uses smoother, less noisy rewards, while Plackett-Luce models extend ranking to k trajectories.
  • Design choices: 30 times smaller reward models can perform well relative to the policy model, although very large reward models may be unstable and overfit.Reward-model size is therefore a trade-off between capacity, trainability, and stability.
  • Design choices: Public preference datasets can notably improve reward-model performance, especially for smaller models and datasets, whereas some alternative initializations provide only slight advantages.Initialization choices materially affect reward-model quality.
  • Challenges: Reward models must generalize beyond their initial prompts and outputs because most policy-training inputs are out of distribution relative to reward-model training.Generalization depends on dataset size, feedback noise, and the pretrained reward model.
  • Challenges: Calibration can deteriorate when helpfulness and harmlessness feedback are mixed, even though helpfulness-only training produced strong calibration in one study.Robustness is commonly assessed by evaluating the policy trained with the reward model.
  • Challenges: Reward modeling also faces sparse rewards, misspecification, and the need for more fine-grained or specialized reward signals.Segment-level rewards, multiple reward models, and other alternatives attempt to address these limitations, with mixed reported outcomes.

7.6 RLHF Finetuning of Language Models

RLHF fine-tunes a language model with scalar rewards from a separately trained reward model, typically using actor-critic policy-gradient algorithms. The procedure faces implementation and modeling challenges, including reward-model imperfections, instability, and sensitivity to hyperparameters.

  • RLHF training pipeline: The reward model scores prompt-output pairs, and policy-gradient reinforcement learning uses those scalar rewards to update the language model.PPO and A2C are commonly used for this update.
  • Training algorithms: PPO and A2C use actor-critic structures in which a critic estimates value and the actor updates the language-model policy.A2C uses advantage estimates, while PPO additionally clips policy updates to limit divergence.
  • Training challenges: Recent work modifies RLHF training algorithms to address high memory usage, NLP-specific requirements, and training instability.These changes target practical implementation and optimization difficulties.
  • Reward-model imperfections: Imperfect reward models can produce inaccurate value functions and gradients, enabling reward overoptimization and nonsensical generated text.Regularization, including KL divergence from an initial policy, is used to limit policy drift.
  • Iterated RLHF: Iterative RLHF retrains the reward model on outputs from the updated policy because calibration can degrade at higher rewards, but the procedure is expensive and time-consuming.New policy outputs are annotated by humans and incorporated into reward-model training.
  • Training challenges: RLHF remains unstable and highly sensitive to hyperparameters, requiring substantial tuning.These limitations complicate reliable training despite the method’s effectiveness.

7.7 Limitations of RLHF Models

RLHF models can align with human preferences and generalize beyond their training data, but they also exhibit alignment costs, low diversity, bias, calibration problems, and reward-model vulnerabilities. These limitations make training difficult and leave important safety and performance gaps.

  • Capabilities and trade-offs: RLHF models align with human preferences and can generalize to new scenarios, while remaining subject to behaviors requiring further refinement.The paper contrasts these capabilities with unresolved model characteristics and behaviors.
  • Alignment capabilities: RLHF can impose an Alignment Tax, with chat models sometimes underperforming the initial policy on downstream tasks.Incorporating the pre-training objective is proposed to reduce this cost.
  • Generalization capabilities: RLHF models can generalize to new prompts, languages, and coding questions despite limited English and code-related fine-tuning.This example suggests that pre-training contributes substantially to these capabilities.
  • Diversity and biases: RLHF models have lower output entropy, which challenges response diversity and can preserve biases from underlying language models.The paper also reports possible loss of probability calibration relative to pre-trained models.
  • Objective misalignment: Reward-model misalignment can favor longer outputs, and over-optimizing an imperfect proxy can produce reward overoptimization.The paper connects this vulnerability to missing calibration and robustness in reward models.
  • Robustness and safety: RLHF training is unstable, hyperparameter-sensitive, memory-intensive, and difficult for practitioners to use.These difficulties motivate simpler supervised fine-tuning objectives that learn directly from feedback.
  • Robustness and safety: Despite substantial improvements, RLHF models can still hallucinate, exhibit bias or toxicity, and generate harmful text when probed.The paper concludes that these models are not fully aligned with human preferences.
  • Sparse rewards: Sparse end-of-episode rewards and sparse preference feedback can make RLHF learning sample-inefficient and limit exploration.The paper identifies richer feedback as one response to these traditional reinforcement-learning challenges.

7.9 Moving Beyond RL Training

Alternatives to RLHF training reduce reliance on reinforcement learning or reward models by using feedback, rankings, principles, or direct objective reformulations. These methods are easier to train and promising in some settings, but their evidence remains limited to specific domains and constrained conditions.

  • Reward-model alternatives: Practitioners can rank multiple sampled outputs with a trained reward model at inference time instead of performing RLHF-based training.This approach is called best-on-n sampling or rejection sampling and requires attention to output diversity.
  • Reward-model alternatives: Other methods use ranked responses or weighted token decoding to train or guide policies without elaborate reinforcement-learning algorithms.These approaches retain a role for the trained reward model while simplifying policy optimization.
  • Methods without reward models: Direct-feedback alignment methods address PPO’s hyperparameter sensitivity, extra-model requirements, and potential reward-model misalignment.They incorporate human feedback without relying on the standard RLHF-PPO pipeline.
  • Feedback-based fine-tuning: Supervised fine-tuning on positive demonstrations is straightforward but excludes negative feedback needed for robustness against adversarial situations and errors.Positive-only training therefore does not use the full information available in feedback.
  • Feedback-based fine-tuning: Methods using positive and negative demonstrations maximize preferred-output likelihood and have outperformed RLHF methods on some summarization and dialogue tasks.The cited evidence concerns specific task settings rather than broad performance across domains.
  • Direct objective reformulation: Objective reformulations can train directly from feedback without a separate reward model, although out-of-distribution generalization may require an explicit reward model.The reported results are described as preliminary.
  • Principle- and refinement-based methods: Constitutional and refinement methods use human-encoded principles, feedback, or generated refinements to fine-tune language models.These approaches create additional training data from critiques, feedback, or guidelines.
  • Evidence boundaries: Alternative methods are easier to train and promising in some tasks, but evaluations remain limited to specific domains and constrained settings.The paper calls for deeper analysis before treating them as broadly validated replacements for RLHF-PPO.

8 Discussion and Conclusion

The paper analyzes RLHF’s foundations and limitations through a reward-model-centered framework, supported by a broad literature review. It identifies evaluative feedback as the persistent core of RLHF while emphasizing uncertainty quantification and improved reward encoding as important future directions.

  • Contributions: The paper examines RLHF’s mechanisms, assumptions, implementation choices, and limitations, centering its analysis on reward models.It introduces oracular rewards as ideal signals that learned reward models should approximate.
  • Literature synthesis: A comprehensive literature review traces RLHF’s development across feedback types, training algorithms, and alignment methods that avoid reinforcement learning.The review positions the paper as a reference on the evolution and variations of RLHF.
  • Discussion: Across RLHF variants, learning from evaluative feedback remains the core principle while agent formulation, reward feedback, and environment definitions continue to evolve.The paper identifies reward construction from existing knowledge and richer encoding of human preferences as future research directions.
  • Future directions: Robust uncertainty quantification is needed to identify and address low-confidence LLM outputs, especially in safety-critical applications.The paper presents understanding RLHF’s implications as important for responsible deployment.
Loading 2404.08555v2…