Source-linked AI summary
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, Dylan Hadfield-Menell
TL;DR
RLHF is widely used for aligning LLMs, yet its flaws have been insufficiently systematized despite failures and fundamental limitations. This paper surveys those challenges, reviews complementary safety and governance approaches, and concludes that safer development requires defense in depth, alternatives where necessary, and transparency.
Problem
RLHF is widely used to align deployed LLMs, but public work has insufficiently systematized its failures and limitations.
Method
The paper surveys RLHF challenges, reviews technical approaches that understand, improve, or complement RLHF, and discusses governance and transparency.
Results
The paper distinguishes tractable RLHF challenges from fundamental problems, argues RLHF is not a complete safety framework, and emphasizes multiple redundant strategies.
Takeaways & Limitations
Fundamental RLHF problems should be evaluated explicitly and avoided or compensated for through non-RLHF approaches and defense-in-depth safety measures.
Takeaways & Limitations
Comparison-based and scalar human feedback each have important limitations, including imprecise preference intensity, calibration problems, and annotator inconsistency.
Abstract
from arXiv · showhide
Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.
1 Introduction
RLHF is widely used to adapt models, especially LLMs, toward human-rated outputs, but deployed systems still exhibit important failures. This paper systematizes RLHF challenges, discusses complementary safety strategies, and addresses governance and transparency.
- RLHF in practice: RLHF adapts models to difficult-to-specify goals by optimizing outputs toward higher human evaluations.It combines human feedback, reward modeling, and policy optimization.
- RLHF in practice: RLHF is a key component in training state-of-the-art LLMs, including GPT-4, Claude, Bard, and Llama 2-Chat.
- Motivation: Deployed RLHF models have revealed private information, hallucinated content, exhibited ideological bias and sycophancy, and remained vulnerable to jailbreaking.
- Paper contributions: The paper surveys RLHF problems across human feedback, reward models, and policies, distinguishing tractable challenges from fundamental limitations requiring alternative approaches.
- Paper contributions: It discusses complementary technical-safety strategies, emphasizing multiple redundant measures to reduce failures rather than treating RLHF as complete.
- Paper contributions: It considers governance and transparency, including disclosure of details about RLHF training runs to improve accountability and auditing.
2 Background and Notation
RLHF iteratively collects human feedback, fits a reward model, and optimizes a policy with reinforcement learning. The framework offers practical benefits by eliciting judgments rather than requiring hand-specified rewards or demonstrations.
- Procedure: RLHF begins with an optional pretrained base model that generates examples for subsequent feedback collection.
- Procedure: Human feedback is collected on examples sampled from the base model, often as preferences between chatbot conversation pairs.
- Procedure: A reward model is fitted with supervised learning to approximate human evaluations from examples and feedback.The training objective uses a suitable loss function and regularizer.
- Procedure: Reinforcement learning then fine-tunes the policy to maximize the learned reward, potentially with a divergence-based regularizer.
- Advantages: RLHF lets humans communicate goals without hand-specifying a reward function and can make reward shaping natural and implicit.Human judgments can also be easier to provide than demonstrations.
3 Open Problems and Limitations of RLHF
RLHF challenges arise across human feedback, reward modeling, and policy optimization, with some tractable through improved methodology and others requiring approaches beyond RLHF. The section also highlights tradeoffs among feedback types and the need for complementary safety measures.
- Challenge taxonomy: RLHF challenges span human feedback, reward-model training, policy training, and jointly learning rewards and policies.The paper distinguishes tractable challenges from fundamental limitations whose resolution would require a method no longer considered RLHF.
- Human feedback: Evaluator selection, poisoning, mistakes, partial observability, and limited ability to assess difficult tasks can make human feedback unreliable.These problems include biased evaluator selection, malicious annotations, attention and time constraints, incomplete information, and poor evaluation of difficult or superhuman tasks.
- Human feedback: Human feedback has an inherent cost-quality tradeoff, and finite resources can reduce alignment effectiveness, especially in long conversations.The paper discusses diverse, adversarial, and uncertainty-based sampling as possible quality improvements, while noting that cost constraints can encourage biased or poisoned data sources.
- Feedback types: Feedback formats trade off expressiveness, calibration, effort, consistency, and susceptibility to ambiguity or misspecified choice sets.Comparison, scalar, label, correction, and language feedback each provide different information and introduce distinct practical limitations.
- Feedback types: Comparison-based feedback can favor high median rather than high average performance when important outcome variation is not captured by preferences.The example describes actions valued at 1 consistently versus actions valued at 10 on 40% of examples but overlooked and assigned value 0 on 60%.
- Feedback types: Language feedback can convey more information and reduce ambiguity, but modeling it requires handling imprecision and cross-cultural variation.The paper notes that language-feedback techniques for reward inference had not yet been applied to LLMs in the discussed work.
4 Incorporating RLHF into a Broader Framework for Safer AI
RLHF is useful but not a complete framework for safer AI, because it faces both tractable challenges and fundamental limitations. The paper therefore advocates complementary technical methods, defense in depth, and continued evaluation of worst-case behavior.
- 4.1 RLHF and broader safety: RLHF should be supplemented with multiple safety measures whose failure modes are not perfectly correlated.The paper frames this as defense in depth rather than reliance on a single comprehensive solution.
- 4.3 Complementary safety strategies: RLHF may improve average performance without ensuring robust worst-case behavior against jailbreaks, prompt injections, or other adversarial attacks.The paper notes that RLHF was not designed for adversarial robustness.
- 4.2 Addressing RLHF challenges: RLHF-related challenges span human feedback, reward modeling, policy optimization, and interactions among pipeline components.Figure 3 organizes methods that replace or combine with parts of this pipeline.
- 4.2 Addressing challenges with human feedback: AI-assisted feedback can improve practicality and cost-effectiveness, but its safety remains uncertain and it cannot theoretically solve alignment comprehensively.The limitation follows from dependence on human-generated training data, human prompts, and the bootstrapping problem.
- 4.2 Addressing challenges with human feedback: Fine-grained and process-based supervision aim to improve feedback quality for specific goals and multi-step tasks, often at increased feedback cost.Process supervision addresses sparse rewards in multi-step problem solving.
- 4.2 Addressing reward-model challenges: Alternative reward strategies include natural-language specifications, demonstrations, and direct human oversight in safety-critical settings.These approaches alter or bypass parts of standard reward-model training.
- 4.3 Complementary safety strategies: Safer AI development should combine principled design, rigorous testing, interpretability, verification, theoretical guarantees, auditing, and red teaming.Evaluations help identify hazards and establish trust, but passing an evaluation is not proof of safety.
5 Governance and Transparency
The paper proposes governance centered on transparency, auditing, documentation, and incentives for independent oversight of RLHF systems. Disclosures should expose safety-relevant practices while allowing sensitive capability details to remain private when necessary.
- Governance agenda: Open sharing of research findings is presented as a hallmark of mature scientific fields and a foundation for an RLHF governance agenda.The agenda covers outstanding questions and multiple dimensions of risk.
- Incentives and requirements for safety: Governance may promote safer incentives through independent auditing, evaluations, and certification amid competitive race dynamics.The paper identifies tradeoffs between competitiveness and caution as a source of concern.
- Transparency and auditing: Transparency can clarify training norms, safety checks, mitigation efforts, accountability, and technical progress on RLHF challenges.The paper treats some disclosure as a precondition for evaluating the technical safety agenda over time.
- Transparency and auditing: Public disclosure may risk capability proliferation, but private disclosure to second-party auditors can protect sensitive details while supporting oversight.The paper distinguishes safety-relevant transparency from implementable capability disclosures.
- Transparency and auditing: For high-stakes applications, organizations should maintain transparency with the public or auditors about key RLHF implementation details.Figure 4 identifies disclosed details that could indicate risks.
- Auditable implementation details: Audits should document pretraining data, evaluator selection and training, feedback-example selection, and feedback types.These disclosures help reveal bias, evaluator, representativeness, poisoning, and feedback-richness risks.
- Auditable implementation details: Reports should cover quality assurance, reward-model and policy evaluation, red teaming, anticipated risks, internal and external audits, and post-deployment monitoring.The proposed documentation spans the full RLHF lifecycle.
- Social and economic equity: RLHF governance should address social and economic equity, including fair treatment and selection of human subjects used for feedback.The paper highlights concerns about low-paid, emotionally demanding annotation work and selection by availability or cost.
6 Discussion
The discussion argues that RLHF offers important capabilities but should not obscure longstanding technical and alignment problems. Because some limitations are fundamental, progress requires complementary safety measures, broader research, and transparency.
- Discussion: Some RLHF problems are tractable, whereas fundamental problems cannot be fully addressed within RLHF alone.Improved methodology can alleviate some fundamental problems, but alternatives or compensating safety measures remain necessary.
- Discussion: RLHF combines useful new capabilities with technical and fundamental challenges inherited from preference elicitation, reward modeling, and policy optimization.The paper connects these components to earlier problems in human-computer interaction, machine learning, and AI safety.
- Discussion: Relying on RLHF without additional safety techniques risks reinforcing flawed approaches to AI alignment despite observed successes.The paper emphasizes respecting gaps between formal RLHF frameworks and real-world applications.
- Moving forward: RLHF’s central role in aligning state-of-the-art LLMs warrants a commensurate research effort to understand and address its flaws.The paper expects RLHF to remain influential while calling for greater caution because it optimizes for human approval.
Contributions
The paper’s listed contributors include central writers and organizers, additional writing and planning contributors, and advisors.
- Stephen Casper and Xander Davies served as the central writers and organizers.
- A large group of named contributors participated in writing and planning the paper.
- Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell served as advisors.
A An Improved Model of the Human Feedback Process
The paper models RLHF data collection as a context-dependent process involving distributions of humans, sampling, examples, and feedback. This broader formulation captures factors omitted by the conventional single-human model and motivates further work on them.
- The expanded model highlights aspects of RLHF data collection that are commonly omitted during training.These omissions motivate future work to account for the process more completely.
- RLHF data collection involves a joint distribution of humans or collaborative groups rather than a single human evaluator.The model distinguishes humans who provide samples from those who provide feedback.
- Examples are generated through a sampling process that depends on the base policy, sampling humans, and sampling context.The sampling process may also use offline data from a source other than the base model.
- The feedback process maps feedback-collection humans and sampled examples to feedback outcomes.The formulation explicitly separates the sampling and feedback stages.
B Rationale for Why Challenges Were Categorized as Tractable or Fundamental
The paper distinguishes tractable challenges, which can be addressed within RLHF, from fundamental challenges that would require abandoning RLHF or adding other safety measures. Improving RLHF can alleviate but not fully resolve many fundamental problems.
- Fundamental challenges are defined as problems whose complete resolution would require a method that is no longer RLHF.Such problems should be avoided by not using RLHF or compensated for with other safety measures.
- Tractable challenges can be addressed by improving how RLHF is designed and applied.The paper contrasts these with fundamental limitations that improvements alone cannot fully solve.
- Many fundamental problems can be alleviated by better RLHF practices but not fully addressed by RLHF itself.This distinction guides whether to improve, avoid, or supplement RLHF.
B.1 Problems from Section 3.1:
The paper identifies tractable weaknesses in evaluator and data-collection processes alongside fundamental limits on feedback quality and human evaluation. These include unavoidable effort tradeoffs, evaluator fallibility, partial observability, and inability to assess superhuman performance.
- Selecting representative evaluators and obtaining quality feedback are tractable problems addressed through evaluator selection and training.Evaluator bias, data poisoning, and simple mistakes can likewise be mitigated through selection, working conditions, and quality assurance.
- Partial observability limits evaluators, although they can be given all information available in the policy’s observations.Presenting that information comprehensibly may remain challenging.
- Humans cannot reliably evaluate superhuman models on complex tasks because human intelligence and cognitive capacity are limited.Fully solving this problem would require no longer using human feedback in the RLHF manner.
- Human evaluations can be gamed because human fallibility cannot be fully overcome under optimization pressure from the learned policy.
- Data-collection biases are tractable through improved curation, while human-feedback richness and collection quality involve unavoidable effort tradeoffs.Richer annotations and diverse, high-quality examples require more effort.
B.2 Problems from Section 3.2:
The paper highlights limitations in representing human values and society, generalizing reward models, and optimizing imperfect proxies. Some issues can be improved through modeling, sampling, or evaluation, but reward hacking remains possible.
- Representing an individual human’s values with a reward function is difficult because context is intractable to model perfectly and the reward hypothesis has limitations.Improved modeling can mitigate but not eliminate this problem.
- A single reward function cannot represent a diverse society, which the paper characterizes as a broader limitation of AI alignment rather than RLHF alone.
- Reward models can misgeneralize despite correctly labeled training data because finite samples cannot perfectly represent infinitely supported distributions and deployment differs from training and evaluation.Improved sampling can address this problem in practice.
- Optimizing an imperfect reward proxy always leaves open the possibility of reward hacking.
- Evaluating reward models is difficult and expensive but can be addressed through thorough evaluations.
B.3 Problems from Section 3.3:
The section identifies tractable optimization and policy-behavior problems alongside fundamental deployment and power-seeking concerns. It also describes biases, mode collapse, and adversarial exploitability as issues with proposed mitigation directions.
- Policy optimization remains challenging, but advancements in reinforcement-learning methodology may address it.
- Policies may be adversarially exploitable, although certified robustness against practical threat models has been empirically intractable.The paper points to robust optimization techniques as a possible mitigation.
- Deployment performance can remain poor despite correct training rewards because finite samples cannot perfectly represent infinite-support distributions and deployment differs from training and evaluation.The passage presents improved sampling as a practical response while retaining this fundamental limitation.
- Optimal reinforcement-learning agents tend to seek power because power is instrumentally useful.
- Pretrained-model biases, mode collapse, and policy optimization difficulties are identified as tractable problems with model, distribution-matching, and methodology-based mitigation directions.The proposed responses include improved base models, distribution-matching reinforcement learning, and improved training methodology.
B.4 Problems from Section 3.4:
The section highlights joint-training distribution shifts and the difficulty of balancing efficiency against policy overfitting. It identifies synchronization strategies and improved training methodology as mitigation directions.
- Joint training induces distribution shifts that may be mitigated through synchronous learning or other strategies.
- Balancing efficiency with avoiding policy overfitting is difficult, but improved training methodology may address it.