PaperlayerView exact revision on arXiv

Source-linked AI summary

The History and Risks of Reinforcement Learning and Human Feedback

Nathan Lambert, Thomas Krendl Gilbert, Tom Zick

arXiv:2310.13595v2cs.CY

TL;DR

Reward models are central to RLHF, yet their capabilities, evaluations, training methods, and open-source availability remain poorly documented. The paper traces the histories behind preference optimization and develops sociotechnical questions and evaluation proposals for contemporary reward models. It argues that assumptions carried across utility quantification and reinforcement learning do not align with contemporary use.

  • Problem

    Reward models are central to RLHF, yet their capabilities, evaluations, training methods, and open-source availability remain poorly documented.

    Source: §1 · p. 1

  • Method

    The paper traces the histories of preference quantification and reinforcement learning and proposes sociotechnical questions and evaluation tools for contemporary reward models.

    Source: §1 · p. 2

  • Results

    The paper argues that assumptions across utility quantification, reinforcement learning, and average preference maximization are not aligned with contemporary technology use.

    Source: §8 · p. 11

  • Takeaways & Limitations

    The authors argue that more information about existing reward models should be shared publicly through open or partially open releases.

    Source: §8 · p. 11

    Mechanistic understanding of deep RL is poorly documented, and evaluation methods are prone to statistical-analysis mistakes as they grow more complex.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective. A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization. This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood. RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist. Given this lack of information, further study and transparency is needed for learned RLHF reward models. In this paper, we illustrate the complex history of optimizing preferences, and articulate lines of inquiry to understand the sociotechnical context of reward models. In particular, we highlight the ontological differences between costs, rewards, and preferences at stake in RLHF's foundations, related methodological tensions, and possible research directions to improve general understanding of how reward models function.

1 Introduction

RLHF relies on preference reward models, but these intermediate models have received less scrutiny than the final language models they help optimize. The paper traces the assumptions behind RLHF and proposes questions and tools to improve understanding, transparency, and evaluation of reward models.

1 Introduction

  • Reward models are central to RLHF, yet historically have not been open-sourced or rigorously evaluated, obscuring how values are encoded.The final language model has faced internal and external scrutiny, unlike the intermediate reward model.Source: §1 · p. 1
  • RLHF carries control-theory optimization into language, where values are more computationally complex or fundamentally vague than control objectives.The paper locates a primary risk in this domain shift even though the optimization stack remains similar.Source: §1 · p. 1
  • The paper traces preference quantification and reinforcement learning histories to identify inherited assumptions and presumptions that may create reward-model shortcomings.Its historical scope ranges from Port-Royal Logic and Bentham to Markov Decision Processes and Bellman.Source: §1 · p. 2
  • The paper proposes questions about reward-model data, model, optimization, and emerging issues to improve transparency and broader stakeholder engagement.The questions extend inquiry beyond aggregate human-data concerns to structural optimization and deployment decisions.Source: §1 · p. 2
  • The paper proposes tools to measure and communicate reward models’ values and potential harms and to add rigor to future empirical evaluations.These proposals address contemporary RLHF reward models.Source: §1 · p. 2

2 Related Works

Related work describes RLHF’s development from control problems to language-model applications, alongside unresolved questions about whose preferences are encoded and what downstream effects follow. It also connects preference optimization to reward hacking and user-facing impacts.

2.1 Reinforcement learning from human feedback

  • Organizations deploying RLHF applications bear responsibility for specifying their ethics and answering whose preferences are included and how they are weighed.This concern accompanies the use of human preferences in optimization.Source: §2.1 · p. 2
  • RLHF methods developed from control problems using varied feedback and optimizers, then expanded across LLM tasks while feedback collection remained burdened by disagreement.LLM work progressed from four-completion choices to pairwise preferences, summarization, question answering, web agents, general chat, and specific objectives.Source: §2.1 · p. 2

2.2 Downstream impacts of optimizing preferences

  • Aggregate preferences and personalization raise questions about encoded values, while reward models can be over-optimized without higher scores tracking preference-correlated training signals.The passage notes uncertainty about how downstream LLM training signals relate to preferences correlated in the data.Source: §2.2 · p. 3
  • RLHF-trained systems may affect users’ moral judgments and exposure to biases, and reports describe troubling tone shifts within single conversations.The passage identifies stability and robustness as open questions.Source: §2.2 · p. 3
  • Reward mis-specification combined with powerful RL optimizers can produce complex feedback dynamics, including desired capabilities appearing and disappearing through repeated training.The passage connects these dynamics to reward hacking in RL.Source: §2.2 · p. 3

3 The Origins of Reward Models: Costs vs. Rewards vs. Preferences

RLHF brings together ideas from preference measurement, reward optimization, and costs, carrying assumptions from fields with clearer notions of optimality into language settings. The section traces how those foundations meet dynamic, context-dependent human preferences.

3 The Origins of Reward Models: Costs vs. Rewards vs. Preferences

  • The section distinguishes theoretically grounded assumptions from practice-based presumptions to trace how their unresolved tensions shape reward modeling.Its historical account is intended to identify blind spots and open questions, not to serve as a survey.Source: §3 · p. 3

3.1 Specifying objectives: from logic of utility to reward functions

  • Preference measurement debates remain unsettled, while social-science fields have developed distinct methods and semantics.Some economists question whether preferences can be measured apart from social relations and task context.Source: §3.1 · p. 4
  • RL reward-to-go combines reward with multi-step optimization, assuming numerical goals and clear optimality.This contrasts with human preferences, which do not necessarily yield an explicit optimal behavior.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 Implementing optimal utility

  • RL’s demonstrated usefulness is grounded in MDPs or tasks with a single closed-form reward, while its success mechanisms remain poorly documented.The paper notes that inverse reinforcement learning is rarely discussed in RLHF and calls for comparison with it in open-ended conversations.Source: §3.2 · p. 5

3.3 Steering preferences

  • Human preferences can drift and depend on context, while RL reward and cost functions are treated as stable and determinative.Preference modeling therefore compresses a multi-reward environment into a single function representation.Source: §3.3 · pp. 5–7
  • Preference representations may miss relevant context: presentation can shape choices, and aggregation faces competing fairness criteria and changing preferences.The section connects these tensions to preference aggregation across people, places, and situations.Source: §3.3 · p. 6

4 Background

This background describes how RLHF reward models are trained from human preference data and how their rewards are used to optimize language-model policies. It distinguishes common practices from the assumptions they presume.

4 Background

  • The paper uses assumptions for theoretically grounded premises and presumptions for commonly accepted practices to identify reward-model blind spots.The questions in the following sections connect to assumptions and presumptions introduced in the history and practice of RLHF.Source: §4 · p. 7

4.1 Reward models of human preferences

  • Preference data commonly comes from prompt-completion ratings, rankings, or pairwise choices, but collection and calibration decisions are not always documented.Workers may receive detailed instructions about which styles, occurrences, or values to prioritize.Source: §4.1 · p. 7
  • Chosen and rejected responses are often binarized for training, presuming pairwise preferences can suitably represent human values.The practice aggregates the binary labels independently of prompt difficulty or mean response quality.Source: §4.1 · p. 7
  • Reward models commonly classify chosen against rejected completions, then return a scalar reward for text.Implementations vary in whether they add a value head, modify output layers, remove layers, or freeze base-model parameters.Source: §4.1 · pp. 7–8

4.2 Reinforcement Learning on Language

  • In language RLHF, the policy selects next tokens from token-history observations, and the reward model scores generated sequences.With discount factor 1 and no further actions until sequence end, the setup is cast as contextual bandits.Source: §4.2 · p. 8

5 Questions

The paper proposes investigations into how data, model, and optimization choices encode priorities in RLHF reward models. The questions connect those choices to assumptions and presumptions identified in the paper’s history and background.

5 Questions

  • The proposed investigations examine assumptions behind reward models across data, model, and optimization choices.The questions are connected to assumptions and presumptions from the paper’s historical and technical accounts.Source: §5 · p. 8

5.1 Model Questions

  • Base-model evaluations should test how model choice affects reward-model behavior, training stability, efficiency, and losses.The paper asks whether some design choices could be addressed by changing the base model rather than increasing data or optimization.Source: §5.1 · p. 8
  • Sequential evaluation asks how biases in base, intermediate reward, and downstream RLHF models reflect changes in representations.Because reward models can still generate text, their outputs may indicate whether training is operating as intended.Source: §5.1 · pp. 8–9

5.2 Data Questions

  • Preference data is sparse relative to the complexity it represents, prompting questions about curation, worker context, feedback type, and annotator diversity.The paper asks whether professional-setting preferences transfer to downstream users and how disagreement should be treated.Source: §5.2 · p. 9

5.3 Optimization questions

  • Optimization research should test whether reward maximization matches intended preferences and examine learning curves, alignment, and preference weighting.The paper notes that RLHF often reports final test accuracy without learning curves, and raises whether averaging across groups can suppress multimodal beliefs, especially for underrepresented groups.Source: §5.3 · pp. 9–10

6 Solutions

The paper argues that reward-model evaluation should reflect how models are used and that preference specification should look beyond downstream-task performance alone.

6 Solutions

  • Held-out agreement misses dataset complexity and cross-intuition comparisons, motivating controlled examples and use-like reward-model benchmarks.The paper suggests representative categories such as reasoning, generation ability, and factuality, drawing an analogy to MT Bench (Zheng et al., 2023).Source: §6 · p. 10
  • Reward-model safety checks should compare scores on potentially harmful snippets with neutral texts and probe adversarial strings for unexpectedly high or low scores.Such adversarial scores could indicate downstream exploitation during the RL step.Source: §6 · p. 10
  • Preference specification should address what reward models should encode, even if this may reduce short-term performance, to support stakeholder engagement and harm reduction.Source: §6 · p. 10

7 Discussions

The discussion considers mismatches between training preferences and downstream users, risks from synthetic feedback, and direct optimization without an intermediate reward model.

7 Discussions

  • Training preferences may differ from downstream users’ contexts, such as English-centered data serving multilingual users; the paper raises transparency and user choice as possible responses.Source: §7 · p. 10
  • Synthetic preference data may reduce human-data costs, but preliminary findings indicate generation instability, reduced robustness, and possible further preference shift.Source: §7 · p. 11
  • Direct preference optimization can use human preference data without an intermediate reward model, potentially reducing misspecification and exploitation while making value encoding more opaque.The passage also says the questions raised in this work still apply, though auditing methods would differ without a distinct reward model.Source: §7 · p. 11

8 Conclusion

The conclusion argues that RLHF's historical assumptions do not align with contemporary use and calls for more reward-model sharing and evaluation.

8 Conclusion

  • The evolution from utility quantification through reinforcement learning to maximizing average preference scores leaves assumptions that are not in step with contemporary use.Source: §8 · p. 11
  • The paper argues that more information about existing reward models should be shared publicly through open or partially open releases.Source: §8 · p. 11
  • The authors say that further development and sharing can support proper evaluation techniques and progress on user tunability and uncertainty quantification.Source: §8 · p. 11

Related papers

Cited by

Loading 2310.13595v2…