PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Reinforcement Learning from Human Feedback

Timo Kaufmann, Paul Weng, Viktor Bengs, Eyke Hüllermeier

arXiv:2312.14925v3cs.LG

TL;DR

RLHF addresses the difficulty of designing effective numeric rewards and obtaining regular human judgments. This survey synthesizes RLHF fundamentals, algorithms, feedback types, applications, and research trends, finding that feedback informativeness depends on labeler rationality.

  • Problem

    Designing effective numeric reward functions is difficult, and directly obtaining sufficiently regular, immediate, consistent human rewards is usually impractical.

  • Method

    The survey synthesizes RLHF’s foundations, algorithms, feedback mechanisms, applications, extensions, open questions, and research directions across multiple domains.

  • Results

    Feedback informativeness depends on human-labeler rationality: demonstrations help more with highly rational labelers, whereas comparisons are preferable in less-rational settings.

  • Takeaways & Limitations

    Adaptive adjustment of feedback type during labeling may be worthwhile, and the relationship between labeler rationality and informativeness remains a promising research direction.

  • Takeaways & Limitations

    RLHF is limited by tasks humans can reliably judge, and current approaches often learn spurious correlations rather than the actual causes of human feedback.

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the intersection of artificial intelligence and human-computer interaction. This positioning provides a promising approach to enhance the performance and adaptability of intelligent systems while also improving the alignment of their objectives with human values. The success in training large language models (LLMs) has impressively demonstrated this potential in recent years, where RLHF has played a decisive role in directing the model's capabilities towards human objectives. This article provides an overview of the fundamentals of RLHF, exploring how RL agents interact with human feedback. While recent focus has been on RLHF for LLMs, our survey covers the technique across multiple domains. We provide our most comprehensive coverage in control and robotics, where many fundamental techniques originate, alongside a dedicated LLM section. We examine the core principles that underpin RLHF, how algorithms and human feedback work together, and the main research trends in the field. Our goal is to give researchers and practitioners a clear understanding of this rapidly growing field.

3 Feedback · 1 Introduction

RLHF addresses the difficulty of engineering reward functions by letting humans define and refine objectives through feedback during learning. This survey presents RLHF broadly across domains, classifies its approaches, and focuses particularly on control and robotics while also covering LLM-specific considerations.

  • 1 Introduction: Reward engineering is difficult in complex or real-world tasks, and poorly specified rewards can produce unintended or unsafe behaviors.Agents may exploit simulation flaws, hack rewards, or optimize intermediate signals without achieving the intended goal.
  • 1.1 Why Human Feedback?: RLHF incorporates human feedback to define and iteratively refine objectives, improving alignment with human values and adapting rewards during learning.Dynamic refinement can address distributional shifts that make initially specified rewards unreliable.
  • 1 Introduction: Although LLM applications motivated recent prominence, the survey examines RLHF across multiple domains with particular depth in control and robotics.The LLM discussion addresses large action spaces, distribution shift, and KL regularization, while drawing on techniques originating in control.
  • 1.4 Outline: The survey classifies RLHF approaches, describes their characteristics and applications, and develops the topic from foundational RL concepts through feedback, reward modeling, policy learning, theory, and applications.It presents itself as a continuation of Wirth et al. (2017) covering the evolution from PbRL to RLHF and subsequent methodological advances.
  • 1.1 Why Human Feedback?: RLHF’s effectiveness for resolving alignment problems remains debated because agents may manipulate teachers or exploit weaknesses in the feedback process.The survey presents RLHF as promising for alignment but does not treat it as a complete solution.
  • 1.2 The Origins of RLHF: The survey treats RLHF as a generalization of PbRL that includes relative feedback such as comparisons and rankings alongside broader feedback types.PbRL originated independently in 2011 and infers objectives from qualitative rather than numerical feedback.
  • 1.2 The Origins of RLHF: The survey defines RLHF broadly as approaches using human feedback to define RL objectives, while acknowledging that terminology and distinctions across PbRL, SSRL, and RLHF remain unsettled.These definitions reflect the authors’ interpretation of predominant usage in the literature.
  • 1.3 Scope of the Survey: The survey covers methods that learn a reward model as the sole objective source from interactive, online, scalable, and asynchronous human feedback.It excludes supplementary guidance, pure demonstrations, standalone inverse RL, and methods requiring synchronous supervision.

2 Preliminaries

This section introduces standard reinforcement learning through Markov decision processes and contrasts it with RLHF, where reward information comes indirectly from an oracle, commonly through trajectory comparisons. It also notes that standard deep RL improvements may transfer to RLHF training.

  • Reinforcement learning: RL models sequential decision-making as an MDP in which an agent observes states, selects actions, transitions probabilistically, and receives rewards.The transition function defines environment dynamics, with transitions conditionally independent of previous states and actions under the Markov property.
  • Reinforcement learning: An H-step trajectory is a sequence of state-action pairs ending in a terminal state, and its return accumulates discounted rewards.The return remains well defined for an infinite horizon when γ < 1.
  • Reinforcement learning: RL agents learn policies that maximize expected return, using model-based methods, model-free methods, value-based methods, or policy search.Policy search includes actor-critic algorithms such as PPO, TD3, and SAC; RL algorithms are also classified as on-policy or off-policy.
  • Reinforcement learning from human feedback: RLHF replaces an assumed reward signal with oracle feedback, where queries receive labels that indirectly provide information about reward.In the common formulation, the oracle compares two trajectory segments, corresponding to preference-based MDPs without a reward function.
  • Reinforcement learning from human feedback: Improvements to standard deep RL, including auxiliary losses, learned-model planning, curriculum learning, and data augmentation, may transfer to RLHF training.A standard RL algorithm can be used in the RL training component of RLHF, although this may not be optimal.

3 Feedback

This section characterizes human feedback as indirect and asynchronous interaction that avoids the burdens of immediate numeric rewards. It organizes feedback types by interaction, query, and communication attributes while emphasizing that suitable feedback depends on the task, user, and agent.

  • Feedback mechanisms: RLHF uses indirect, asynchronous feedback instead of immediate numeric rewards, reducing human effort and improving learning progress and user experience.Direct rewards require sufficiently regular human input and would slow learning while awaiting feedback.
  • Feedback attributes: Feedback types are classified by arity, involvement, granularity, abstraction, explicitness, and intent.These dimensions cover how feedback is given, the query instance’s form, and features of human interaction.
  • Feedback attributes: Feedback attributes involve tradeoffs: coarse feedback provides more context but complicates credit assignment, while fine-grained feedback simplifies learning but can be tedious to provide.Non-unary feedback supplies grounding but requires comparable instances; abstract feedback may be easier to learn from but harder to extract or judge.
  • Interactive feedback types: Within-trajectory feedback can address the difficulty of comparing trajectories when starting conditions or tasks vary across episodes.Trajectory comparisons require roughly comparable trajectories, which may be unavailable in real-world environments.
  • Language feedback: Instructions work better in low-autonomy settings, whereas desired-outcome descriptions are more effective in high-autonomy settings.The comparison concerns how language feedback is elicited and used during learning.
  • Feedback selection: The best feedback type may change over time and depends on the task, user, agent, task complexity, and time horizon.Feedback selection can also consider user interaction and expertise, including granularity, abstraction, and involvement.

4 Label Collection

RLHF label collection requires actively generating and selecting preference queries while accounting for uncertainty, feedback type, and human-computer interaction effects. The survey highlights that label quality is limited by rationality differences, ordering effects, systematic biases, noise, and disagreement among humans.

  • Query selection: RLHF agents actively generate and select queries to learn human preferences and efficiently train a reward model.Query selection is framed as an active-learning problem in the reinforcement-learning setting, where the agent can influence the data distribution and query sequence.
  • Query selection: Uncertainty-based acquisition criteria often outperform random or diversity-based selection, although random selection can remain competitive in some cases.Ensemble-based uncertainty is commonly represented by output variance or its equivalent standard deviation.
  • Feedback type: The most informative feedback type depends on the human labeler’s rationality: demonstrations are favored for highly rational humans, whereas comparisons suit less-rational settings.Human reliability and rationality affect how informative feedback is for reward inference, and noisy Boltzmann-rational behavior can be more informative than perfectly rational behavior.
  • Human-computer interaction: Preference labels are affected by ordering effects, stochastic noise, and biases toward qualities that are easier to judge than qualities requiring external assistance.Reported examples include humans favoring assertiveness over factuality and an agent learning behavior that looked good only from the labelers’ camera angle.
  • Label quality: Human agreement is imperfect: researchers agreed with each other about 60% of the time but with labelers only 38% or 46%, while other tasks reported agreement near 73%.The near-73% figures include 73% ± 4% among researchers and 72.6 ± 1.5% among screened labelers on instruction fine-tuning.

5 Reward Model Training · 5.1 Human Feedback Model

This section frames reward-model training as learning a reward function from mathematically modeled human choices, with query-defined candidate labels and explicit treatment of rationality, utility, feedback-model alternatives, misspecification, preference diversity, and non-Markovian feedback. The widely used Boltzmann approach is useful but has identifiable limitations and motivates alternative models and robustness analyses.

  • 5 Reward Model Training · 5.1 Human Feedback Model: Reward-model training connects human feedback to the reward function by modeling each query as a choice among candidate labels.The human returns a label from the query-dependent choice set, with noiseless feedback recovered as a degenerate choice distribution.
  • 5.1.1 Boltzmann Distribution: The Boltzmann distribution models label-selection probabilities from perturbed latent utilities and is attractive because it has a closed analytic form and satisfies Luce’s axiom of choice.For discrete choice sets it is also called the multinomial logit or Gibbs model, and for pairwise choices it corresponds to Bradley-Terry.
  • 5.1.1 Boltzmann Distribution: Grounding functions connect labels to trajectory distributions, allowing utilities to represent trajectory returns and extend feedback modeling to pairwise comparisons, rankings, and trajectory segments.For pairwise comparisons, a label’s utility essentially represents the preferred trajectory’s utility.
  • 5.1.2 Human-Specific Rationality Coefficient: The rationality coefficient β controls human-choice precision: higher β approaches a point mass on the highest-utility option, whereas lower β approaches uniform random choice.The standard Boltzmann model assumes β = 1; estimating β can positively influence reward learning but requires a calibration reward function.
  • 5.1.3 Alternative Utility Notions: Using trajectory return as utility can make rewards non-identifiable, while trajectory regret provably yields identifiable rewards and empirically improves alignment with human preferences.Regret depends on the unknown optimal policy, and human-policy Q-values have also been proposed as utilities.
  • 5.1.4 Human Feedback Models Beyond Boltzmann: Beyond Boltzmann, RLHF research considers Thurstone, ridge-noise, binary, mixed, link-function, two-stage, and generative human-feedback models.For pairwise comparisons, link functions map utility differences to preference probabilities; a generative model directly interprets generated human feedback as reward but performed worse than an inter-temporal Bradley-Terry model.
  • 5.1.5 Misspecification: Misspecifying feedback semantics or choice sets can drastically change performance, while robustness depends on the assumed model: optimality is not robust, Boltzmann is robust to specific misspecifications, and maximal causal entropy is intermediate.Literal and pedagogical feedback represent different assumptions about whether feedback targets the task or what helps the learner.
  • 5.1.6 Diverse Preferences · 5.1.7 Relaxation of the Markov Assumption: With diverse labelers, RLHF can learn individual reward functions or a group preference model, while active labeler selection, preference aggregation, and non-Markovian preference models address further challenges.Researchers also use regularization and consistency constraints for inconsistent feedback, social-choice tools for aggregation, and transformers or hidden-state-dependent rewards to relax Markov assumptions.

5.2 Utility Learning

Utility learning recovers latent utilities from observed human feedback by reducing the problem to supervised learning, primarily through empirical risk minimization or Bayesian methods. The section also discusses reward-model design, identifiability and explainability, and the trade-offs between online and offline training.

  • Utility learning methods: Utility learning reduces recovery of latent utilities from human feedback to supervised learning, commonly using empirical risk minimization or Bayesian approaches.Empirical risk minimization finds reward functions that minimize a suitable loss, while Bayesian methods update a prior using observed feedback data.
  • Reward-model design: Reward models commonly use parameterized functions, but recent work increasingly favors nonlinear differentiable architectures over potentially misspecified linear models.Linear models require suitable state-action features known in advance; regularization and auxiliary losses are used to reduce overfitting and improve generalizability.
  • Identifiability: Reward learning must address partial identifiability, because distinct reward functions can induce exactly the same human feedback model.This ambiguity has been studied for demonstrations and pairwise trajectory preferences under the Boltzmann feedback model.
  • Explainability: RLHF explainability remains sparse and is mostly pursued through transparent tree-based reward models or simultaneous learning of rewards and state importance.These approaches aim to make reward-model behavior more interpretable for experts and non-experts.
  • Online and offline RLHF: Online reward-model improvement is important because offline overoptimization of an imperfect model can degrade performance, motivating a growing trend toward fully offline RLHF.Offline RLHF trains both the reward model and policy from offline behavior data, preference responses, and offline reinforcement-learning algorithms.

5.3 Evaluating Learned Reward Functions

Learned reward functions can be evaluated through online rollouts, off-policy estimates, reward-function distances, or visual and human inspection. Each approach addresses different practical constraints, while distance-based methods seek policy-relevant comparisons but can be unreliable when they ignore transition dynamics.

  • Rollout Method: Rollout evaluation learns an optimal policy under the learned reward, then estimates its value using online trajectories and the ground-truth reward.This method may be infeasible in safety-critical domains such as medicine or autonomous driving.
  • Off-policy Evaluations: Off-policy evaluation estimates the optimal policy’s value from an available dataset when online rollouts are unavailable.Importance sampling, regression or classification methods, and their combinations address policy-mismatch bias or high variance; the approach is particularly suited to LLM reward models.
  • Distance Functions: EPIC distance assigns zero distance to reward functions differing only by policy-preserving transformations such as potential shaping or positive scaling.It provides a pseudometric with attractive Lipschitz-continuity properties relating reward distance to differences between induced optimal-policy values.
  • Distance Functions: EPIC distance can be unreliable on physically non-realizable trajectories because its canonicalization does not encode sufficient underlying transition dynamics.EPIC-like distances and STARC metrics generalize the underlying three-step construction by varying canonicalization, normalization, and metric functions.
  • Visual and Human Inspection: Visual, human, or expert inspection evaluates reward quality by examining agent behavior on the target task, with preprocessing used to improve reward interpretability.The evaluator can also be another trained model in reinforcement learning from AI feedback.

5.4 Reward Model Inputs

Reward model inputs typically comprise agent observations and actions, spanning true states or images and discrete or continuous actions. Input modality shapes reward learning difficulty and varies across control, Atari, and NLP applications.

  • Input modalities: Reward model inputs usually consist of agent observations and actions, with observations ranging from true states to images and actions from discrete to continuous.The input modality is an explicit factor in reward-model design.
  • Control tasks: In continuous-control benchmarks such as MuJoCo robotics tasks, reward models can use true-state representations and simple discrete actions.Christiano et al. (2017) train reward models from these inputs.
  • Visual inputs: Raw-image control tasks make reward learning harder because observations are partially observable and rewards are generally non-Markov with respect to them.A common remedy is approximating the true state with a sequence of frames, as in Atari reward-model training.
  • Language and multimodal inputs: In NLP applications, policies take natural language as input and output while reward models take language as input; language-and-vision settings have also been studied.Ouyang et al. (2022) exemplify language-based reward modeling, while Abramson et al. (2022) represent multimodal scenarios.

5.5 Increasing Feedback Efficiency

RLHF feedback efficiency can be improved by leveraging prior offline data, extracting more value from labeled and unlabeled data, and generating more informative experiences. These methods reduce the feedback required for learning human preferences and improve the effectiveness of available interactions.

  • Leveraging Prior Data: Prior data, including task-related datasets and unsupervised text corpora, can reduce the human feedback needed to learn a new objective.Approaches include meta- and transfer learning, foundation models, reward-model initialization, reward-model pretraining, and supervised representation learning.
  • Reward Model Initialization: Initializing reward models from related pretrained models is particularly common in language-model fine-tuning and complements policy initialization.Self-supervised pretraining provides parameters that can be used to initialize both the policy and reward model.
  • Using Available Data More Efficiently: Self-supervised and semi-supervised training improve data utilization by learning task information from unannotated data alongside human-labeled examples.Semi-supervised methods commonly generate pseudo-labels from model predictions or assumptions.
  • Data Augmentation: Temporal data augmentation creates multiple derivative preference pairs by extrapolating preferences for complete trajectories to cropped trajectory segments.This approach increases the examples available from a single labeled trajectory pair.
  • Generating Informative Experiences: Intrinsic motivation driven by reward-model uncertainty can steer exploration toward state-space regions where human feedback is most beneficial.This generates more targeted experiences rather than only improving the use of existing data.

6 Policy Learning

Policy learning in RLHF either adapts conventional RL algorithms to learned rewards or bypasses reward modeling through direct preference optimization. These approaches address non-stationary and approximate rewards while exploiting simplified bandit formulations, especially for LLMs.

  • Policy learning: Policy learning maximizes expected accumulated reward using either adapted conventional RL algorithms or direct preference optimization (DPO).The survey organizes policy-learning methods into these two main techniques.
  • Conventional RL adaptations: Learned rewards create two challenges: non-stationarity from periodic reward-model updates and inaccuracy during early training.These issues can undermine direct application of standard RL algorithms.
  • Conventional RL adaptations: Conventional policy-gradient and value-based methods can work in RLHF, with replay-buffer experience relabeling helping value-based methods accommodate reward-model updates.Policy-gradient methods have been applied without modification, while value-based approaches can use updated rewards to relabel stored experiences.
  • Approximate rewards: Because approximate rewards can cause overoptimization or reward hacking, policy training may regularize divergence from human demonstrations using KL divergence.KL regularization is common in language-model fine-tuning and is important in some, but not all, control cases.
  • Preference-based bandits: The preference-based bandit formulation treats each response as an action with a terminal reward, simplifying exploration under deterministic dynamics despite an exponentially large trajectory space.It also enables policy improvement without training through best-of-n sampling, which selects the highest-rated response among multiple samples.
  • Direct preference optimization: DPO circumvents reward learning by using preferences directly to increase better-output probabilities and decrease worse-output probabilities relative to a reference policy.The reference policy can be a pretrained LLM, with β controlling deviation from it.

7 Theory

RLHF theory studies provably near-optimal policy learning in online and offline settings, alongside RLHF’s relationship to standard reward-based reinforcement learning. Recent work develops preference-based algorithms with regret, PAC, and comparison-complexity guarantees, while questioning link-function models that impose transitive human preferences.

  • Online and Offline Learning: Theoretical RLHF distinguishes offline learning from fixed data and online learning through direct environment interaction with continuously updated strategies.This distinction mirrors standard reinforcement learning and organizes the literature’s theoretical results.
  • Online Learning: Online methods adapt dueling- or preference-based bandit algorithms to trajectory-comparison feedback for learning optimal policies.Examples include Dueling Posterior Sampling, Preference-based Policy Search, Preference-based Exploration and Policy Search, and Logistic Preference Reinforcement Learning.
  • Theoretical Guarantees: Theoretical guarantees include expected or high-probability regret minimization, (ϵ, δ)-PAC optimal-policy learning, and bounds on trajectory-comparison complexity.These guarantees appear under assumptions including tabular or linear MDPs, link functions, and structured feedback or transition models.
  • Function Approximation: Later work extends preference-based policy learning beyond tabular MDPs using general function classes with finite Eluder dimension and confidence-set estimation.Preference-based Optimistic Planning estimates both human feedback and transition dynamics, while related methods target low trajectory-comparison query complexity.
  • Limitations and New Directions: Most theoretical approaches model pairwise feedback with link functions, but this imposes transitivity even though human preferences may contain cycles.This limitation motivates theoretical work that dispenses with parametric feedback modeling.

8 Applications and Benchmarks

RLHF has been applied across language models, control and robotics, generative models, and recommender systems. Libraries, benchmarks, and static datasets support implementation, reproducible evaluation, and research development across these settings.

  • Applications: RLHF applications span language models, control tasks, generative models, and recommender systems.ChatGPT is identified as the most prominent application, while other work addresses robotics, image generation, and long-term recommender engagement.
  • Applications: In control, RLHF can match ground-truth-reward agents in games and simulated continuous control using a fraction of the feedback.Control tasks are especially relevant because their objectives are often difficult to specify.
  • Applications: Generative models can be framed as policies that assemble outputs through actions, enabling RLHF to fine-tune language and imaging models toward preferred outputs.For language models, tokens serve as the policy’s actions.
  • Libraries and datasets: Libraries such as imitation and Clean-Offline-RLHF provide tools for implementing and experimenting with imitation, reward-learning, and offline RLHF algorithms.These libraries reduce implementation difficulty created by RLHF’s interacting components and human involvement.
  • Benchmarks: Benchmarks are important because RLHF evaluations are difficult to reproduce when objectives lack ground truth and humans remain in the loop.Proposed benchmarks target different applications and challenges, including synthetic feedback, offline evaluation, complex Minecraft tasks, and language-model reward assessment.
  • Benchmarks: Offline RLHF is well suited to benchmarking because static datasets can be used, although many existing offline RL benchmarks require overly simple reward functions.Shin et al. identify suitable benchmarks and add one for evaluation, while Yuan et al. propose a data-based approach using synthetic rewards.

9 Discussion and Conclusion

The survey positions RLHF as a rapidly expanding approach spanning multiple domains, while emphasizing that its broader applicability remains constrained by technical, theoretical, ethical, and social limitations. It also identifies extensions beyond standard online human-feedback reward learning and opportunities for future advances.

  • Scope and contributions: The survey traces RLHF’s evolution from PbRL and reviews applications across control, natural language processing, and computer vision.The authors note that the field’s rapid expansion limits how fully every extension and application can be covered.
  • Extensions and outlook: Promising extensions include RLAIF, which uses evaluations from pre-trained AI systems, and approaches that move beyond scalar reward objectives.The survey also identifies opportunities for more robust algorithms that use human feedback more efficiently.
  • Limitations: RLHF is limited by feedback quality and interpretation, including humans’ difficulty judging some tasks, preference for assertive but incorrect responses, and reliance on spurious correlations.These issues constrain what humans can reliably evaluate and whether learned feedback reflects its actual causes.
  • Limitations: Separating the policy and reward model can prevent agents from reasoning about gaps in their understanding of human preferences.Cooperative inverse RL integrates preference learning with action to maximize those preferences.
  • Limitations: RLHF faces theoretical challenges in relaxing restrictive assumptions while preserving computationally efficient guarantees.Open questions include avoiding explicit policy-space maintenance, eliminating sub-optimal policies, and reliance on computation oracles.
  • Limitations: Ethical and social concerns include whose preferences guide alignment, privacy-respecting data collection, human-bias amplification, and labeler-manipulation incentives.These concerns arise from current RLHF training methodologies and require careful attention alongside technical solutions.

A RLHF for Language Models

RLHF for language models uses human feedback to fine-tune pretrained models toward more helpful, honest, and harmless outputs. The setting commonly treats text generation as a single-turn RL problem, while practical extensions introduce dialogue, tools, and richer RL complexities.

  • A RLHF for Language Models: RLHF preference-based fine-tuning steers pretrained language models toward more helpful, honest, and harmless outputs.These models are typically pretrained self-supervised on large datasets whose objectives are not aligned with intended usage.
  • A RLHF for Language Models: Human feedback on generated outputs trains a reward model that guides the language model toward better responses.Early applications focused on summarization, treating the language model as an RL agent.
  • A RLHF for Language Models: The standard formulation is a deterministic, single-turn MDP where actions generate tokens, the state is the text so far, and one terminal reward evaluates the output.The reward is supplied by a reward model based on human feedback, making environment-dynamics exploration unnecessary.
  • A RLHF for Language Models: Training typically combines behavioral cloning from a small labeled dataset with preference-based fine-tuning because supervised fine-tuning alone is often insufficient.Limited size and diversity of supervised data can cause overfitting and poor generalization, whereas preference data is often more diverse and efficiently collected.
  • A RLHF for Language Models: Multi-turn and tool-using extensions reintroduce nondeterministic transitions, partial observability, and delayed rewards, while dense rewards can guide agents more effectively.Dialogue interactions can include follow-up questions, web-browser access, and other actions beyond text output.
  • A RLHF for Language Models: Direct methods are especially relevant to LLM fine-tuning because separating reward learning from policy training is less useful than in control applications.The separation is useful in control because it allows more policy-training samples than reward-model-training samples under limited human supervision.

B Prior Surveys

This section differentiates the survey from related human-in-the-loop reinforcement learning surveys and compares it with previous surveys or survey-like articles in RLHF. It also uses criteria from Section 1.3 to structure these comparisons.

  • Related human-in-the-loop RL surveys: The survey first distinguishes itself from surveys in marginally related human-in-the-loop RL areas.These comparisons are based on criteria discussed in Section 1.3.
  • Survey comparison criteria: Table 6 overviews prior human-in-the-loop RL surveys according to whether each criterion is a main focus, partially addressed, or not covered.The table uses ✓, (✓), and ✗ for these three coverage levels.
  • RLHF surveys: It then describes differences between the survey and previous RLHF surveys or survey-like articles.The comparison covers articles within the RLHF field.

B.1 Human-in-the-Loop RL

RLHF belongs to human-in-the-loop and interactive RL, but differs from related approaches by using human feedback to define tasks or objectives. The survey takes a broader and more detailed perspective than prior work, covering applications beyond LLMs.

  • Prior Surveys: Prior surveys address human-in-the-loop RL, guidance, alignment, and LLM applications, but generally do not provide RLHF’s level of detail or breadth.Related work includes broad surveys of human involvement, reward learning, AI alignment, and applications such as ChatGPT and GPT-4.
  • Interactive RL: RLHF is a subfield of interactive RL in which human feedback can define the objective, unlike other interactions that support exploration or learning speed.Interactive RL studies algorithms that learn through interaction with humans.
  • Demonstrations: Unlike learning from demonstrations, RLHF uses feedback rather than desired-behavior demonstrations, avoiding the need for near-optimal demonstrations.Demonstrations are usually provided non-interactively and offline, limiting their applications and final performance.
  • Assisted RL: Unlike assisted RL, RLHF uses external information to define the task rather than merely aiding action selection or modifying an existing reward function.Assisted RL generally assumes an initial reward function and limits updates to reward shaping or supplementary signals.
  • Applications: Although LLMs receive substantial attention, they represent only one RLHF application area; this survey examines diverse applications and impacts beyond LLMs.The survey explicitly adopts a broader perspective than application-focused work on LLMs such as ChatGPT and GPT-4.

B.2 PbRL and RLHF

Prior surveys and related frameworks cover narrower aspects of RLHF, including binary-preference PbRL, feedback types, language generation, and open problems. This survey distinguishes itself through broader coverage while complementing domain-specific and problem-focused work.

  • Preference-Based RL: Wirth et al. (2017) survey binary trajectory preferences and policy-learning methods that do not derive a reward model, whereas reward modeling later became dominant.Their work is described as a direct precursor to this survey.
  • Feedback Types: Jeon et al. (2020) provide a reward-rational implicit-choice framework that unifies prior PbRL and RLHF studies across feedback types.Metz et al. (2023) and Yuan et al. (2024) likewise study user interaction and interface design for multiple feedback types.
  • Domain-Specific: Fernandes et al. (2023) focus on human feedback for language generation and discuss language-specific issues outside this survey’s scope, such as feedback models at generation time.Their domain-specific focus makes the survey less comprehensive than this work.
  • Open Problems: Casper et al. (2023) survey RLHF limitations with emphasis on security, governance, and transparency, but omit theoretical approaches and existing benchmark overviews.Their article covers reward modeling and broader human feedback, making it complementary further reading.
Loading 2312.14925v3…