PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Reward Models

Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, Alon Albalak

arXiv:2410.12832v1cs.LG

TL;DR

Human-feedback alignment requires substantial preference-label collection, while synthetic feedback may not match human judgments. GenRM unifies RLHF and RLAIF by training an LLM judge with self-generated reasoning traces; its variants maintain competitive in-distribution accuracy and outperform traditional methods on out-of-distribution tasks.

  • Problem

    RLHF requires substantial human preference labels, while synthetic preference labels may not align well with human judgments.

    Source: §1 · pp. 1–2

  • Method

    GenRM trains an LLM judge on preference data using self-generated reasoning traces in an iterative framework combining RLHF and RLAIF.

    Source: §2.2.2 · p. 4, §4 · p. 5, §7 · p. 10

  • Results

    GenRM and its variants maintain competitive in-distribution accuracy while outperforming traditional methods on out-of-distribution tasks.

    Source: §7 · pp. 10–11

  • Takeaways & Limitations

    The authors present GenRM as a framework for combining human and AI feedback to improve preference modeling for LLMs.

    Source: §7 · p. 10

    Robustness in adversarial settings, including reward-hacking evaluation, remains a direction for future work.

    Source: §7 · p. 11

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) has greatly improved the performance of modern Large Language Models (LLMs). The RLHF process is resource-intensive and technically challenging, generally requiring a large collection of human preference labels over model-generated outputs. Reinforcement Learning from AI Feedback (RLAIF) addresses this data collection challenge by leveraging synthetic preferences generated by an LLM. However, recent work has shown that synthetic preferences labels may not align well with human preference judgments. To address this, we propose a hybrid approach that unifies RLHF and RLAIF methodologies. We introduce GenRM, an iterative algorithm that trains an LLM on self-generated reasoning traces, leading to synthetic preference labels matching human preference judgments. Empirically, we show that zero-shot LLM-based judgments under-perform compared to Bradley-Terry reward models on in-distribution tasks (between 9-36%). In contrast, GenRM achieves in-distribution accuracy comparable to Bradley-Terry models, while significantly outperforming them on out-of-distribution tasks (between 10-45%). Moreover, GenRM surpasses the performance of using LLMs as judges on both in-distribution (by 9-31%) and out-of-distribution tasks (by 2- 6%). Our results show that combining the strengths of RLHF and RLAIF offers a promising approach for improving the quality of synthetic preference labels.

1 INTRODUCTION

RLHF improves LLM performance but requires substantial human preference data and infrastructure, while synthetic feedback offers an alternative whose alignment with human judgments is a concern. The paper proposes a unified RLHF–RLAIF framework that trains an LLM judge on user preferences.

1 INTRODUCTION

  • RLHF requires substantial human preference data, reward-model training infrastructure, and an online optimization loop such as PPO.The passage also notes that PPO is used to fine-tune an LLM policy against the reward model.Source: §1 · p. 1
  • RLAIF offers synthetic preference data as an alternative to human feedback, with reported Pareto improvements in balancing helpfulness and harmlessness.Bai et al. (2022b) use an LLM guided by a predefined constitution to generate and select outputs (Askell et al., 2021).Source: §1 · p. 1
  • DPO and IPO train policies directly from preference data offline, but cited evidence reports that purely offline methods may underperform online approaches.The passage also notes that some state-of-the-art models combine online DPO optimization with separate reward models.Source: §1 · pp. 1–2
  • GenRM matches Bradley-Terry reward models on in-distribution prompts, improves generalization on out-of-distribution prompts, and outperforms the base LLM in both settings.The framework uses a strong pretrained LLM as an evaluator and aligns it with user choices using a STaR-like methodology.Source: §1 · p. 2
  • Bradley-Terry reward models are described as effective on in-distribution tasks but less robust on out-of-distribution data, while LLM judges may misalign with user preferences.The proposed framework is presented as a response to both limitations.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries review RLHF, STaR, and LLM-as-a-judge as background for preference modeling. They describe a pipeline from supervised fine-tuning through reward modeling and reinforcement learning, alongside iterative rationale generation and rationalization.

2.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK

  • RLHF has three stages: supervised fine-tuning, reward modeling, and reinforcement learning.The SFT model is the base for subsequent stages.Source: §2.1 · p. 2

2.1.1 SUPERVISED FINE-TUNING (SFT)

  • SFT trains an LLM to follow instructions by maximum-likelihood next-token prediction on prompts and responses.The resulting model is denoted πSFT(y | x).Source: §2.1.1 · p. 3

2.1.2 BRADLEY-TERRY REWARD MODELING

  • Bradley-Terry modeling represents preferences through a latent reward and logistic preference probability, then trains a parameterized reward model from rankings.The reward model adds a linear predictor to the final embedding layer of the SFT model to produce a scalar estimate.Source: §2.1.2 · p. 3

2.1.3 REINFORCEMENT LEARNING (RL)

  • In RL, the learned reward model guides on-policy policy optimization, while a KL penalty limits deviation from the supervised fine-tuned reference model.The hyperparameter β controls the trade-off between reward exploitation and consistency with the reference model.Source: §2.1.3 · p. 3

2.2 SELF-TAUGHT REASONER

  • STaR iteratively bootstraps reasoning by training models to generate and refine rationales.The section introduces the approach as targeting reasoning capability.Source: §2.2 · p. 3

2.2.1 RATIONALE GENERATION BOOTSTRAPPING

  • Rationale-generation bootstrapping prompts a model for a rationale and answer, retains rationales leading to correct answers, and uses them for supervised fine-tuning.The setup does not require strong or ground-truth rationales for the problems.Source: §2.2.1 · p. 3

2.2.2 POST-RATIONALIZATION

  • STaR rationalization supplies the correct answer when the model fails, enabling backward generation of a post-rationale for training.The authors use rationalization models with different capabilities to evaluate the effect of bootstrapped reasoning-chain quality.Source: §2.2.2 · p. 4

2.3 RLAIF AND LLM-AS-A-JUDGE

  • RLAIF uses AI-generated feedback, and LLM-as-a-judge studies report over 80% agreement with human preferences, comparable to inter-expert agreement.The cited work also reports that AI harm identification improves as language-model capabilities increase, particularly with chain-of-thought reasoning.Source: §2.3 · p. 4

3 CONNECTIONS BETWEEN RLHF AND PREFERENCE MODELLING

This section connects the RLHF optimization objective to preference likelihood modeling and motivates a more general preference model. It presents an LLM-based preference distribution as an alternative to point-wise reward modeling.

3 CONNECTIONS BETWEEN RLHF AND PREFERENCE MODELLING

  • The authors describe the Bradley-Terry reward-model objective as underconstrained, which can create optimization challenges and arbitrarily low signal-to-noise ratio.Prior work uses a baseline reward from a fixed reference distribution to alleviate the issue.Source: §3 · p. 4
  • Under the Bradley-Terry formulation, the standard RLHF optimization procedure is also optimizing a preference-likelihood objective.The passage frames this connection through inversion of the reward-modeling equation.Source: §3 · p. 4
  • The proposed preference model uses an LLM to represent pϕ(yw ≻ yl|x) without assuming point-wise rewards, a special architecture, or a specific preference-distribution parameterization.Preference probabilities can be extracted from LLM output likelihoods or majority-voting counts; the model can be used with PPO or sampled preferences for iterative/online PO.Source: §3 · pp. 4–5

4 GENERATIVE REWARD MODELS: A UNIFIED RLHF-RLAIF APPROACH

The framework uses an LLM as a preference judge and trains it on preference pairs, with variants that add reasoning traces. It describes supervised and preference-optimization strategies for training reasoning-capable generative reward models.

4 GENERATIVE REWARD MODELS: A UNIFIED RLHF-RLAIF APPROACH

  • GenRM predicts answer-indicator probabilities directly, while CoT-GenRM also generates intermediate chain-of-thought reasoning before its answer.The methods begin by prompting an LLM as a zero-shot judge over two responses.Source: §4 · p. 5
  • The GenRM variant without chain-of-thought is trained with supervised fine-tuning as next-token prediction of the answer indicator.The training data consists of response pairs and their winning choice.Source: §4 · p. 5
  • CoT-GenRM with rationalization trains on both reasoning chains and rankings, including post-rationalized chains.The passage describes this as a post-rationalization approach.Source: §4 · p. 5
  • CoT-GenRM-STaR self-bootstraps intermediate reasoning and uses an SFT objective on reasoning chains that yield correct judgments.The variant is called STaR-SFT.Source: §4 · p. 5
  • STaR-DPO uses preference pairs containing rationales for correct and incorrect rankings in a DPO objective.The passage states that this variant uses all sampled data, including reasoning chains that yield wrong judgments.Source: §4 · p. 5

5 EXPERIMENTS

The experiments compare generative reward-modeling approaches with Bradley-Terry models, PairRM, and zero-shot LLM evaluation across in-domain and out-of-domain datasets.

5 EXPERIMENTS

  • Models are evaluated against Bradley-Terry reward models, PairRM, and zero-shot RLAIF using the LLaMa-3.1-8B-Instruct model.The experiments use UltraFeedback and UltraInteract for training and evaluation, with RewardBench as an additional benchmark.Source: §5 · pp. 5–6
  • RewardBench includes Chat, Chat (Hard), Reasoning, and Safety subsets for evaluating general reward modeling.UltraFeedback contains 61k preference pairs focused on instruction following; UltraInteract contains multi-turn reasoning trees involving math, code, logic, and tool use.Source: §5 · p. 6

2. How robust are reward models and how well do they generalize to OOD data?

The experiments test generalization, reasoning, rationale bootstrapping, and inference-time compute in generative reward models. STaR-DPO matches leading in-domain performance and has the strongest reported RewardBench generalization in general assistant tasks.

2. How robust are reward models and how well do they generalize to OOD data?

  • The evaluation asks whether reward models generalize out of distribution, whether reasoning and inference-time compute improve them, and whether reasoning data can be bootstrapped from the model itself.Source: §2 · p. 6

5.1 PERFORMANCE OF GENERATIVE RMS ON GENERAL ASSISTANT TASKS

  • STaR-DPO matches the best in-domain performance and has the strongest out-of-domain generalization across RewardBench categories.The comparison covers UltraFeedback in-domain data and RewardBench out-of-domain data.Source: §5.1 · pp. 6–7

5.2 PERFORMANCE ON REASONING TASKS

  • On UltraInteract, STaR-DPO reaches 90.2% versus 68.8% for the base model, while explicit reward models reach about 94%; on RewardBench reasoning, it scores 87.2%.The Bradley-Terry model, PairRM, and GenRM struggle on RewardBench; for non-reasoning subsets, STaR-DPO scores 75.0%, while LLM-as-a-judge scores 78.0%.Source: Figure 4 · p. 8, §5.2 · p. 8

5.3 BOOTSTRAPPING RATIONALES

  • STaR-Rationalizer matches STaR-DPO in-domain but underperforms it and the base LLM on RewardBench; the authors hypothesize off-policy rationales create a training distribution mismatch.Across bootstrap sources, GPT-4 rationales underperform standard STaR methods, while additional on-policy training alleviates this; LLaMa 3.1 70B rationales slightly improve UltraFeedback but worsen RewardBench generalization.Source: §5.2 · p. 8, §5.3 · p. 9, Table 1 · p. 9

5.4 USING INFERENCE-TIME COMPUTE

  • Majority voting over 32 samples improves accuracy by 1.6% on UltraFeedback and 3.8% on RewardBench, and by 4.6% on UltraInteract and 4.9% on RewardBench.Figure 5 compares models trained on UltraFeedback or UltraInteract across in-domain evaluations and RewardBench; shading indicates 95% confidence intervals.Source: Figure 5 · p. 9, §5.4 · p. 10

6 RELATED WORK

Related work spans RLAIF and LLM-as-a-judge methods, generative verifiers, hybrid reward models, and self-bootstrapped generative critics. The paper distinguishes its approach by using self-bootstrapped rationales without stronger-model supervision or additional reward-model architecture.

6 RELATED WORK

  • Prior work combines language-model feedback with constitutional AI or RLAIF, and uses LLM-as-a-judge, supervised CoT scoring, or CoT-based verification.Generative Verifiers use an SFT objective and full reference solutions; Ankner et al. combine CoT generation with Bradley-Terry reward modeling and separate strong-model rationales.Source: §6 · p. 10
  • Concurrent self-bootstrapping critics use synthetic preference pairs, DPO, or augmentation; these data-generation techniques are described as complementary to the paper’s STaR-based alignment approach.Source: §6 · p. 10

7 CONCLUSION AND FUTURE DIRECTIONS

GenRM combines human and AI feedback with self-generated reasoning traces and iterative training to improve preference modeling. Its variants retain competitive in-distribution accuracy while outperforming traditional methods on out-of-distribution tasks, with future work spanning robustness, optimization, and broader feedback settings.

7 CONCLUSION AND FUTURE DIRECTIONS

  • GenRM combines RLHF and RLAIF, using self-generated reasoning traces and iterative training to fine-tune LLMs toward human preferences.The conclusion frames this as addressing some limitations of human and AI feedback mechanisms.Source: §7 · p. 10
  • Integrating chain-of-thought reasoning into preference modeling is reported to improve in-distribution and out-of-distribution performance, including on reasoning and safety-related tasks.The conclusion attributes better generalization on complex tasks to more logical, step-wise reasoning.Source: §7 · p. 10
  • GenRM and its variants maintain competitive in-distribution accuracy while outperforming traditional methods on out-of-distribution tasks; STaR-DPO shows superior robustness and performance across benchmarks.The conclusion also describes improved generalization to out-of-distribution tasks as important for deployment in diverse environments.Source: §7 · p. 11
  • Synthetic preferences reduce reliance on resource-intensive human annotation, which the conclusion presents as a route to more scalable improvement of LLM behavior.The paper also identifies pluralistic alignment as a potential application because generative reward models can represent the reasoning underlying diverse preferences.Source: §7 · p. 11
  • Incorrect generated rationales can misalign CoT-GenRM, while iterative online optimization remains a proposed way to improve adaptation to new environments and feedback.The paper also points to multimodal feedback, adversarial robustness, and data selection for iterative training as directions for further work.Source: §7 · p. 11
  • Future directions include extending GenRM to multimodal feedback, evaluating adversarial reward hacking, and improving sample selection for iterative SFT and pairwise training.The paper characterizes optimal sample selection for these iterative RLAIF methods as understudied.Source: §7 · p. 11

A.1 PROMPTS

  • The appendix prompt variants remove tie options for LLM-as-a-Judge and STaR, alter GenRM instructions, and ask Rationalizer prompts to explain why A or B is better.The prompt structures are shown for LLM-as-a-Judge and STaR-SFT/DPO, Rationalizer, and GenRM.Source: §A.1 · p. 18, Figure 6 · p. 18, Figure 7 · p. 18

A.2.1 TRAINING HYPERPARAMETERS

The appendix records training and generation settings, then describes how STaR iterations use separate dataset splits and fresh samples from the latest model.

A.2.1 TRAINING HYPERPARAMETERS

  • The appendix states that models use the same hyperparameters across each dataset and points to Table 2 for method-specific choices.Source: §A.2.1 · p. 19, Table 2 · p. 19

A.2.2 GENERATION HYPERPARMETERS

A.3 STAR ITERATIONS

  • Each STaR iteration uses a distinct one-third dataset split, samples each pairwise data point once, and trains online on fresh samples from the latest model.UltraFeedback uses 3 epochs of generated data, while UltraInteract uses 1 epoch.Source: §A.3 · pp. 19–20

B RESULTS ACROSS ITERATIONS

The section presents STaR evaluations across training iterations, organized by training and evaluation datasets and two majority metrics.

B RESULTS ACROSS ITERATIONS

Related papers

Loading 2410.12832v1…