PaperlayerView exact revision on arXiv

Source-linked AI summary

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn

arXiv:2401.12086v2cs.CLcs.AIcs.LG

TL;DR

Human feedback is costly to collect, and reward-model quality depends on the preference data used for training. West-of-N self-training selects the best and worst of policy-generated responses to form synthetic preference pairs. The authors report gains comparable to or greater than adding a similar amount of human preference data.

  • Problem

    Human preference collection is costly and time-consuming, while accurately modeling preferences is critical to RLHF alignment.

    Source: §1 · p. 1

  • Method

    West-of-N samples policy responses for a query and uses a base preference model to select the best and worst as synthetic on-policy preference pairs.

    Source: §1 · p. 2

  • Results

    West-of-N produces up to a ∼2.3% increase in reward-model accuracy, compared with ∼1% from doubling training data, and also improves model quality after Best-of-N sampling or RL fine-tuning.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    West-of-N improves reward-model performance across different preference data, multiple datasets, and model architectures.

    Source: §6 · p. 10

    Pseudolabel quality depends on the base preference model, which can assign incorrect labels when imperfect.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

The success of reinforcement learning from human feedback (RLHF) in language model alignment is strongly dependent on the quality of the underlying reward model. In this paper, we present a novel approach to improve reward model quality by generating synthetic preference data, thereby augmenting the training dataset with on-policy, high-quality preference pairs. Motivated by the promising results of Best-of-N sampling strategies in language model training, we extend their application to reward model training. This results in a self-training strategy to generate preference pairs by selecting the best and worst candidates in a pool of responses to a given query. Empirically, we find that this approach improves the performance of any reward model, with an effect comparable to the addition of a similar quantity of human preference data. This work opens up new avenues of research for improving RLHF for language model alignment, by offering synthetic preference generation as a solution to reward modeling challenges.

1 Introduction

Human feedback is costly to collect, yet reward-model quality depends partly on the quantity and distribution of evaluated responses and the accuracy of preference labels. West-of-N uses the language model’s own outputs to create synthetic preference pairs for reward-model training.

1 Introduction

  • Collecting human feedback is costly and time-consuming, while RLHF uses that feedback to steer language models toward preferred outputs.The paper notes that text quality is subjective, complex, and context-dependent.Source: §1 · p. 1
  • The approach generates on-policy synthetic preference data to improve reward-model training, addressing factors that shape preference-model quality beyond data quantity alone.Those factors include the distribution of evaluated responses and preference-label accuracy.Source: §1 · p. 1
  • Best-of-N sampling selects high-scored outputs and has been used in language-model training, but the authors identify reward-model training as an unexplored application.They present West-of-N as an extension of that sampling strategy to preference modeling.Source: §1 · p. 1
  • West-of-N selects the best and worst of N responses to an unlabeled prompt and adds the resulting preference pair to the reward-model training mixture.The figure depicts the pairs being added back into training.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors report that West-of-N improves reward-model performance, with gains comparable to or greater than adding a similar amount of human preference data.They frame the work as an initial demonstration of Best-of-N sampling and semi-supervised learning for reward-model training.Source: §1 · p. 2

2 Related Work

Prior work uses Best-of-N for language-model training and synthetic methods to expand preference data, but the authors identify reward-model optimization as an uninvestigated use of Best-of-N. Existing synthetic approaches can introduce off-policy responses or noisy labels.

2 Related Work

  • Best-of-N has been used to steer language-model outputs and improve training, while its potential for reward-model optimization had not yet been investigated.The related-work passage also notes its use as an inference baseline, though it can be computationally expensive.Source: §2 · p. 2
  • Synthetic preference methods can scale data, but may introduce off-policy responses or noisy labels; the authors state that West-of-N avoids these issues.The cited alternatives include using models of differing quality or prompting for positive and negative responses.Source: §2 · pp. 2–3
  • Self-training uses a teacher trained on labeled data to generate pseudolabels for unlabeled data, then trains a student on the combined datasets.Prior applications cited include computer vision, machine translation, and vision-language models.Source: §2 · p. 3

3 Preliminaries

The preliminaries describe RLHF as using a reward model trained from pairwise preferences to steer a language model toward higher-reward outputs. The policy objective includes a regularization term that constrains deviation from a reference policy.

3 Preliminaries

  • In RLHF, a language model π maps queries to a distribution over responses and is typically initialized by supervised fine-tuning on high-quality outputs.The framework uses X and Y for query and response spaces, respectively.Source: §3 · p. 3
  • Human feedback is represented as pairwise preferences between two responses to the same query, forming the preference dataset used for reward modeling.The preferred response is denoted y+ and the other y−.Source: §3 · p. 3
  • Under the Bradley-Terry model, a pointwise reward function determines the modeled probability of one response being preferred over another.The reward model is estimated from the preference dataset by maximum likelihood.Source: §3 · p. 3
  • Reinforcement-learning optimization steers the language model toward high-reward outputs while regularizing it to remain near a reference policy.The parameter β controls the strength of that regularization.Source: §3 · p. 3

4 Generating Synthetic Preference Data

The paper constructs synthetic preferences from unlabeled queries using a base preference model and policy-generated responses, then trains a student reward model on the original and synthetic data. West-of-N selects high- and low-ranked responses, optionally filters pairs, and its labeling reliability depends on the base model.

4 Generating Synthetic Preference Data

  • The general setup maps unlabeled queries to response pairs labeled as preferences, producing synthetic data for reward-model training.That reward model’s inference distribution is the policy’s response distribution.Source: §4 · p. 3

4.1 Self-Training for Preference Modeling

  • A base preference model trained on an initial dataset labels policy-generated responses, and a student reward model is then trained on the original and synthetic datasets.The initial preferences may be human-generated or synthetic; the base model need not itself be a pointwise Bradley-Terry reward model.Source: §4.1 · p. 4
  • RL from AI Feedback is an example of the broader setup in which a language model expresses preferences over response pairs, with few-shot prompting used when no initial preference dataset is available.In that special case, the initial dataset is empty.Source: §4.1 · p. 4

4.2 West-of-N Self-Training

  • West-of-N pseudolabel quality depends on the base preference model, which can assign incorrect labels when it is imperfect.The paper notes that high-confidence filtering is a mitigation used in prior self-training work.Source: §4.2 · p. 4
  • West-of-N seeks an on-policy response pair that maximizes the base preference model’s probability of the preferred response being judged better, approximated by selecting the best- and worst-scored candidates from a sampled pool.For pairwise base reward models, the authors note that tournaments can approximately recover the extremes.Source: §4.2 · p. 4
  • As the candidate pool grows without bound, the theorem guarantees West-of-N pair-label accuracy of at least 1 − 2ε when the base preference model’s error is uniformly bounded by ε.The guarantee concerns the ground-truth preference function under the stated assumption.Source: §4.2 · p. 4
  • Optional filtering retains pairs above a preference-confidence quantile and applies likelihood thresholds to both responses to keep them in-distribution.Thresholds are selected using validation performance.Source: §4.2 · pp. 4–5

5 Experiments

Across three datasets, West-of-N improves reward-model performance using synthetic preferences selected from policy-generated responses. The experiments compare its gains with additional human feedback and other synthetic methods, examine why it helps, and test iterative self-training.

5 Experiments

  • Experiments span Reddit TL;DR, AnthropicHH, and UltraFeedback, using 50% human feedback as base and a 1:1 base/West-of-N preference mix.The study also compares synthetic-generation baselines and evaluates held-out human-preference accuracy and downstream generations.Source: §5 · pp. 5–6

5.1 Performance Gains with West-of-N

  • Up to ∼2.3% increase in reward-model accuracy from West-of-N compares with ∼1% from doubling human-feedback data, and gains exceed other synthetic methods across datasets.The authors report smaller West-of-N gains when the base reward model has lower accuracy.Source: Figure 2 · p. 6, §5.1 · pp. 6–7
  • West-of-N improves human- or AI-labeled preference bases across model backbones; RLAIF combined with West-of-N forms a fully synthetic pipeline with gains similar to a human-feedback base.The reported gains also hold when the base preferences are synthetically generated through prompting.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 Understanding West-of-N Gains

  • A pairwise teacher avoids the Bradley-Terry transitivity assumption, and its higher-quality labels are associated with greater gains; tournament ranking estimates extremes because full comparisons scale with N^2.Teacher test accuracy is 72.5% on Reddit TL;DR, 66.9% on Anthropic Helpful, and 71.4% on Anthropic Harmless.Source: §5.2 · p. 8
  • As N increases, West-of-N gains and pseudo-preference accuracy rise; for N = 2, labeling every generation pair introduces enough noise to harm performance.Pseudo-preference accuracy rises beyond that of high-confidence human-feedback data.Source: §5.2 · p. 8
  • Filtering low-confidence pseudo-pairs or lowest-likelihood responses adds gains; likelihood filtering trades smaller off-policy accuracy gains for higher-quality on-policy Best-of-N generations.The pseudo-label confidence criterion is Pθ(y+ ≻y−|x), while response likelihood is log π(y±).Source: §5.2 · pp. 8–9, Figure 5 · p. 8
  • West-of-N responses are orders of magnitude more likely under the generating policy than human-feedback responses; at N = 64, they remain more likely, with no observed reward hacking.Likelihood declines as N grows, creating a potential out-of-distribution risk for the policy and base reward model.Source: §5.2 · p. 9

5.3 Iterative West-of-N

  • Iterative training reduces held-out human-feedback accuracy gains yet significantly improves Best-of-N candidates; each round uses the previous reward model and keeps human-feedback/West-of-N pairs 1:1.The authors report that this on-policy improvement also benefits downstream reward-model use.Source: §5.3 · pp. 9–10

6 Conclusion

The conclusion presents West-of-N as a strategy for improving reward models with on-policy synthetic preference pairs, reporting gains across preference data types, datasets, and model architectures.

6 Conclusion

  • West-of-N improves reward-model performance across different preference-data types, multiple datasets, and model architectures.The conclusion frames the results as evidence for Best-of-N sampling and semi-supervised learning in preference modeling.Source: §6 · p. 10
  • The approach boosts reward models for RLHF by generating high-quality on-policy preference pairs.Source: §6 · p. 10
  • The authors identify Best-of-N sampling and semi-supervised learning as promising strategies for preference modeling.Source: §6 · p. 10

7 Impact Statement

The impact statement links improved language-model alignment to potential societal benefits and says synthetic preferences may reduce human-annotation needs. It also notes that West-of-N performance depends on the quality of the base preference data.

7 Impact Statement

  • Better alignment to human values is presented as a potential route to more responsible and beneficial AI systems.Source: §7 · p. 10
  • Synthetic preferences may reduce human-annotated data needs for RLHF, potentially lowering costs and facilitating development and deployment.Source: §7 · p. 10
  • West-of-N performance depends on base preference-data quality, and saved collection costs could support quality control of the data and final reward models.Source: §7 · p. 10

A Theoretical details

The theoretical details state that the West-of-N pair is correctly labeled with probability at least 1 − 2ε under a uniform preference-model error bound. Head-on TL;DR evaluations also report superior performance for policies trained with West-of-N.

A Theoretical details

  • The West-of-N preference pair is selected from the N responses to a prompt by maximizing the model's preference probability for y+ over y−.Source: A Theoretical details · p. 13
  • Under a uniform preference-model error bound of ε, the selected West-of-N pair is correctly labeled with probability at least 1 − 2ε.Source: A Theoretical details · p. 13

B.1 Head-on Win Rates

  • West-of-N-trained policies outperform those trained with HF100% or HF50% + RLAIF in head-on TL;DR comparisons.Figure 7 reports wins, draws, and losses averaged over 1,000 test queries; the prose says these results correlate with Figure 2a.Source: Figure 7 · p. 13, §B.1 · p. 13

B.2 Base Preference Data Quality

West-of-N improves performance across the six tested base preference datasets, while the gains depend on base-model accuracy and label noise.

B.2 Base Preference Data Quality

  • West-of-N improves performance in all six tested base-preference-data settings, with gains depending on base-model accuracy.The settings include human-feedback, RLAIF, and RLCD preference pairs across Reddit TL;DR, Anthropic HH, and UltraFeedback.Source: §B.2 · p. 13, Figure 8 · p. 14
  • Noisy labels in Anthropic Helpful or Reddit RLCD increase pseudo-label deviation from ground truth and weaken self-training.Source: §B.2 · p. 13

B.3 Minimizing Pseudolabeling Error

West-of-N is presented as minimizing incorrect pseudolabels. Unlike naive self-training, which can harm performance through poorly labeled preferences, it overcomes base-model limitations by ensuring preferences are correctly labeled.

B.3 Minimizing Pseudolabeling Error

  • West-of-N minimizes incorrectly labeled pseudodata, while naive self-training can harm performance by fitting poorly labeled pseudo-preferences.The comparison includes naive self-training, self-training with (yBoN, yrandom), and West-of-N with (yBoN, yWoN).Source: §B.3 · p. 14
  • West-of-N overcomes base-model limitations by ensuring pseudo-preferences are correctly labeled.Source: §B.3 · p. 14

C Implementation details

Implementation uses sampled best- and worst-of-N responses to create preference pairs, filters those pairs by a selected score quantile, and trains reward models and policies with specified configurations.

C Implementation details

  • Reward models train for 20,000 steps and policies for 10,000 or 8 epochs, with checkpoints selected by validation accuracy or reward.The stated settings also include 4,000 supervised-finetuning steps, batch sizes of 32 for supervised finetuning and 8 for reward and policy training, and a KL coefficient β = 0.05.Source: C Implementation details · pp. 14–15
  • The paper estimates that training individual reward models takes approximately a day and running all experiments takes two weeks, with training and inference performed on TPUs with 32 GiB HBM memory.Source: C Implementation details · p. 14
  • Teacher models are pairwise, while student and baseline implementations use pointwise reward models for RLHF.Pointwise models output good or bad; pairwise models choose between responses A and B.Source: C Implementation details · p. 15
  • Best- and worst-of-N responses are selected from SFT or latest RL-finetuned policy samples, using scores or an elimination tournament depending on the base reward model.Pointwise models score all candidates; pairwise models compare random pairs and then successive winners and losers until one of each remains.Source: C Implementation details · p. 15
  • West-of-N pairs are filtered by retaining examples above a selected quantile of joint response log-likelihood or base-model preference probability.The quantile is chosen using validation performance.Source: C Implementation details · p. 15
  • RLAIF and RLCD are synthetic-preference baselines, and RLCD’s TL;DR prompts use adjective pairs inspired by coherence, quality, accuracy, and completeness.The adjective pairs are listed in Table 3.Source: C Implementation details · p. 15, Table 3 · p. 15

D Evaluation

Evaluation measures reward-model accuracy on human-preference test sets and compares sampled or RL-finetuned responses against SFT responses using autoraters and GPT-4. The included prompts illustrate judgments for summary quality, helpfulness, and harmlessness.

D Evaluation

  • Reward-model accuracy is measured on human-preference test sets, while sampled and RL-finetuned responses are compared with SFT responses using autoraters and GPT-4.Win rates average 1,000 response pairs; autoraters are trained on test data, and GPT-4 judgments average two calls with response order flipped.Source: D Evaluation · p. 16

D.1 Evaluation prompt for Reddit TL;DR dataset (Liu et al., 2023)

  • Reddit TL;DR prompt examples judge whether summaries preserve key facts, avoid redundancy, and remain concise.Examples include choosing between equivalent paraphrases and preferring a summary that avoids misattributing who confronted someone.Source: §D.1 · pp. 16–18

D.2 Evaluation prompt for Anthropic Helpful dataset (Liu et al., 2023)

  • Anthropic Helpful prompts compare response helpfulness, including whether a reply addresses the user’s perspective or offers specific information.The examples include questions about garam masala and whether to start space exploration on the Moon.Source: §D.2 · pp. 18–19

D.3 Evaluation prompt for Anthropic Harmless dataset

  • Anthropic Harmless prompts ask which response is least harmful, with examples involving illegal conduct, insults, and potentially enabling advice.The examples include choosing between specific advice about hiding keys and more general guidance.Source: §D.3 · pp. 19–20

Related papers

Loading 2401.12086v2…