PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward Model Ensembles Help Mitigate Overoptimization

Thomas Coste, Usman Anwar, Robert Kirk, David Krueger

arXiv:2310.02743v2cs.LG

TL;DR

RLHF reward models are imperfect proxies whose optimization can produce overoptimization, improving proxy scores while degrading true-reward performance. This paper evaluates ensemble-based conservative objectives in BoN and PPO, including a 25% label-noise extension of Gao et al.’s synthetic setup. The authors report that these methods practically eliminate BoN overoptimization and reduce PPO overoptimization, with PPO requiring a small KL penalty for complete mitigation.

  • Problem

    RLHF learned reward models approximate human preferences imperfectly, so optimizing them can eventually improve proxy reward while true-reward performance regresses.

  • Method

    The paper systematically evaluates ensemble-based conservative optimization, including WCO and UWO, with BoN and PPO in Gao et al.’s synthetic setup extended to 25% label noise.

  • Results

    Conservative optimization practically eliminates BoN overoptimization with improvements up to 70%; for PPO, it reduces overoptimization and outperforms single-reward-model optimization, while a small KL penalty completes mitigation without performance cost.

  • Takeaways & Limitations

    Ensemble-based conservative optimization can effectively counter reward-model overoptimization, with gains that are orthogonal to those from scaling reward-model size.

  • Takeaways & Limitations

    The study uses offline RLHF and therefore does not establish whether ensembling provides similar gains in online RLHF with periodically retrained reward models.

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.

1 INTRODUCTION

This work systematically studies ensemble-based conservative optimization for mitigating RLHF reward-model overoptimization, extending Gao et al.’s setup with 25% label noise. The authors report improved performance and strong mitigation of overoptimization in both noisy and noiseless settings.

  • Motivation: RLHF reward models can be exploited: proxy-reward optimization may improve measured reward while regressing on the true reward.Gao et al. studied this systematically using a larger gold reward model to generate synthetic preference labels.
  • Approach: The study tests whether reward-model ensembles combined with conservative objectives can mitigate overoptimization.The setup follows Gao et al. (2023) while adding 25% label noise.
  • Approach: 25% label noise extends the synthetic setup toward real-world RLHF conditions, where human-annotator agreement is typically 60–75%.
  • Motivation: Ensemble-based conservative optimization provides gains beyond simply scaling reward-model size, while requiring fine-tuning multiple pretrained copies rather than costly additional pretraining.The scaling results indicate that ensemble gains are orthogonal to reward-model-size gains.
  • Contributions: The paper’s contributions include the first ensemble study for RLHF overoptimization, strong BoN gains, PPO improvements, and robustness studies for newly introduced hyperparameters.The reported BoN improvement reaches up to 70% in some cases.

2 BACKGROUND

This section introduces BoN and PPO as policy-optimization methods and explains overoptimization as divergence between proxy-reward progress and true-reward performance. It also describes Gao et al.’s synthetic gold-reward setup for studying the problem.

  • Best-of-n Sampling: BoN sampling generates n responses per prompt and returns the response with the highest proxy-reward score.The KL distance is defined analytically as a function of n to measure optimization degree.
  • Proximal Policy Optimization: PPO repeatedly makes small policy-gradient updates to maximize a reward function during online reinforcement learning.
  • Proximal Policy Optimization: A KL penalty regularizes PPO by preventing the optimized policy from deviating too far from the initial pretrained policy.Optimization degree is measured using KL distance between the initial and optimized policies.
  • Overoptimization: Overoptimization occurs when optimizing a learned proxy reward appears successful but true-reward performance eventually regresses.The learned reward model approximates human preferences but is not identical to the true reward function.
  • Overoptimization: Gao et al. study overoptimization by using a much larger gold reward model to score responses and generate preference labels for proxy reward models.

3 METHOD

The method trains an ensemble of proxy reward models and uses conservative objectives during policy optimization to reduce exploitation of inaccurate reward estimates.

  • Ensemble reward models: An ensemble trains multiple proxy reward models with identical data and hyperparameters but different random seeds.Different reward-head initialization and data-shuffling orders produce ensemble diversity.
  • Mean optimization: Mean optimization averages outputs across ensemble members but is not conservative against one model overestimating reward.A single faulty member can still make the mean exploitable.
  • Worst-case optimization: Worst-case optimization selects the lowest ensemble reward, helping avoid overoptimization when at least one member does not overestimate the true reward.WCO has no tuning hyperparameters but may impose a performance penalty because it is highly conservative.
  • Uncertainty-weighted optimization: Uncertainty-weighted optimization combines mean reward with intra-ensemble variance and penalizes responses receiving high disagreement.The uncertainty coefficient λ controls the penalty weight.

4 EXPERIMENTAL SETUP

The experiments reproduce Gao et al.’s setup with open-source models and datasets, adding 25% label noise, and evaluate BoN and PPO under a defined RLHF pipeline.

  • Setup: The study uses open-source models and datasets while training proxy reward models with 25% label noise.The setup is based on Gao et al. (2023), with a qualitative reproduction in the appendix.
  • Pretrained models: The policy model is the 1.4B Pythia model, while proxy reward models have sizes 7M, 44M, and 1.3B.These proxy models are created from Pythia models by removing unembedding layers and adding scalar reward heads.
  • Pretrained models: A 7B AlpacaFarm human-preference reward model serves as the gold reward model and is substantially larger than every proxy reward model.Its size is comparable to the gold reward model used by Gao et al. (2023).
  • RLHF pipeline: The pipeline begins with supervised fine-tuning of policy and proxy reward models on 10k AlpacaFarm instruction demonstrations.This stage is intended to improve instruction-following capabilities.
  • RLHF pipeline: Preference data are generated by sampling two responses per instruction and scoring them with the gold reward model before labeling preferences.The setup incorporates human disagreement rates of about 25% or more.
  • RLHF pipeline: Proxy reward models are trained with cross-entropy on 46,000 prompts for five epochs unless otherwise specified.The reported validation losses for trained reward models are provided separately.
  • RLHF pipeline: Ensembles use all available training data for every reward model, with diversity arising from random seeds and training order.Training on less data produced higher validation loss and poorer performance.
  • Policy optimization: BoN is evaluated up to nmax = 12,500 samples, while PPO is trained for 3000 steps.The BoN limit reflects exponentially increasing evaluation cost at higher KL distances.

5 RESULTS

Across BoN and PPO, ensemble-based conservative objectives reduce overoptimization and improve final gold-reward performance, including under 25% label noise. Robustness studies show these gains persist across reward-model sizes, dataset sizes, ensemble cardinalities, and uncertainty-penalty values.

  • 5.1 BEST-OF-N SAMPLING: BoN ensembles avoid overoptimization with WCO and UWO, while mean optimization overoptimizes under 25% label noise.
  • 5.1 BEST-OF-N SAMPLING: ∼30% and ∼75% improvements in BoN final performance occur with ensembles in noiseless and 25% noisy settings, respectively.
  • 5.2 PROXIMAL POLICY OPTIMIZATION: PPO WCO and UWO reduce overoptimization and outperform other objectives without a KL penalty, but do not eliminate overoptimization completely.
  • 5.2 PROXIMAL POLICY OPTIMIZATION: 0.01 KL penalty coefficient lets PPO WCO and UWO prevent overoptimization without notable performance reduction, whereas KL-only prevention requires 0.2 and significantly harms performance.
  • 5.2 PROXIMAL POLICY OPTIMIZATION: WCO and UWO match or outperform single-reward-model PPO optimization for every KL penalty value, with comprehensive gains at small penalties.
  • 5.3 MODEL AND DATA SCALING RESULTS: Ensemble gains remain additional to scaling reward-model size or training-data size, while 4- or 5-member ensembles and most uncertainty-penalty values perform similarly well.The scaling results use fixed training budgets; 1.3B PPO models receive 6000 steps because larger reward models optimize more slowly.

6 DISCUSSION

The study demonstrates that ensemble-based conservative optimization improves performance and combats overoptimization in RLHF, while identifying settings that future work should test.

  • Future work should replicate these findings on other RLHF datasets and larger-scale language models.
  • The experiments use offline RLHF, where human feedback is collected upfront and reward models are not updated during policy optimization.
  • Online RLHF periodically retrains reward models on freshly collected human data, so whether ensembling provides similar gains remains an open question.

7 RELATED WORKS

Related work situates overoptimization within RLHF and reward hacking, reviews ensemble uncertainty estimation, and introduces the paper’s policy-optimization context and implementation details.

  • Overoptimization in RLHF: Gao et al. (2023) systematically studied overoptimization in language-model fine-tuning, and this work evaluates ensembles within their setup.
  • Use of Ensembles in (Model-Based) RL: Ensembles are used in deep learning to estimate uncertainty and in model-based reinforcement learning to help avoid distribution shift.
  • Reward Hacking: Overoptimization in RLHF is a form of reward hacking in which optimizing a proxy reward can diverge from the true reward.
  • The appendix documents estimators, pipeline hyperparameters, and tables for SFT, reward modeling, PPO, and generation.

D.2 ALPACAFARM DATASET DETAILS

The experiments use AlpacaFarm instructions and preference data, with specified splits and minimal prompt formatting for generation and reward modeling.

  • Dataset: The AlpacaFarm dataset uses Alpaca’s 52,000 samples and includes labeled, preference, unlabeled, validation, and unused splits.
  • Dataset: The 10k preference split supplies pairwise preference labels, while the 20k unlabeled split supports algorithms such as PPO.
  • Prompt Formatting: Instructions are formatted minimally using the OpenAssistant v2 format and GPTNeoXTokenizer-compatible special tokens.
  • Prompt Formatting: Answer-generation prompts combine instructions and optional inputs, then delimit the instruction and answer with designated tokens.
  • Prompt Formatting: Reward-modeling prompts append an answer to the instruction prompt for evaluation.

E REPRODUCTION OF RESULTS OF GAO ET AL. (2023)

The reproduction study confirms Gao et al.’s overoptimization trends using open-source Pythia-based models, AlpacaFarm data, and a gold reward model, across BoN and PPO experiments.

  • Reproduction Setup: The study reproduces Gao et al.’s results using Pythia-derived open-source policy and proxy reward models with the AlpacaFarm reward model as gold evaluator.
  • BoN: 25% label noise is used in the BoN reproduction because overoptimization was difficult to observe under the study’s limited BoN optimization capability.
  • BoN: BoN gold-reward scores follow a trend similar to Gao et al. when reward-model and data sizes vary.
  • PPO: PPO shows greater overoptimization as optimization is pushed further, with reward-model data size varied and model size fixed at 44M.
  • Reward Model Training: Validation-loss variation is not predictive of whether a reward model is robust to overoptimization.
  • Training Curves: Separate training curves are reported for single reward models, while averaged curves appear in the main figures.

F.5 ADDITIONAL NO LABEL NOISE RESULTS

Additional experiments examine how conservative optimization behaves across KL penalties, reward-model sizes, training-data amounts, ensemble cardinalities, uncertainty weights, noise levels, and extended PPO optimization.

  • Conservative optimization is evaluated across KL penalty weights in the no-label-noise setting.
  • Final gold reward performance is compared across objectives for reward models with varying parameter sizes and fixed noiseless training data.
  • Final gold reward performance is also compared across objectives for 44M reward models trained with varying amounts of noiseless data.
  • Ensemble cardinality is varied to assess its impact on mean, UWO, and WCO performance.
  • Performance is tested across uncertainty penalty weights and ensemble sizes, with considerable robustness to the uncertainty-penalty value.
  • With 35% label noise, ensembles retain performance gains and overoptimization mitigation for both BoN and PPO.
  • With 25% label noise and PPO optimization extended to 10k steps, ensemble methods remain stable, reducing the need for early stopping.

F.9 WIN-RATE EVALUATION OF ENSEMBLE METHODS

The section evaluates ensemble methods by comparing final policies against single reward models using win-rates across reward-model scales.

  • Final policy win-rates compare ensemble methods against single reward models across multiple reward-model scales.
  • For BoN, the evaluation uses the unbiased estimator at n = 12500 and compares gold scores prompt by prompt.
  • Table 6 reports win-rates for ensemble methods against 44M single reward models, including standard deviation across five single reward models.
  • Table 7 reports win-rates for smaller 7M and larger 1.3B reward models with 25% noise, using the 6000-step PPO policy for 1.3B models.

F.10 QUALITATIVE SAMPLES

Qualitative samples compare outputs from single and ensemble-based objectives on evaluation prompts, highlighting differences in answer quality and overoptimization behavior.

  • The qualitative evaluation samples final policies at n = 12500 for BoN and 3000 steps for PPO.
  • Single reward model optimization produces poor answers, including hallucinations for BoN and long, repetitive responses for PPO.
  • Conservative optimization methods produce strong qualitative results when single reward models are struggling.
  • The airport responses are likewise labeled by method, including Mean, WCO, and UWO under PPO.
  • The displayed cloud-computing responses are labeled by objective and optimization method, including WCO, UWO, Mean, and Single RM.
  • The examples include responses to prompts asking for a cloud-computing analogy and the closest airport to Seattle, Washington.
Loading 2310.02743v2…