PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Reward Model Overoptimization

Leo Gao, John Schulman, Jacob Hilton

arXiv:2210.10760v1cs.LGstat.ML

TL;DR

Reward-model overoptimization is difficult to measure because human preference labels are expensive. The paper uses a fixed gold reward model to generate synthetic labels for proxy models, then studies reinforcement learning and best-of-n optimization. It finds method-dependent functional forms whose coefficients scale smoothly with proxy reward-model size, while noting important limitations of the synthetic setting.

  • Problem

    Human preference labels are expensive, limiting careful measurement of how reward-model overoptimization scales.

  • Method

    A fixed gold reward model supplies synthetic labels for proxy reward models evaluated under reinforcement learning and best-of-n optimization.

  • Results

    The gold-reward relationship has method-dependent functional forms, with coefficients that vary smoothly with proxy reward-model parameter count.

  • Takeaways & Limitations

    These scaling laws allow prediction of attained gold reward and inform theoretical work on Goodhart’s law and AI alignment.

  • Takeaways & Limitations

    The main limitation is that the setting does not capture overoptimization caused by mismatch between ground-truth labels and actual human intent.

Abstract

from arXiv · show

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-$n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

1 Introduction

The paper measures reward-model overoptimization using a synthetic gold reward and finds method-dependent scaling relationships whose coefficients vary smoothly with reward-model size.

  • Motivation: Goodhart overoptimization occurs when optimizing proxy objectives eventually hinders the true objective, motivating measurements of its scale.The paper connects empirical measurement to theoretical models relevant to avoiding dangerous AI misalignment.
  • Scope: The study examines overoptimization in language-model reward models optimized by reinforcement learning or best-of-n sampling.Both methods are studied to determine whether overoptimization behaves differently across optimization procedures.
  • Synthetic setup: A synthetic setup replaces expensive human preference labels with labels from a fixed gold-standard reward model.This enables many measurements needed to fit scaling laws.
  • Scaling relationships: Gold reward as a function of KL divergence follows empirically validated functional forms that depend on the optimization method.The KL distance increases monotonically during reinforcement-learning training and can be computed analytically for best-of-n sampling.
  • Main findings: The α and β coefficients for both methods vary smoothly with proxy reward-model parameter count, following approximate logarithmic trends that enable prediction of attained gold reward.The paper also reports that reinforcement learning tends to be slower than best-of-n at both optimization and overoptimization when compared by KL divergence.
  • Main findings: Policy size has weak dependence: larger policies benefit less in gold reward, yet show similar overoptimization gaps and KL locations of maximum gold reward.A KL penalty raises attainable proxy scores for a given KL divergence but does not measurably improve the gold-reward–KL frontier, though this may be hyperparameter-sensitive.

2 Methodology

The experiments use an InstructGPT-like language-model environment with synthetic gold-reward labels, proxy reward models, and optimization by reinforcement learning or best-of-n sampling.

  • Environment: The InstructGPT-like environment uses text prompts, generated responses, and a learned reward model to provide optimization rewards.The reward signal is optimized using either reinforcement learning or best-of-n sampling.
  • Modeling caveat: The proposed reinforcement-learning functional form may fail near the origin because it has infinite slope there.The authors report that alternative forms produced worse fits and extrapolation.
  • Scaling experiments: Figure 1 holds policy size at 1.2B while varying reward-model size; its square-root x-axes differ across plots.The gold reward initially rises and later falls as the learned proxy is optimized.
  • Reinforcement learning: Reinforcement-learning experiments use PPO, with KL penalty set to zero except in the dedicated KL-penalty study.Most PPO hyperparameters use defaults, so other configurations may show different trends.
  • Best-of-n sampling: Best-of-n generates n trajectories and selects the response with the highest proxy reward-model score, with KL distances computed analytically.An unbiased estimator is used for intermediate n values to reduce variance and improve efficiency.
  • Reward-model training: Synthetic labels come from a fixed 6B gold reward model, while proxy reward models range from 3M to 3B parameters.The gold model labels rollout pairs for synthetic reward-model training.
  • Score processing: The paper recenters reward models so the initial policy has average reward zero and normalizes the gold-score variance.Proxy logits are recalibrated using a validation set because hard-threshold synthetic labels are miscalibrated.

3 Results

The paper finds empirically validated overoptimization scaling laws whose forms depend on optimization method, with coefficients that vary smoothly with reward-model size. Additional experiments examine reward-model data, policy size, proxy-score fitting, and KL-based comparisons.

  • Fitting and validating functional forms: The BoN functional form was hypothesized through n=1,000 and correctly advanced-predicted an experiment reaching n=60,000 (KL ≈10 nats).The validation used data only up to KL ≈6 nats when the form was proposed.
  • Fitting and validating functional forms: Proxy-score functional forms were not fit satisfactorily, and extrapolations systematically underestimated proxy rewards at higher KL.The authors leave better modeling of proxy-score behavior to future work.
  • Scaling with RM parameter count: Gold-score coefficients scale smoothly with reward-model size, while RL fits hold αRL constant and yield a clean scaling curve for βRL.These laws also predict peak gold scores for different reward-model sizes.
  • Scaling with RM data size: More reward-model data improves gold scores and reduces goodharting, although coefficient scaling is less clean than for reward-model parameter count.With fewer than roughly 2,000 comparisons, improvement over near-chance loss is minimal; performance improves thereafter.
  • Scaling with RM data size: Validation loss provides weak evidence as a common predictor of robustness across reward models with different size and data combinations.The comparison uses BoN reward-model score at n=1000.
  • Scaling with policy size: Larger policies gain less from reward-model optimization but do not overoptimize more, reaching peak gold scores at nearly the same KL and showing similar proxy-gold gaps.The reported comparison contrasts 6B and 1.2B policies; the result may be specific to this setup.
  • RL versus BoN: KL is an inadequate cross-method measure of optimization because RL consumes more KL than BoN, even though their proxy-versus-gold relationships are more similar.RL initially has a larger proxy-gold gap but eventually reaches a higher gold score than BoN.

4 Discussion

The discussion interprets overoptimization through Goodhart-effect categories and cautions that KL distance is not a valid cross-method measure of optimization. Regressional and extremal Goodhart provide explanations for observed scaling-law terms and nonmonotonic gold scores, while adversarial Goodhart remains outside the study’s scope.

  • KL as a measure of amount of optimization: KL distance should not compare optimization amounts across algorithms because different methods spend KL differently.Policy perturbations can increase KL without improving proxy or gold reward, while small targeted changes can substantially alter behavior within a small KL budget.
  • Regressional Goodhart: Regressional Goodhart divides optimization between improving gold reward and selecting for proxy noise.Under the stated distributional assumptions, the gold reward remains monotonically increasing with proxy reward; observed nonmonotonicity therefore indicates additional effects or violated assumptions.
  • Regressional Goodhart: The α term can be interpreted as regressional Goodhart because proxy scores are roughly linear in KL while the gold score has a differing linear component.
  • Extremal Goodhart: Extremal Goodhart is expected to drive nonmonotonic gold scores and primarily determine β, whose limiting behavior corresponds to unbounded utility loss.As optimization shifts samples outside the reward model’s training distribution, proxy–gold agreement weakens; the smooth decrease of β with reward-model size is interpreted as improving robustness.
  • Adversarial Goodhart: Adversarial Goodhart is not expected to appear because the studied models cannot actively manipulate the proxy, so extrapolating these scaling laws requires caution.The authors note that future increases in model capability could make adversarial strategies possible and break the observed trends.

4.3 Implications for iterated RLHF

The paper models iterated RLHF by assuming constant scaling coefficients and additive KL-like distance across iterations. Under these assumptions, iteration increases the final gold reward through a βRL d log(k) term without changing αRL-associated Goodharting, but the scaling law has a limited validity range.

  • Implications for iterated RLHF: Under constant αRL and βRL and additive distance assumptions, k iterations covering distance d/k yield RRL(d) = d(αRL − βRL log(d) + βRL log(k)).
  • Implications for iterated RLHF: Iteration increases final gold reward by βRL d log(k) while leaving αRL-associated Goodharting unchanged.The paper associates the αRL term with effects such as regressional Goodharting.
  • Implications for iterated RLHF: The iterative result holds only up to a maximum k and may break down below a minimum distance that remains to be determined.The authors identify testing the simplifying assumptions in multi-iteration RLHF as future work.
  • Policy size: Larger policies exhibit similar overoptimization during RL despite higher initial gold performance, although the policy-size exploration used only two sizes.The paper offers a Bayesian-inference-based hypothesis that policy scaling improves modeling of the human-demonstration distribution.

4.5 Limitations and Future Work

The paper identifies limitations in its synthetic, InstructGPT-based setting and calls for broader validation, alternative optimization methods, improved reward-model robustness, and empirical study of adversarial Goodhart. It also highlights uncertainty in extrapolating proxy-score behavior and policy-size trends.

  • Limitations: The setting does not capture overoptimization caused by mismatch between ground-truth labels and actual human intent.
  • Future work: The results require validation in environments and experimental setups beyond InstructGPT to assess their generality.
  • Limitations: The synthetic setting may not transfer to real-world settings, particularly because real reward models may be substantially correlated.
  • Future work: Future work should investigate reward-model robustness and optimization methods beyond reinforcement learning and best-of-n sampling.Examples include GeDi-like steering, Decision Transformers, beam-search variants, and other RL algorithms.
  • Future work: Proxy reward-model scores are difficult to extrapolate because their substantial variation makes linear regression a poor predictor at higher values.
  • Future work: Adversarial Goodhart may involve phase changes that break the observed trends, but the paper does not study it because its systems lack sufficient capability.

5 Related Work

The paper situates reward-model overoptimization within broader work on Goodhart’s law, specification gaming, overfitting, adversarial robustness, and scaling laws. It also connects the topic to reinforcement learning from human feedback and AI alignment.

  • Reward-model overoptimization is framed as a form of specification gaming, also known as reward hacking.
  • Overfitting is described as a special case of Goodhart’s law in which the proxy is performance on a finite sample set.
  • Prior work studies related phenomena including adversarial examples, robustness, and theoretical models of Goodhart’s law and reward hacking.
  • Scaling laws have been used to predict language-model properties and improve theoretical understanding of these models.
  • Reinforcement learning from human feedback is widely used in language models and is presented as an initial step toward recursive reward modeling.
  • The broader alignment problem concerns ensuring that AI systems’ goals match human goals, including for future systems that may exceed human capabilities.

Acknowlegements

The authors acknowledge colleagues who provided valuable discussion and feedback.

  • The authors thank Vivek Hebbar, Jared Kaplan, Jan Leike, Kyle McDonell, Dan Mossing, Ethan Perez, Laria Reynolds, and Jeff Wu for discussion and feedback.

A Proof of Regressional Goodhart identity

The proof establishes a conditional-expectation identity for a normally distributed variable observed through additive independent noise. The result is exact for Gaussian noise and asymptotically accurate when the noise is sufficiently concentrated.

  • The lemma assumes independent absolutely continuous X and Z, with X normally distributed and Z either normal or concentrated within δ of its mean.
  • E[X | X + Z = c] is expressed as a variance-weighted interpolation between E[X] and the observed sum offset c − E[Z].
  • The proof centers variables without loss of generality, then denotes their variances by σ^2 and τ^2.
  • For Gaussian noise, the result follows from standard conditional-distribution properties of the bivariate normal pair (X, X + Z).
  • For concentrated noise, the proof uses the probability density functions f_X and f_Z and a local expansion around the conditioning value.

B RL form details

The authors compare alternative functional forms for reinforcement-learning overoptimization. They prefer a form with finite slope at the origin because alternatives have poorer extrapolation or introduce additional degrees of freedom.

  • The candidate logarithmic form d(α_RL − β_RL log(1 + d)) has slope α_RL at the origin but substantially worse extrapolation behavior.
  • Power-law forms d(α_RL − β_RL d^γ_RL) also have slope α_RL at the origin but add another degree of freedom and fit best with small γ_RL.
  • For small γ_RL, the power-law forms approximate the selected reinforcement-learning form through the limit n(x^(1/n) − 1) = log x.

C Hyperparameters

The experiments use fixed hyperparameters and examine how reward-model data and model sizes, policy size, and KL penalties affect proxy–gold score relationships. Figures also assess fit quality, extrapolation, and individual versus averaged best-of-n behavior.

  • Table 1 lists the hyperparameters used throughout the experiments.
  • Figures 10–12 compare maximum gold scores and predicted best-of-n gold scores across reward-model sizes and dataset sizes.
  • Figure 13 reports that total data points appear less influential than unique data points for gold reward-model scores, averaged across model sizes.
  • Figures 14–16 examine KL distance during reinforcement-learning training across KL penalties, policy sizes, and reward-model sizes.
  • Figure 26 evaluates extrapolation quality, while Table 2 notes that individual-question gold scores are less orderly than averages over many questions.
Loading 2210.10760v1…