PaperlayerView exact revision on arXiv

Source-linked AI summary

Confronting Reward Model Overoptimization with Constrained RLHF

Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer

arXiv:2310.04373v2cs.LGcs.AI

TL;DR

Composite reward models are difficult to weight, and optimizing any reward model too far can make it a worse proxy for human preferences. This paper identifies component proxy points and uses constrained RLHF with dynamically adapted weights to keep rewards within those ranges. It reports improved evaluation performance and finds that component-reward correlation affects proxy points.

  • Problem

    Composite reward models make it difficult to choose component weights, while reward-model optimization can eventually reduce ground-truth performance.

    Source: §1 · p. 2

  • Method

    The paper identifies proxy points and uses constrained RLHF to limit each component reward to its useful range while dynamically adapting its weight.

    Source: §1 · p. 2

  • Results

    The study finds that component-reward correlation significantly influences proxy points and that constrained RL approaches prevent overoptimization and improve evaluation performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Gradient-free identification of proxy points during a single run significantly saves computation.

    Source: §1 · p. 2

    All approaches require some access to the true objective or evaluation metric, such as occasional queries to human evaluators.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models are typically aligned with human preferences by optimizing $\textit{reward models}$ (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from a composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these component RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to $\textit{overoptimization}$, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform, to our knowledge, the first study on overoptimization in composite RMs, showing that correlation between component RMs has a significant effect on the locations of these points. We then introduce an approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers. As a result, each RM stays within the range at which it is an effective proxy, improving evaluation performance. Finally, we introduce an adaptive method using gradient-free optimization to identify and optimize towards these points during a single run.

1 INTRODUCTION

Reward models can cease to track human preferences when optimized too far, and combining component models adds weighting and attribution challenges. The paper proposes constrained RLHF that dynamically weights components while keeping them within their useful proxy ranges.

1 INTRODUCTION

  • Optimizing a reward model beyond a threshold can reduce ground-truth performance, while composite rewards make it harder to choose weights and identify which component drives reward changes.Different components can stop being useful proxies at different values, making early stopping difficult.Source: §1 · p. 2
  • Composite reward weights require hyperparameter optimization, and their best values depend on training duration because of overoptimization.Combining models also loses information about individual component rewards, which may oppose one another.Source: §1 · p. 2
  • Constrained optimization keeps each component reward model at, but not beyond, its proxy point while dynamically adapting component weights.The constraints aim to prevent overoptimization of the composite reward.Source: §1 · p. 2
  • Proxy points can be identified by fitting a surface across multiple runs or by using gradient-free optimization during a single run.Both approaches periodically query the ground-truth reward; the single-run method searches for candidate proxy points as training proceeds.Source: §1 · p. 2
  • The paper studies how component-reward correlation influences proxy points and proposes constrained RL approaches that improve evaluation performance.It also reports that single-run gradient-free identification significantly saves computation.Source: §1 · p. 2

2 PRELIMINARIES: REINFORCEMENT LEARNING FROM HUMAN FEEDBACK

The preliminaries formulate reinforcement learning as policy optimization over rewards and describe how human feedback supplies reward models for language-model training. They also define reward-model overoptimization as a divergence between model reward and the human-oriented objective.

2 PRELIMINARIES: REINFORCEMENT LEARNING FROM HUMAN FEEDBACK

  • An RL agent seeks a policy maximizing expected return, with trajectories and values defined over an episodic Markov decision process.The MDP specifies states, actions, transitions, reward, discounting, and an initial-state distribution.Source: §2 · pp. 2–3
  • RLHF for language models collects preference feedback, trains a reward model to score utterances, then fine-tunes the model with reinforcement learning.The language model is treated as a policy that appends tokens to a context.Source: §2 · p. 3
  • Reward-model overoptimization occurs when increasing an agent’s score on a reward model eventually lowers quality on the preferences it is meant to learn.The paper frames the response as finding where available rewards stop being useful proxies, then limiting optimization to those points.Source: §2 · p. 3

3 FINDING PROXY POINTS

This section studies proxy points for component rewards in a DailyDialog dialogue-generation case study. It shows that the fitted composite-reward maximum differs from the proxy points obtained by considering individual reward models separately.

3 FINDING PROXY POINTS

  • The case study combines METEOR and intent-matching rewards with fixed coefficients and adds a KL penalty to discourage deviation from the initial model.The KL multiplier increases when KL exceeds its threshold and decreases otherwise.Source: §3 · pp. 3–4
  • Evaluation score initially rises as an individual reward model and KL divergence increase, then falls at the proxy point.The plotted results average five seeds, with shading showing standard error.Source: Figure 3.2 · p. 4
  • Correlation between component rewards influences where proxy points occur.The figure caption identifies correlated rewards as affecting proxy points but does not specify the direction of the effect.Source: Figure 3.1 · p. 4
  • The fitted composite-reward maximizing point differs from proxy points found by considering one reward model at a time.The maximizing point is on the fitted surface, not necessarily one attained by an individual run.Source: §3 · p. 5

4 CONSTRAINED RLHF

Constrained RLHF treats component reward models as constraints, using learned Lagrange multipliers to balance their influence while optimizing the policy. The section compares constraint formulations, practical updates, and the theoretical conditions behind convergence.

4 CONSTRAINED RLHF

  • Constrained MDPs maximize a task reward while requiring expected values of constraint rewards to meet their thresholds.The paper proposes reformulating optimization with constraints so each component reward reaches its proxy point.Source: §4 · p. 5
  • Lagrangian relaxation associates learned multipliers with constraints, and gradient descent-ascent updates the policy and multipliers using constraint violations.Multipliers are learned during optimization rather than fixed as hyperparameters.Source: §4 · p. 5
  • Convergence to a saddle point is guaranteed under basic assumptions when optimizing over occupancy measures, but direct policy optimization is non-concave in the policy.For direct policy updates, cited work gives regret or convergence results under particular approaches and assumptions.Source: §4 · p. 6
  • The proposed objective choices include equality or inequality constraints, a negative-KL task reward, and a combined reward constrained by component thresholds.A pure constraint-satisfaction objective is also considered, but it retains the challenge of setting component weights.Source: §4 · p. 6
  • µ-PPO and ξ-PPO produce the best evaluation performance over training, while constraint-based balancing makes performance more robust to longer training.The figure caption identifies µ-PPO and ξ-PPO as the best-performing constrained methods.Source: Figure 5.1 · p. 7
  • In practice, sigmoid-bounded multipliers create convex combinations of task and constraint advantages; low optimizer momentum and reward-to-go estimates can reduce update problems.The paper reports sticky multiplier weights and inaccurate early value estimates as practical issues.Source: §4 · p. 7

5 EXPERIMENTAL EVALUATION

In the evaluation, µ-PPO and ξ-PPO achieve better evaluation performance than other methods, with ξ-PPO slightly better at the end of training. Further tests examine constraint attainment, proxy-threshold quality, and single-run Nelder–Mead threshold search.

5 EXPERIMENTAL EVALUATION

  • µ-PPO and ξ-PPO outperform the other methods, with ξ-PPO performing slightly better at the end of training.The comparison uses fixed reward-model weightings for PPO and PPO-SAT selected after 10 initial runs.Source: §5 · pp. 7–8
  • µ-PPO approximately reaches or exceeds the intent and METEOR thresholds, ξ-PPO approximately hits them, and All-PPO remains below them.µ-PPO continues to increase above the intent proxy point, which may contribute to its slightly worse final performance compared with ξ-PPO.Source: §5 · p. 8, Figure 5.2 · p. 8
  • Thresholds 10% lower or higher than proxy-point thresholds reduce ξ-PPO performance compared with using the proxy points.Lower thresholds can improve initial performance but lead to a lower plateau; thresholds set too high allow proxy rewards to be optimized beyond their useful points.Source: §5 · p. 8, Figure 5.3 · p. 9
  • Proxy points that account for reward-model correlation are more effective than proxy points estimated independently.The comparison uses correlated proxy points and independent estimates that consider one reward model at a time.Source: §5 · p. 8, Figure 5.3 · p. 9

5.1 IMPROVING THRESHOLD IDENTIFICATION

  • Nelder–Mead threshold search (NM-PPO) obtains strong performance in a single run while significantly saving computation.The illustrated simplex converges to a local maximum; the authors note that the feasible region is relatively small and further tests are needed with larger regions and more than two component reward models.Source: §5.1 · p. 9, Figure 5.4 · p. 9

6 DISCUSSION

The discussion identifies continued access to the true evaluation objective and broader testing as limitations, alongside directions for improving optimization guarantees and threshold search.

6 DISCUSSION

  • All approaches require at least minimal access to the true objective or evaluation metric.The authors note that this could involve occasionally freezing training and querying human evaluators.Source: §6 · p. 10
  • Gradient descent-ascent for primal-dual policy optimization guarantees convergence only for averages, not the final policy and Lagrange multipliers.The authors suggest exploring approaches with convergence guarantees, such as ReLOAD.Source: §6 · p. 10
  • The methods need further testing on more domains and composite reward models with more components.The discussion also proposes exploring alternatives such as Bayesian optimization for single-run threshold search.Source: §6 · p. 10

A EXPERIMENTAL DETAILS

Experiments use dialogue generation on conversation snippets, with a RoBERTa-based intent reward and specified data splits and decoding.

A.1 SETTING

  • The dialogue data was split into 35k training, 3k validation, and 3k test utterances, using a context window of length 5.Inputs concatenate conversation snippets, with speaker changes marked by a special end-of-utterance token.Source: §A.1 · p. 17
  • Decoding uses top-k sampling with k = 20.Source: §A.1 · p. 17
  • The intent reward comes from a fine-tuned RoBERTa classifier, scoring 1 when the generated and reference utterances have matching intent and 0 otherwise.Source: §A.1 · p. 17

A.2 THE EVALUATION METRIC

The evaluation metric is designed to test Goodhart’s Law with respect to METEOR and intent rewards, rather than directly model human preferences.

A.2 THE EVALUATION METRIC

  • The metric targets Goodhart’s Law for METEOR and intent rewards, rather than directly modeling human preferences.Source: §A.2 · p. 17
  • It averages three lexical metrics—SACREBLEU, ROUGE2, and BLEU—and three diversity metrics: unique-3, vocab_size-3-nopunct, and max_pred_length-nopunct.The cited metrics are SACREBLEU xs (Post, 2018), ROUGE2 xr (Ganesan, 2018), BLEU xb, unique-3 xu, vocab_size-3-nopunct xv, and max_pred_length-nopunct xm.Source: §A.2 · p. 17
  • Each metric is normalized between 0 and 1 using observed values across PPO–METEOR and PPO–Intent runs; category means are then averaged.Source: §A.2 · p. 17

A.3 FITTING THE EVALUATION SCORE SURFACE

The appendix describes how the evaluation-score surface and proxy points are estimated, then gives implementation details for constrained PPO and Nelder-Mead search. It also situates the method among related constrained and regularized policy-optimization work.

A.3 FITTING THE EVALUATION SCORE SURFACE

  • The evaluation-score estimator uses degree-10 polynomials, with a Gaussian-kernel density mask hiding low-density regions below 50 per square unit.The authors describe the masking procedure as heuristic and potentially improvable.Source: §A.3 · p. 17

A.4 NELDER-MEAD PPO DETAILS

  • Nelder-Mead PPO initializes thresholds near the pretrained policy’s METEOR and intent scores, caps each evaluation at one-eighth of training, and caches prior scores within ±5%.The Nelder-Mead hyperparameters are untuned; the authors note they could be adjusted to improve performance.Source: §A.4 · p. 17

A.5 COMPUTATIONAL RESOURCES

  • The two-phase procedure gathers proxy-reward and evaluation-reward measurements, fits an evaluation predictor, selects its maximizing proxy point, then applies constrained optimization.The algorithm gathers measurements while fitting a policy on each proxy reward model, then fits the predictor to the collected data.Source: §A.5 · p. 19
  • Nelder-Mead search ranks threshold sets by evaluation score and iteratively reflects, expands, contracts, or shrinks candidates before returning the constrained policy.The algorithm descriptions specify fitting ξ-PPO to candidate thresholds and evaluating the resulting policy.Source: §A.5 · p. 19

A.6 ALGORITHM HYPERPARAMETERS

  • Table 2 is identified as reporting the experiment hyperparameters.The supplied passage gives the table caption but not its values.Source: Table 2 · p. 20

B THE KL REGULARIZATION COEFFICIENT

C ADDITIONAL RELATED WORK

  • Related work places constrained policy optimization in the CMDP literature and connects RLHF’s KL regularization with broader regularized policy-optimization approaches.The discussion also identifies non-stationarity as a factor in Lagrangian CMDP approaches.Source: C ADDITIONAL RELATED WORK · pp. 20–21

D.1 ADDITIONAL METRICS

The additional figures illustrate oscillations during constrained PPO training and show examples of Nelder-Mead search and reward-weighting sweeps.

D.1 ADDITIONAL METRICS

  • Intermediate constraint thresholds can produce wild oscillations in All-PPO’s multipliers and values, even when their averages converge.The figure passage says oscillations are worse for thresholds neither high nor low relative to an individual reward’s range.Source: Figure D.2 · p. 22
  • One Nelder-Mead trajectory converges to the global maximum proxy-point setting.The caption characterizes this as another example trajectory.Source: Figure D.3 · p. 22
  • Figure D.4 sweeps possible reward-model weightings for PPO.The supplied caption does not state which weighting performs best.Source: Figure D.4 · p. 23

D.2 SAMPLE OUTPUTS

The appendix provides sample model outputs for several prompts alongside reference responses. The examples include outputs from PPO, PPO-SAT, µ-PPO, All-PPO, and ξ-PPO.

D.2 SAMPLE OUTPUTS

  • One set of samples lists outputs from PPO, PPO-SAT, µ-PPO, All-PPO, and ξ-PPO, including PPO’s “You’re right” and All-PPO’s “So we’re waiting..” responses.These are presented as random prompts and model outputs.Source: §D.2 · p. 24
  • For the NYU-rejection prompt, the listed reference apologizes, while PPO, PPO-SAT, µ-PPO, All-PPO, and ξ-PPO provide different responses.The outputs include ξ-PPO asking about the speaker’s condition and fever.Source: §D.2 · p. 24
  • A further prompt describes waiting for Paul, and its listed reference asks what he is trying to pull..Source: §D.2 · p. 24
  • For a dialogue about illness, the reference says “Ok, go ahead,” while listed model outputs include questions about a name, food, and a quest.The outputs are attributed to the reference, PPO/PPO-SAT, µ-PPO, and All-PPO.Source: §D.2 · p. 24

Related papers

Loading 2310.04373v2…