PaperlayerView exact revision on arXiv

Source-linked AI summary

Discretizing Reward Models

Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal, Tongshuang Wu, Graham Neubig, Yuning Mao

arXiv:2606.21795v1cs.LG

TL;DR

Reward models can be oversensitive, assigning different scores to equally good responses despite strong preference agreement. The paper proposes reward discretization via clustering and finds it reduces reward hacking while producing better policies across controlled and natural RL settings.

  • Problem

    Reward-model evaluation and reinforcement learning often overlook oversensitivity, leaving unclear whether continuous scores distinguish utility differences from variation among equally good responses.

  • Method

    The paper defines discriminative ability and specificity, proves suitable discretizations can reduce oversensitivity, and estimates thresholds using Monte Carlo dropout reward clustering.

  • Results

    Discretization is never significantly worse and is frequently much better than raw rewards across IFEval, GSM8K, and MATH, while producing less reward hacking and better policies.

  • Takeaways & Limitations

    Reward clustering can retain the benefits of learned rewards while limiting oversensitivity and potential negative impacts on reinforcement-learning policies.

  • Takeaways & Limitations

    Experiments cover only language models, while the theory assumes linear binary utility with constant-variance bounded-uniform or Gaussian noise.

Abstract

from arXiv · show

Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate response quality in the absence of verifiers or human judges. Unlike "verifiable rewards" which typically produce binary scores, reward models typically produce continuous scores, allowing them to be sensitive to fine-grained differences in responses. However, we show this apparent strength is a serious weakness: many popular reward models are oversensitive, assigning different scores to equally good responses. Theoretically, we show that seemingly perfect reward models can be highly oversensitive; empirically, this oversensitivity can lead to bad policies. In place of existing notions of "reward model accuracy," we propose evaluating reward models using distinct measures of "discriminative ability" and "specificity" (the complement of oversensitivity). As a solution, we describe a training-free algorithm that uses Monte Carlo dropout on any neural reward model to produce discrete reward clusters. Theoretically, we prove there exist discretizations that reduce oversensitivity at minimal expense of discriminative ability; empirically we show, in both controlled and natural RL settings, that discretizing rewards leads to less reward hacking and better policies than training on the original rewards.

1 Introduction

Reward models can achieve high preference agreement while remaining oversensitive to equally good responses, allowing reinforcement learning to exploit spurious rating artifacts. The paper proposes evaluating discriminative ability separately from specificity and using Monte Carlo dropout to discretize rewards, reducing oversensitivity while preserving policy quality.

  • Problem: High preference agreement does not establish that reward modeling is solved, because standard evaluation often assumes one response is always better.Top models achieve 94% and 84% agreement on RewardBench 1 and 2, respectively, while the Ties subset contributes only 1/16th of RewardBench 2’s average score.
  • Problem: Reward models can assign different scores to equally good responses, reflecting oversensitivity rather than genuine discrimination.Such differences may arise from personal context, subjective interpretations, or harmful biases in preference data.
  • Evaluation: The paper separates discriminative ability between responses of different utility from specificity among equally good responses.Leading reward models can discriminate good from bad responses while showing poor specificity, and variance among different-utility responses is beneficial whereas variance among equal-utility responses is harmful.
  • Method: Monte Carlo dropout estimates predictive variance to cluster responses into discrete reward groups without prior knowledge of discretization thresholds.The method is training-free and uses reward-model predictive variance to perform reward clustering.
  • Results: Discretization consistently reduces oversensitivity at modest discriminative cost and is frequently better than raw rewards across IFEval, GSM8K, and MATH.In the multi-task setting, discretization is never significantly worse than learning from raw rewards and can preserve primary task efficacy against stylistic overoptimization.

2 Problem Formulation

The paper formulates reward modeling with integer-valued utility classes and separates discriminative ability from specificity, the complement of oversensitivity. It shows that perfect discrimination can coexist with oversensitivity, while suitable discretization can preserve discrimination and improve specificity and combined performance.

  • 2.1 Accuracy, Discriminative Ability, and Oversensitivity: The setting seeks a policy maximizing unknown integer-valued utility u(x, y), using a learned real-valued reward model r instead.For each prompt, utility has finitely many equivalence classes of equally good responses.
  • 2.1 Accuracy, Discriminative Ability, and Oversensitivity: Discriminative ability measures correct reward ordering across unequal utilities, whereas specificity measures equal rewards within equal-utility classes and complements oversensitivity.The paper presents these constructs as more granular than prior reward-model accuracy.
  • 2.2 Reward models with perfect discriminative ability can be highly oversensitive: A linear utility signal plus bounded response-dependent noise can have perfect discriminative ability while remaining vulnerable to reward hacking.The analysis assumes r_x(y) = ϕ_x(u_x(y)) + η_x(y), with noise easier to learn than the utility transformation.
  • 2.3 There exists a discretization that maintains perfect discriminative ability: A threshold τ_x within the non-overlapping reward ranges of adjacent utility classes yields D_disc(ϵ) = 1.The construction is presented for binary discretization but can generalize to multi-level utility functions.
  • 2.4 Discretization can minimize oversensitivity: Proper discretization improves specificity over raw rewards, and thresholds at ϕ_x(u_x(a)) ± s_x/2 can achieve zero oversensitivity while maintaining perfect discrimination.For binary utility, the midpoint threshold attains both perfect discriminative ability and perfect specificity.
  • 2.5 Discretization maximizes the average of discriminative ability and specificity: T_disc(ϵ) = 1: under binary utility, the optimal midpoint threshold attains the maximal combined score at every ϵ, while the raw model never exceeds T_r(ϵ) > 5/6.The improvement gap is at least 1/6, minimized at ϵ = 2/3 and rising to 1/2 as ϵ → 0.

3 Discretization via Reward Clustering

Reward clustering treats discretization as one-dimensional clustering based on the probability that reward pairs are within a tolerance, then assigns ordinal integer rewards to clusters. It estimates predictive variance with MC dropout, using four hyperparameters and T = 4 in experiments.

  • Discretization as Clustering: Reward discretization becomes a 1-D clustering problem using pairwise probabilities P(|r_i − r_j| < ∆).Hierarchical complete-linkage clustering is applied to these pairwise distances.
  • Discretization as Clustering: Clusters require P(|r_i − r_j| < ∆) > p∗ for every pair, then receive sequential integer rewards ordered by cluster mean.∆ and p∗ are hyperparameters controlling equivalence and merging.
  • Estimating Equivalent Rewards via MC Dropout: MC dropout estimates predictive variance by running T stochastic reward-model forward passes with dropout probability d.The resulting samples provide the variance estimate needed for Gaussian reward-equivalence probabilities.
  • Estimating Equivalent Rewards via MC Dropout: Increasing T improves variance estimation but linearly increases reward-computation cost.This creates a direct accuracy–computation tradeoff for the dropout-based estimator.
  • Practical Implementation: Reward clustering has four hyperparameters, with experiments using d = 0.02 and T = 4.The parameters are ∆, p∗, d, and T; d = 0.02 was selected for sufficiently diverse samples.

4 Experiments

Experiments evaluate reward processing on RewardBench 2 and reinforcement-learning tasks, finding that discretization improves specificity and overall reward quality while reducing reward hacking and degenerate policies. Across settings, discretized rewards preserve primary task performance better than directly optimizing noisy reward models.

  • RewardBench 2 evaluation: RewardBench 2 “Ties” evaluates reward models using 0.6 × accuracy + 0.4 × margin, alongside specificity and discriminative ability under an equivalence tolerance.Rewards are normalized per prompt before computing the proposed metrics.
  • RewardBench 2 evaluation: Reward clustering uniformly improves the mean of specificity and discriminative ability for all four evaluated reward models, trading modest discriminative-ability reductions for higher specificity.The evaluated models are Skywork V1, Skywork V2, GRM, and ArmoRM.
  • RewardBench 2 evaluation: On standard RewardBench 2 metrics, reward clustering increases the margin between chosen and rejected responses for 3 out of 4 models.The table reports this alongside improvements in the proposed metrics.
  • Signal-versus-noise experiment: In mixed instruction-following and hedging-reward environments, discretization curbs overoptimization of the secondary reward while preserving primary task performance.With a 20% spurious preference ratio, continued optimization degrades task correctness; clipping is disastrous in the 80-20 setting.
  • Reinforcement-learning experiments: Discretized rewards are never worse than the raw baseline up to overlapping confidence intervals and rarely produce degenerate policies, unlike raw reward-model training.The results support using discretized reward models as drop-in replacements for direct reward-model optimization.

5 Limitations

The paper identifies three limitations: experiments cover only language models, while the theory assumes a specific linear reward model with bounded-uniform or Gaussian noise. This assumption may not reflect real-world utility functions with multiple equivalence classes of responses.

  • 5 Limitations: Experiments evaluate the algorithm only on language models, despite its few assumptions about model type.The limitation concerns the scope of empirical validation rather than the algorithm’s stated model-type assumptions.
  • 5 Limitations: The theory assumes the reward model is a linear function of binary utility plus bounded-uniform or Gaussian noise.These assumptions are made in §2.
  • 5 Limitations: Real-world utility functions often feature multiple equivalence classes of responses, which may challenge the theory’s linearity assumption.The paper notes that linearity may be sensible given a supposed utility function but may not capture this real-world structure.

6 Conclusion

The paper introduces reward-model oversensitivity, showing that many popular models can be highly oversensitive despite strong discriminative ability. It proposes reward clustering to improve the specificity–discriminative-ability tradeoff and produce better policies while limiting reward models’ negative impacts.

  • 6 Conclusion: Many popular reward models are highly oversensitive despite strong discriminative ability.The paper also shows theoretically that continuous-valued rewards with perfect discriminative ability must exhibit oversensitivity.
  • 6 Conclusion: Reward clustering improves the tradeoff between specificity and discriminative ability.The method is introduced to address undesirable behaviors arising when reinforcement learning is supervised by reward models.
  • 6 Conclusion: Discretization leads to better policies while retaining the upside of learning from reward models and limiting potential negative impacts.Discretization can reduce models’ peak ability by slowing learning, creating a risk that the paper explicitly identifies.

A Proof of Proposition 2.1 (Accuracy is the weighted sum of discriminative ability and specificity)

The proof decomposes reward-model accuracy into a specificity term for equally utility-ranked responses and directional terms measuring discriminative ability. The specificity contribution is the probability of equal utility scores weighted by the complement of reward-score disagreement.

  • Specificity: Accuracy includes a specificity term measuring whether equally utility-ranked responses receive equal reward scores.This term is (1 − P[rx(a) ≠ rx(b) | ux(a) = ux(b)]) × P[ux(a) = ux(b)].
  • Discriminative ability: The remaining terms capture correct reward ordering when one response has greater utility than the other.They combine conditional probabilities of matching reward orderings with the corresponding probabilities of utility orderings.
  • Discriminative ability: Together, these directional terms constitute the reward model’s discriminative ability.The proof labels the two weighted conditional-ordering contributions as discriminative ability.

B Proof of Proposition 2.5 (The oversensitivity of a binary-discretized reward

The proof shows that binary discretization can distinguish equally useful responses whenever their discretized rewards differ. Perfect discriminative ability is therefore possible only when the task has exactly two utility values, corresponding to binary correctness.

  • Proof of Proposition 2.5: For sufficiently small ϵ, unequal binary-discretized rewards are equivalent to a discretized-reward difference exceeding ϵ, including when utilities are equal.The proof identifies oversensitivity with nonzero disagreement between discretized rewards for responses having equal utility.
  • Proof of Proposition 2.5: Perfect discriminative ability under binary discretization requires exactly two unique utility values, so correctness must be binary.This condition follows because a binary reward cannot preserve perfect distinctions across more than two utility levels.
  • Proof of Proposition 2.5: The probability of unequal discretized rewards decomposes into the sum of the two possible strict orderings between the responses.The two events are discretized reward for a above b and discretized reward for b above a.
  • Proof of Proposition 2.5: Conditioned on equal utility, the two reward orderings are equally likely, allowing the proof to compute one ordering and double it.This symmetry accounts for both directions of discretized-reward disagreement.
  • Proof of Proposition 2.5: The ordering probability factors because the threshold-crossing events are conditionally independent when the two responses have equal utility.The proof multiplies the conditional event probabilities before doubling for the two orderings.

C Ablation of Number of Dropout Samples · D Discretization still improves reward models if we use a Gaussian noise mode · D.1 A relaxed noise model: Gaussian-distributed rewards

MC-dropout reward clustering remains effective across dropout-sample counts, while the analysis is extended from bounded uniform noise to Gaussian-distributed rewards. Under the Gaussian model, per-prompt signal-to-noise ratio determines reward-model accuracy, and raw rewards are no longer perfect.

  • C Ablation of Number of Dropout Samples: MC-dropout sample count has surprisingly little impact on the intrinsic efficacy of reward discretization for Skywork V1.The same pattern was observed with the other reward models used in the paper.
  • C Ablation of Number of Dropout Samples: Reward clustering via MC dropout is robust to the number of dropout samples used.Table 3 reports this result for Skywork V1, with T = 4 as the default setting.
  • D Discretization still improves reward models if we use a Gaussian noise mode: The bounded uniform-noise assumption guaranteed perfect discriminative ability for raw rewards, but relaxing it removes that guarantee.The prior analysis no longer applies without hard noise limits.
  • D.1 A relaxed noise model: Gaussian-distributed rewards: The relaxed model uses Gaussian noise, so the raw reward model can no longer be perfect.The model assumes binary true utility, with the concepts stated to generalize to multi-level utility functions.
  • D.1 A relaxed noise model: Gaussian-distributed rewards: For a fixed prompt, rewards are modeled as r_x(a) = ϕ_x(u_x(a)) + η_x(a), with ϕ_x(v) = s_xv + d_x and η_x(a) ∼ N(0, σ_x^2).The Gaussian noise model is specified per prompt.
  • D.1 A relaxed noise model: Gaussian-distributed rewards: Homoskedastic per-prompt noise shares σ_x and s_x across utility classes, and ρ_x := s_x/σ_x determines reward-model accuracy.The per-prompt signal-to-noise ratio is defined as ρ_x := s_x/σ_x.

D.2 Discriminative ability and oversensitivity under Gaussian noise

This section analyzes discriminative ability and oversensitivity for reward models under a homoskedastic Gaussian model with binary utility. It characterizes reward differences through utility gaps and Gaussian noise, including within-class variation scaled by signal-to-noise ratio.

  • Model assumptions: Under the homoskedastic Gaussian model, the analysis assumes binary utility.This is the setting of Proposition D.1.
  • Reward differences: For responses with ux(a) > ux(b), the reward difference equals the utility gap sx plus noise difference ηx(a) − ηx(b).The noise difference ηx(b) − ηx(a) follows N(0, 2σ2).
  • Within-class noise: Noise within each utility class is normalized as ηx/sx ∼ N(0, 1/ρ2).Rewards are measured in utility units to compare raw and discrete rewards directly.
  • Oversensitivity under equal utility: Conditional on equal utility, the normalized reward difference follows N(0, 2/ρ2).Specifically, (rx(a) − rx(b))/sx = (ηx(a) − ηx(b))/sx under ux(a) = ux(b).

D.3 Discretization preserves most discriminative ability and substantially reduces oversensitivity

Midpoint discretization maximizes discriminative ability for binary utility and yields a favorable tradeoff by substantially reducing oversensitivity. Unlike raw rewards, discretized oversensitivity is flat across tolerance levels ϵ ∈ [0, 1).

  • τx = (ϕx(ux(a)) + ϕx(ux(b)))/2 maximizes discretized discriminative ability between adjacent binary utility classes.This threshold is the midpoint between the reward means for the two utility classes.
  • The midpoint threshold maximizes discriminative ability but does not globally minimize oversensitivity, instead providing a favorable tradeoff.The paper uses this threshold throughout its analysis.
  • Ddisc(ϵ) = Φ(ρx/2)2 under the maximizing threshold.The result follows from optimizing the probability that higher-utility responses receive higher discretized rewards.
  • Specdisc(ϵ) = 1 −2Φ(ρx/2)Φ(−ρx/2), because the midpoint gives equal oversensitivity across utility classes.This specificity is the complement of the discretized reward’s oversensitivity.
  • 2Φ(ρx/2)Φ(−ρx/2) is the discretized reward’s flat oversensitivity across all ϵ ∈[0, 1).Raw oversensitivity instead follows 2Φ(−ϵρx/2), starting at 1 and decaying as ϵ increases.

D.4 The discretization tradeoff √

Discretization improves reward-model specificity while preserving enough discriminative ability to yield a strictly better combined tradeoff across all signal-to-noise ratios and tolerances considered. Its net benefit vanishes only as the raw reward approaches perfection.

  • Specificity gains: For every ρx > 0 and ϵ ∈[0, 1/2], discretization strictly improves reward-model specificity over the raw reward.Theorem D.7 establishes Specdisc(ϵ) > Specraw(ϵ) throughout this range.
  • Specificity gains: Specificity improves when rewards are treated as meaningfully different only above approximately 70% of the proportional difference in true utility values.This threshold covers most plausible practitioner choices of ϵ.
  • Averaged tradeoff: The discretization advantage grows as ϵ approaches 0, while the tradeoff gap approaches zero as ρx →∞.When the raw reward itself approaches perfection, discretization becomes unnecessary.
  • Averaged tradeoff: For every ρx > 0 and ϵ ∈[0, 1), the averaged tradeoff satisfies Tdisc(ϵ) − Traw(ϵ) > 0, so specificity gains exceed discriminative costs.The combined score averages specificity and discriminative ability at a common tolerance.
  • Global guarantee: The tradeoff gap is guaranteed positive across all ρx > 0 and ϵ ∈[0, 1), including the intermediate range where its minimum exceeds 0.0053.The intermediate signal-to-noise-ratio analysis establishes minϵ∈[2√log 2,4.2) ∆(ρ) > 0.0053.
Loading 2606.21795v1…