PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Reward Model a Good Teacher? An Optimization Perspective

Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora

arXiv:2503.15477v4cs.LGcs.AIcs.CLstat.ML

TL;DR

RLHF reward models are commonly evaluated by accuracy, but it is unclear whether ranking accuracy fully captures their teaching quality. This paper analyzes the issue through optimization, proving and empirically demonstrating that policy-dependent reward variance governs landscape flatness and optimization rate. It concludes that efficient reward models need sufficient variance in addition to accuracy, while evaluation should account for the language model being aligned.

  • Problem

    It remains unclear whether accuracy, the primary reward-model evaluation criterion, fully measures what makes a reward model an effective RLHF teacher.

  • Method

    The paper combines theoretical analysis of the relationship between reward variance and policy-gradient optimization with experiments across reward models, language models, and RLHF datasets.

  • Results

    Reward models with low variance induce flat RLHF objectives and slow reward maximization; experiments up to 8B parameters corroborate the interplay among variance, accuracy, and optimization rate.

  • Takeaways & Limitations

    Reward-model evaluation should consider sufficient reward variance and its interaction with the language model, rather than relying on accuracy alone.

  • Takeaways & Limitations

    How to modify or train reward models to induce higher reward variance remains an open question, and naive reward scaling can amplify gradient-estimation noise.

Abstract

from arXiv · show

The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even a perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another. These results establish a fundamental limitation of evaluating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B parameters corroborate our theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Overall, our findings highlight that beyond accuracy, a reward model needs to induce sufficient variance for efficient optimization.

1 Introduction

RLHF reward models are usually judged by accuracy, but the paper asks whether accuracy fully captures what makes a reward model an effective teacher. From an optimization perspective, it argues that reward variance and the guided language model also shape optimization efficiency.

  • 1 Introduction: RLHF trains a proxy reward model from preference data before aligning a language-model policy by maximizing that proxy reward.The preference data is assumed to reflect an unknown ground-truth reward encoding desirable traits.
  • 1 Introduction: Accuracy measures ranking agreement with the ground-truth reward, yet recent evidence shows that more accurate reward models do not necessarily produce stronger language models after RLHF.This motivates evaluating reward-model quality beyond accuracy alone.
  • 1 Introduction: Low reward variance can make the RLHF objective landscape flat, causing both proxy and ground-truth rewards to increase extremely slowly during policy-gradient optimization.This can occur when the reward model does not sufficiently separate outputs probable under the current policy.
  • 1 Introduction: A perfectly accurate reward model can be a worse teacher than a relatively inaccurate model when the former induces lower reward variance and therefore slower ground-truth reward maximization.The paper proves that reward variance is not tied to accuracy.
  • 1 Introduction: The same reward model can induce high reward variance for one language model but low variance for another, making reward-model quality dependent on the policy it guides.This limits evaluations that rank reward models solely by accuracy or independently of the language model.
  • 1 Introduction: Experiments with models up to 8B parameters corroborate the theory and show the interplay among reward variance, accuracy, and reward maximization rate.Even with accessible ground-truth reward, a proxy can perform better when it induces higher reward variance.

2 Preliminaries

The preliminaries define RLHF’s pipeline and distinguish reward-model accuracy from policy-dependent reward variance. Accuracy concerns ranking correctness, whereas reward variance concerns the separation of rewards assigned to outputs likely under the policy.

  • 2.1 Aligning Language Models via Reinforcement Learning: RLHF commonly follows supervised fine-tuning, using the resulting policy as the initial policy for reward-model-guided alignment.The standard pipeline then trains or selects a proxy reward model and updates the policy with policy-gradient methods.
  • 2.1 Aligning Language Models via Reinforcement Learning: The proxy reward model approximates an inaccessible ground-truth reward inferred from preference data, and policy gradients maximize expected proxy reward while KL regularization limits deviation from the initial policy.The intended link is that increasing proxy reward also increases expected ground-truth reward.
  • 2.2 Accuracy: Accuracy measures whether a reward model correctly ranks output pairs according to the ground-truth reward.For a prompt set, accuracy is averaged across prompts.
  • 2.2 Accuracy: Benchmark accuracy can be misleading when its prompts or outputs differ substantially from the policy-gradient training prompts and on-policy outputs.The analysis therefore considers accuracy over the training prompt set and supports arbitrary output-pair distributions.
  • 2.3 Reward Variance: Reward variance measures how well a reward model separates rewards assigned to outputs probable under a given policy and prompt.Unlike accuracy, it depends on the degree of reward separation rather than only output rankings.
  • 2.3 Reward Variance: Reward scaling by c ≥ 0 changes reward variance by c2, so normalization is used to distinguish separation from arbitrary reward scale.The paper notes that reward variance should not be confused with variance in gradient estimates.

3 Theory: Optimization Perspective on What Makes a Good Reward Model

The theory links efficient RLHF optimization to reward variance induced jointly by a reward model and the initial policy, rather than accuracy alone. It also identifies technical assumptions and boundaries under which the comparisons are established.

  • Reward variance and optimization: Low reward variance makes policy-gradient optimization slow because it produces a flat objective landscape.The lower bound on improvement time grows inversely with the initial policy’s average induced reward variance.
  • Technical setting: The analysis studies small-learning-rate policy gradient as gradient flow, with some results proved for tabular policies and exact gradients.These simplifications are empirically checked against standard language-model policy-gradient methods, including RLOO and GRPO.
  • Accuracy is insufficient: Accuracy alone does not determine teaching quality: a perfectly accurate reward model can yield arbitrarily slow ground-truth reward maximization, while a less accurate model can be faster.Accuracy measures ranking agreement, whereas reward variance depends on the degree of reward separation among outputs.
  • Scope and interpretation: Highly accurate reward models are not inherently poor teachers, and accuracy remains useful because it can reduce susceptibility to reward hacking.The paper also notes that the role of accuracy may differ across alignment methods; for Best-of-N sampling, a perfectly accurate model is always optimal.
  • Policy dependence: Reward-model quality depends on the aligned policy, because the same model can induce high variance for one policy and low variance for another.The theory therefore predicts different reward models can be better for different initial policies, with one model achieving tγ = O(1) where another can be arbitrarily slow.

4 Experiments

Experiments across reward models, policies, and datasets support the theory that reward variance and accuracy jointly shape RLHF optimization. Higher variance often accelerates reward maximization, while accuracy alone does not guarantee effective teaching.

  • 4 Experiments: The experiments covered Pythia and Llama-3.2 models, reward models up to 8B parameters, and AlpacaFarm and UltraFeedback datasets.Additional experiments varied the initial model, SFT data, and ground truth reward model.
  • 4.1.2 Results: Reward variance strongly correlates with increases in both proxy and ground truth rewards during policy gradient.For ground truth reward, combining reward variance with accuracy is more indicative because proxy improvements from inaccurate models translate less reliably.
  • 4.1.2 Results: A perfectly accurate, low-variance reward model can maximize ground truth reward significantly more slowly than less accurate models.Accuracy by itself was not indicative of reward increase in the reported experiments.
  • 4.1.2 Results: Proxy reward models can outperform direct ground truth optimization during the first few epochs, although prolonged training may cause saturation or reward hacking.The comparison used policy gradient with RLOO and evaluated both proxy and ground truth reward increases.
  • 4.1.2 Results: High off-policy accuracy can be misleading when on-policy accuracy is low, producing poor teachers that tend to cause reward hacking.The experiments measure accuracy separately on policy-sampled outputs and UltraFeedback outputs.
  • 4.2.2 Results: For different initial policies, different reward models achieved the highest ground truth reward increase.RM-Gemma-2B induced the highest reward variance and largest proxy-reward increase for all initial policies, but its inaccuracy prevented higher ground truth reward.

5 Related Work

Related work connects this paper to analyses of policy-gradient optimization, reward transformations and ensembles, and teacher quality in knowledge distillation. The paper distinguishes its focus on how reward-model properties affect optimization in the standard RLHF pipeline.

  • Theoretical analyses of RLHF: Prior RLHF analyses largely study specialized algorithms that differ from the standard RLHF pipeline and provide guarantees under technical assumptions.This work characterizes how reward-model properties affect policy-gradient optimization in RLHF.
  • Transforming and ensembling reward models: Reward-shaping research studies transformations that preserve optimal policies, while RLHF work has proposed reward-model transformations and ensembles.The paper notes that the optimization effects of these modifications remain relevant to the broader question of good reward models.
  • Optimization difficulties in policy gradient: Policy-gradient objectives are non-concave, and gradients can vanish for softmax policies when reward variance is low.The paper uses this relationship to analyze reward-model effects on RLHF optimization.
  • Optimization difficulties in policy gradient: Natural policy gradient and entropy regularization can theoretically address poor optimization landscapes but are impractical for large language models in the discussed setting.Natural policy gradient requires inverting a P × P matrix, while entropy regularization often shows little improvement for language generation.
  • Stronger teachers do not always provide better supervision: The finding that more accurate reward models are not always better teachers resembles knowledge-distillation results where stronger models do not necessarily provide better labels.The settings differ because strong models inherently provide weak supervision in the cited distillation work, whereas RLHF concerns reward-model teaching.

6 Conclusion

The paper concludes that reward-model quality should be evaluated through optimization-relevant properties beyond accuracy, especially reward variance and its interaction with the aligned language model. It also identifies open directions for improving reward variance, extending the analysis beyond RLHF, and developing holistic evaluation protocols.

  • Low reward variance causes slow reward maximization through a flat RLHF objective landscape, regardless of reward-model accuracy.The limitation follows from the relationship between reward variance and policy-gradient optimization rate.
  • More accurate reward models are not necessarily better teachers, and the best reward model can depend on the language model being aligned.These findings challenge benchmarks that rank models solely by accuracy or independently of the target language model.
  • Improving reward variance: Improving reward variance remains open because naive reward scaling can amplify gradient-estimation noise when expected reward gradients are small.The paper suggests larger reward margins as an alternative direction, while noting that more sophisticated scaling may help with improved gradient estimates.
  • Reward model evaluation: Reward-model evaluation should account for properties beyond accuracy and for the specific language model being aligned.The paper identifies reward variance as arising from the interaction between reward model and language model, while leaving additional factors and robust protocols for future work.
  • Verifiable rewards and general reinforcement learning environments: The theoretical analysis is agnostic to reward origin and can extend to reinforcement-learning environments whose policies produce output distributions via softmax.The paper highlights automatically verifiable tasks such as math and coding as directions for investigating these insights.
  • Alignment methods beyond RLHF: For Best-of-N, perfectly accurate reward models are always optimal, unlike in RLHF where less accurate models can outperform them.This indicates that what makes a good reward model depends on the alignment method.

A Detailed Statements of Theoretical Results

The detailed theoretical results formalize how reward variance controls optimization speed and establish two consequences: accuracy does not determine teacher quality, and reward-model effectiveness depends on the initial policy.

  • Theorem 4 lower-bounds the time needed for expected reward to increase by an additive constant, with the bound growing inversely with initial-policy reward variance.The result applies to general autoregressive policies and remains valid with or without KL regularization.
  • Theorem 5 constructs reward models with prescribed attainable accuracies where the more accurate model produces substantially slower ground-truth reward increase.The optimization-rate gap arises because the more accurate model induces lower reward variance for the initial policy.
  • Theorem 5 therefore shows that more accurate reward models are not necessarily better teachers.The claim holds even when the more accurate model has higher accuracy than its comparison model.
  • The theorem’s comparison predicts slow ground-truth reward increase under rRM and fast increase under r′RM in the corresponding policy-gradient settings.The supplied statements present these as contrasting outcomes of the two reward-model constructions.
  • The detailed statements define attainable accuracy relative to output-pair rankings and specify initial conditions for the reward-increase comparison.The construction allows arbitrary valid distributions over unordered output pairs and attainable accuracy values within the theorem’s scope.

A.3 For Different Initial Policies, Different Reward Models Are Better

Theorem 6 shows that reward-model teaching quality depends on the initial policy: distinct reward models can optimize ground-truth reward faster for different policy families.

  • Theorem 6 establishes that different initial policies can have different reward models that yield faster ground-truth reward increase.The result is stated for gradient flow over tabular policies maximizing the RLHF objective.
  • There exist policy families Π and Π′ such that rRM is a better teacher for Π, whereas r′RM is better for Π′.Thus, no single reward model is uniformly best across the specified initial-policy families.
  • The theorem compares policies initialized at the same expected ground-truth reward while assigning different probabilities to outputs capable of exceeding that baseline.The conditions use outputs yγ and y′γ that each exceed V0 by the desired reward increase γ.
  • The construction explicitly provides reward models rRM and r′RM whose teacher quality changes with the initial-policy family.This formalizes the dependence of optimization outcomes on the interaction between reward model and policy.

B Perfectly Accurate Reward Models Are Optimal for Best-of-N

For Best-of-N sampling, the reward model is used to select the highest-scoring candidate at test time rather than to train the policy by gradient optimization. The paper proves that perfect accuracy is therefore optimal for this alignment method.

  • Best-of-N samples N independent candidate outputs from policy π and returns the candidate with highest reward under rRM.Unlike RLHF, the reward model is applied only at test time.
  • Proposition 1 states that a perfectly accurate reward model is optimal for Best-of-N for any policy and any candidate count N.The proposition assumes perfect pairwise ranking accuracy for the prompt.
  • The role of accuracy differs across alignment methods: perfect reward models can underperform less accurate models in RLHF but are always optimal for Best-of-N.The paper uses this contrast to show that reward-model quality cannot be characterized independently of the alignment procedure.
  • Perfect accuracy preserves the same ordering of candidate outputs under proxy reward rRM and ground-truth reward rG.Consequently, selecting the maximal proxy-reward candidate also selects a ground-truth-maximal candidate among the sampled outputs.

C.4 Proof of Theorem 4

The proof bounds policy movement under low KL-regularized reward variance and uses this to show that optimization can remain slow. It combines gradient and variance bounds with KL-reward boundedness arguments.

  • Policy movement under low variance: Low reward variance makes the RLHF objective gradient vanish or remain small, keeping policy parameters close to their initial values.The proof treats the zero-variance case as a critical point and separately bounds variance growth when it remains positive.
  • Conclusion: The resulting argument links low induced reward variance to slow improvement of expected rewards under gradient flow.The proof establishes this connection for the proxy and ground-truth reward analyses used by the theorem.
  • Policy movement under low variance: The KL-regularized reward is bounded over a sufficiently long interval, despite possible unboundedness if policy probabilities approach zero.The proof tracks the KL regularization term and uses an interval threshold at ±8.
  • Gradient and variance bounds: For tabular policies, the proof separately bounds parameter changes for each prompt and uses the KL-regularized reward vector and its gradient structure.The tabular parameter block associated with one prompt evolves independently of other prompt blocks.

C.6 Improved Lower Bound on Reward Increase Rate for Tabular Policies

This section strengthens the reward-increase lower bound for tabular policies by exploiting independent prompt-specific parameters. It then develops sufficient conditions ensuring faster ground-truth reward improvement.

  • Improved tabular bound: Tabular policies yield a stronger lower bound because parameters governing different prompts are distinct and evolve independently.This removes the averaging over training prompts present for general autoregressive policies.
  • Improved tabular bound: The tabular lower bound depends on reward variance for the individual prompt and has exponent −1/2 rather than Theorem 4’s −1/3.The different exponents arise from the gradient-norm bounds used in the two analyses.
  • Zero-variance case: If KL-regularized reward variance reaches zero, gradient flow remains at a critical point and the expected reward never increases from its initial value.The policy and expected reward remain unchanged for that prompt at all later times.
  • Positive-variance case: When variance remains positive, the proof bounds its growth and translates parameter-distance bounds into a lower bound on the time needed for an additive reward increase.Expected reward is treated as a 1-Lipschitz function of the prompt-specific tabular parameters.
  • Fast improvement conditions: Sufficient conditions for faster ground-truth improvement require outputs with higher proxy and ground-truth rewards, controlled proxy-reward spread, and limited competing outputs.The proof tracks the probability mass assigned to the favorable-output set until it reaches a threshold guaranteeing the desired ground-truth increase.

C.8 Proof of Theorem 5

The proof constructs reward models and initial policies that separate accuracy from optimization speed. It shows that low variance can make ground-truth reward increase slow, while suitable reward separation can support faster increase.

  • Reward-model constructions: The construction assigns bounded reward values so reward-model accuracy can be selected independently of the reward variance induced by the initial policy.The ordering and constants are chosen to achieve a desired attainable accuracy while controlling the favored output’s probability.
  • Policy dependence: The same reward model may induce low variance for one initial policy but high variance for another, because variance depends on the policy’s output probabilities.Low variance for the initial policy does not imply low variance for every later policy.
  • Slow improvement: A reward model that induces low variance for the initial policy yields a lower bound on the time required for ground-truth reward to increase.The slow-increase proof applies the tabular lower bound after bounding the initial variance.
  • Fast improvement: A reward model can instead assign the highest reward to an output with sufficiently high ground-truth reward and satisfy conditions for faster improvement.The favorable output is placed above the initial expected proxy reward, while other outputs receive lower rewards.

D Additional Experiments

Additional experiments extend the main evaluations across alternative initial models, larger models, KL-based plots, and a different policy-gradient method. Supplementary tables report reward variance and accuracy for model-policy combinations.

  • Additional evaluations: Figure 4 plots reward increases against KL divergence from the initial policy rather than training epochs.This supplements the corresponding epoch-based analysis.
  • Additional evaluations: Tables 3–6 and Figures 5–8 repeat the main experiments with a Pythia-1B initial policy instead of Pythia-2.8B.These experiments are described as identical apart from the initial policy model.
  • Additional evaluations: Figure 13 extends the comparison to language models of roughly 3B parameters, with a supplementary table reporting reward variance and accuracy.The reported accuracy measures include both on-policy and off-policy values.
  • Additional evaluations: Figure 14 repeats the Figure 3 experiment using GRPO instead of RLOO, while Table 12 reports reward variance and accuracy for each combination.The supplementary reporting again includes on-policy and off-policy measures.

E Additional Implementation Details

The appendix details the data, training, normalization, evaluation, and policy-gradient procedures used in the experiments, including variants across initial policies and supervised fine-tuning datasets.

  • Data: UltraFeedback samples were filtered to 512-token limits, relabeled with the ground truth reward model, and split into 33135 training and 8284 held-out samples.The resulting training and test sets contained 41419 and 1329 samples, respectively.
  • Training setup: Supervised fine-tuning used either 10000 AlpacaFarm samples or preferred outputs from the UltraFeedback-based reward-model training set for one epoch.Training minimized cross-entropy with Adam at a 1e-6 learning rate and batch size 32.
  • Training setup: Reward models were trained for one epoch with the Bradley-Terry log-likelihood loss and Adam, using a 5e-7 learning rate, batch size 32, and center_rewards_coefficient 0.01.The learning rate was selected to reduce training loss while maintaining stability.
  • Evaluation: Rewards were normalized separately using 500 prompts and 10 outputs per prompt, then reward-model accuracy and variance were evaluated on-policy and off-policy.On-policy evaluation used generated outputs, whereas off-policy accuracy used original UltraFeedback outputs.
  • Policy gradient: Policy-gradient experiments used RLOO or GRPO with Adam, learning rate 1e-7, batch size 32, two minibatches, and KL coefficient 0.05.Runs used two GPUs for roughly 1B-parameter models and four GPUs for roughly 3B-parameter models; some experiments ran for two epochs.
  • Experimental variants: Additional experiments changed the initial policy to Pythia-1B and, in one variant, used reward-model-training data for SFT; KL values averaged the final 20 batches of each epoch.Figures 4, 6, and 8 plot reward increase against KL divergence from the initial policy.
Loading 2503.15477v4…