PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Reward Model a Good Teacher? An Optimization Perspective

Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, Sanjeev Arora

arXiv:2503.15477v4cs.LGcs.AIcs.CLstat.ML

TL;DR

RLHF reward-model quality is usually measured by accuracy, but it is unclear whether accuracy alone identifies effective teachers. This paper combines theoretical analysis with experiments to study reward variance and finds that low variance can flatten optimization, making accuracy insufficient for evaluating reward models.

  • Problem

    It remains unclear how to measure reward-model quality beyond accuracy, despite its importance to RLHF success.

    Source: §1 · p. 1

  • Method

    The paper analyzes the relationship between reward variance and RLHF policy-gradient optimization theoretically and empirically.

    Source: §1 · p. 2, §2.3 · p. 3, §2.1 · p. 3

  • Results

    Low reward variance produces a flat objective and slow reward maximization regardless of accuracy, while effective reward models can differ across language models.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Reward-model evaluation should account for reward variance and its interaction with the language model, not accuracy alone.

    Source: §6 · p. 11, §6.1 · p. 11

    How to modify or train reward models to induce higher reward variance remains an open question.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even a perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another. These results establish a fundamental limitation of evaluating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B parameters corroborate our theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Overall, our findings highlight that beyond accuracy, a reward model needs to induce sufficient variance for efficient optimization.

1 Introduction

This paper reframes reward-model quality as an optimization question: accuracy alone does not ensure effective RLHF, because reward variance governs objective flatness and optimization speed.

1 Introduction

  • RLHF trains a preference-based reward model and then aligns a policy by maximizing that proxy reward with policy-gradient methods [11] [95] [55] [6].The pipeline assumes preference data reflects a ground-truth reward and uses methods including PPO, RLOO, and GRPO.Source: §1 · p. 1
  • Reward-model benchmarks mainly measure pairwise preference accuracy, but recent evidence shows more accurate models do not necessarily produce stronger post-RLHF language models [44] [25] [47] [37] [10].Accuracy estimates whether maximizing the proxy reward will improve the ground-truth reward, yet it may not predict downstream RLHF quality.Source: §1 · p. 1
  • Low reward variance can make the RLHF objective flat regardless of accuracy, causing both proxy and ground-truth rewards to increase extremely slowly under policy gradient.The issue arises when the reward model insufficiently separates outputs probable under the current policy.Source: §1 · p. 2, Figure 1 · p. 2
  • A perfectly accurate reward model can be an arbitrarily worse teacher than a less accurate model when its reward variance is lower.Reward variance is not tied to accuracy, so accuracy does not determine optimization rate.Source: §1 · p. 2
  • Reward-model effectiveness depends on the language model: the same reward model may induce high variance for one policy but low variance and a flat landscape for another.This shows reward models should not be evaluated solely by accuracy or independently of the language model they guide [44] [25] [47].Source: §1 · p. 2
  • Experiments on models up to 8B parameters corroborate the theory, showing that higher reward variance can make a proxy reward outperform an accessible ground-truth reward.The results demonstrate interactions among reward variance, accuracy, and reward-maximization rate.Source: §1 · p. 2

2 Preliminaries

The preliminaries define the RLHF pipeline, accuracy, and reward variance, emphasizing that variance measures reward separation for policy-likely outputs rather than ranking agreement alone.

2.1 Aligning Language Models via Reinforcement Learning

  • RLHF begins from an SFT policy, trains or selects a proxy reward model, and updates the policy with policy-gradient methods under KL regularization [95] [55] [6].The proxy reward is intended to improve ground-truth reward while KL regularization preserves pretrained and SFT capabilities.Source: §2.1 · p. 3

2.2 Accuracy

  • Accuracy measures agreement with ground-truth rankings, but benchmark estimates may mislead when policy-training prompts or on-policy outputs differ from benchmark distributions.The analysis therefore evaluates accuracy over the training prompt set S while allowing any output-pair distribution.Source: §2.2 · p. 3

2.3 Reward Variance

  • Reward variance measures how strongly a reward model separates outputs likely under a given policy and prompt, unlike accuracy, which ignores separation magnitude.Reward variance is central because the theory connects it to the RLHF objective gradient.Source: §2.3 · pp. 3–4
  • Reward variance must be normalized across models because multiplying rewards by c scales variance by c^2, otherwise scale could masquerade as separation.Rewards are assumed bounded in [−1, 1], and normalization is used in experiments to compare meaningful separation.Source: §2.3 · pp. 3–4

3 Theory: Optimization Perspective on What Makes a Good Reward Model

The theory identifies reward variance—not accuracy alone—as central to efficient RLHF optimization. Low variance produces flat landscapes and slow reward improvement, while teacher quality depends on the aligned model’s initial policy.

3.1 Technical Setting

  • The analysis uses tabular policies for separation results and exact-gradient gradient flow, while empirical results indicate the implications extend to sample-based policy-gradient methods.The authors identify these simplifications as limitations because language-model output spaces prohibit exact gradient computation.Source: §3.1 · p. 5

3.2 Low Reward Variance Implies Slow Reward Maximization

  • Low reward variance induces a flat objective landscape, slowing increases in both proxy and ground-truth rewards during policy-gradient optimization.The lower bound on optimization time scales inversely with the average reward variance induced by the reward model for the initial policy.Source: §3 · p. 4, §3.2 · p. 5

3.3 More Accurate Reward Models Are Not Necessarily Better Teachers

  • A perfectly accurate reward model can require arbitrarily long ground-truth reward maximization, whereas an almost completely inaccurate model can achieve tγ = O(πθ(0)(yγ|x)^−1).Accuracy measures ranking, while reward variance measures reward separation; the theorem therefore shows that accuracy alone is insufficient for RLHF teacher evaluation.Source: §3.3 · pp. 5–6

3.4 For Different Initial Policies, Different Reward Models Are Better

  • Different initial policy families favor different reward models: each model achieves tγ = O(1) for one family, while optimization can be arbitrarily slow for the other.The construction holds the initial policies’ expected ground-truth rewards equal, isolating the effect of policy-dependent reward variance.Source: §3.4 · pp. 6–7

4 Experiments

Experiments show that reward variance is strongly associated with faster reward maximization, while accuracy alone does not determine teacher effectiveness. The best reward model also depends on the language model being optimized.

4.1.1 Setting

  • The experiments vary reward-model accuracy and variance by training models on mixtures containing 100%, 75%, 50%, 25%, or 0% on-policy preference pairs.Models are normalized to the same reward scale, and a perfectly accurate low-variance model is created by weakening score separation while preserving rankings.Source: §4.1.1 · pp. 7–8

4.1.2 Results

  • Reward variance strongly correlates with proxy and ground truth reward increases, whereas accuracy alone is not necessarily predictive.For ground truth reward, combining reward variance with accuracy is most indicative because proxy-reward improvements translate less reliably when the model is inaccurate.Source: §4.1.2 · p. 9, Table 2 · p. 9
  • A perfectly accurate but low-variance reward model increases ground truth reward more slowly than less accurate models, and proxy rewards can initially outperform direct ground-truth optimization.With sufficiently long training, proxy-reward optimization can cause earlier saturation and declining ground truth reward through reward hacking.Source: Figure 2 · p. 8, §4.1.2 · p. 9

4.2.1 Setting

  • The cross-model study uses Pythia and Llama-3.2 policies with four publicly available reward models spanning RewardBench scores from 90.9 to 65.4.Policies are trained with RLOO on UltraFeedback, including SFT and instruction-tuned variants, with rewards normalized for fair comparison.Source: §4.2.1 · p. 9

4.2.2 Results

  • Different initial policies achieve their highest ground truth reward increases with different reward models, confirming that teacher quality depends on the language model.Low reward variance and low accuracy can each hurt RLHF: RM-Gemma-2B produces the largest proxy-reward increase through high variance, but its inaccuracy prevents a higher ground truth reward.Source: §4.2.2 · p. 10, Figure 3 · p. 10

5 Related Work

Prior work has analyzed RLHF guarantees, reward transformations, and policy-gradient optimization, but this work focuses on how reward-model properties affect standard RLHF policy-gradient optimization. It also distinguishes reward-model teaching effectiveness from accuracy alone and identifies practical limitations of existing optimization remedies.

Theoretical analyses of RLHF

  • This work is presented as the first to characterize how reward-model properties affect policy-gradient optimization in standard RLHF, beyond prior analyses of specialized algorithms and reward-estimation sample complexity.Existing analyses provided optimal-policy guarantees under varying assumptions, while other work bounded ground-truth reward estimation from preference data.Source: §5 · p. 10

Transforming and ensembling reward models

  • Reward transformations and ensembles were studied for policy preservation, convergence, or reward hacking, whereas this work shows that ensembling can additionally reduce flat landscapes caused by low reward variance.Prior work proposed reward-model transformations [7] [79] and ensembles [13] [62] [23] [91], while reward shaping analyzed transformations preserving optimal policies and value-based convergence [19, 34].Source: §5 · p. 10

Optimization difficulties in policy gradient

  • Policy-gradient objectives are non-concave, and low reward variance causes vanishing gradients; this provides the optimization basis for analyzing effective reward models in RLHF.Although vanishing gradients near deterministic policies are less relevant because language models rarely produce such distributions, low reward variance remains directly relevant [64].Source: §5 · pp. 10–11
  • Natural policy gradient and entropy regularization can theoretically address poor policy-gradient landscapes, but are impractical for large language models.Natural policy gradient requires inverting a P × P matrix for P language-model parameters, while entropy regularization often gives little improvement for language generation because of the large output space [64].Source: §5 · p. 11

Stronger teachers do not always provide better supervision

  • As in knowledge distillation, stronger teachers need not provide better supervision, but RLHF differs because high reward-model accuracy does not imply poor teaching quality.In RLHF, factors beyond accuracy can determine effectiveness, unlike the setting of [58], where strong models inherently provide weak supervision.Source: §5 · p. 11

6 Conclusion

The paper argues that reward-model quality depends not only on accuracy but also on inducing sufficient reward variance for efficient RLHF optimization. It further calls for language-model-specific evaluation and future work beyond RLHF, including improved variance methods, verifiable rewards, and alternative alignment procedures.

6 Conclusion

  • Low reward variance creates a flat RLHF objective and slow reward maximization, so effective reward models must combine accuracy with sufficient variance.This limitation holds regardless of reward-model accuracy.Source: §6 · p. 11
  • More accurate reward models are not necessarily better teachers, and a model effective for one language model can perform poorly for another.These findings challenge accuracy-focused benchmarks and universal rankings independent of the aligned language model [44] [25] [47].Source: §6 · p. 11

6.1 Limitations and Future Work

  • Increasing reward variance remains open: naive scaling by c > 1 amplifies variance by c^2 but can amplify gradient noise when expected gradients vanish.Potential alternatives include improved gradient estimates and training reward models for larger reward margins [64] [60] [51].Source: §6.1 · p. 11
  • Reward-model evaluation should measure properties beyond accuracy and account for the specific language model, especially their interaction through reward variance.Characterizing additional factors and robust evaluation protocols remains an important direction [10].Source: §6.1 · p. 11
  • The analysis extends beyond RLHF to reinforcement-learning environments with softmax policies, including tasks with automatically verifiable outputs such as math and coding [43] [33].Future work could apply these insights to data selection and verifiable-reward design.Source: §6.1 · pp. 11–12
  • Reward-model effectiveness depends on the alignment method: perfectly accurate models can underperform less accurate ones in RLHF but are always optimal for Best-of-N [52].Future research should identify which reward-model properties matter for Best-of-N, contrastive objectives, and rejection sampling [61] [27] [18] [65] [17] [32].Source: §6.1 · p. 12

A Detailed Statements of Theoretical Results · A.1 Low Reward Variance Implies Slow Reward Maximization · A.2 More Accurate Reward Models Are Not Necessarily Better Teachers

The appendix provides detailed theoretical results showing that low initial reward variance slows RLHF reward maximization, while higher reward-model accuracy alone does not guarantee better teaching. It also formalizes attainable-accuracy conditions and generalizes the comparison across reward models.

A Detailed Statements of Theoretical Results

  • The appendix gives full statements for the theorems underlying Sections 3.2–3.4, using the technical setting introduced in Section 3.1.Appendices A.1–A.3 contain the detailed theorem statements.Source: A Detailed Statements of Theoretical… · p. 18

A.1 Low Reward Variance Implies Slow Reward Maximization

  • Theorem 4 lower-bounds the time needed for expected reward to increase by a fixed additive amount, with the bound worsening as initial reward variance decreases.The result applies to any reward function, including ground-truth or model reward, and holds with or without KL regularization.Source: §A.1 · p. 18
  • The low-variance slowdown can theoretically be offset by growing policy Jacobian norms, but experiments indicate Jacobian changes do not compensate in practical settings.The caveat applies especially to complex neural parameterizations, unlike tabular policies whose Jacobians remain constant.Source: §A.1 · p. 18

A.2 More Accurate Reward Models Are Not Necessarily Better Teachers

  • Theorem 5 generalizes the comparison between reward models by showing that prescribed attainable accuracies can produce substantially different ground-truth reward-increase rates.The optimization-rate gap arises because one reward model induces lower initial reward variance.Source: §A.2 · p. 18
  • Even when α > α′, the more accurate reward model can be the slower teacher because it induces lower reward variance for the initial policy.Thus, accuracy alone does not determine teaching quality.Source: §A.2 · p. 18
  • The theorem restricts accuracy values to attainable accuracies, because tied ground-truth rewards can make some intermediate pairwise-accuracy values impossible.Attainability is defined both generally and subject to a specified output being ranked first.Source: §A.2 · p. 18
  • Under tabular-policy gradient flow, Theorem 5 constructs reward models with specified attainable accuracies that yield slow or fast ground-truth reward increase for a desired target improvement.The construction compares a model rRM producing slow increase with r′RM producing fast increase.Source: §A.2 · p. 19
  • Theorem 2 is a special case using uniform output-pair sampling, perfect accuracy for one model, and a constrained comparison model whose accuracy is at most 2/|Y|.The bound follows from counting unordered output pairs involving the target output yγ.Source: §A.2 · p. 19

A.3 For Different Initial Policies, Different Reward Models Are Better

Theorem 6 shows that reward-model quality as a teacher depends on the initial policy: different policy families can favor different reward models for faster ground-truth reward improvement.

A.3 For Different Initial Policies, Different Reward Models Are Better

  • Theorem 6 establishes that different initial policy families can have different reward models that produce faster increases in ground-truth reward.The result applies under gradient flow maximizing the RLHF objective with tabular policies over a prompt set S.Source: §A.3 · p. 19
  • For initial policies in Π, rRM is a better teacher, whereas for policies in Π′, r′RM is a better teacher.The policy families differ in which high-ground-truth-reward output initially has probability at least c and which has probability at most T^-2.Source: §A.3 · pp. 19–20

B Perfectly Accurate Reward Models Are Optimal for Best-of-N … C.3 Low Reward Variance Implies Vanishing Gradient

The paper contrasts alignment methods: perfect reward-model accuracy is optimal for Best-of-N, whereas low KL-regularized reward variance yields vanishing RLHF gradients. The appendix supplies notation, general lemmas, and proofs supporting these results.

B Perfectly Accurate Reward Models Are Optimal for Best-of-N

  • Perfectly accurate reward models are always optimal for Best-of-N, unlike in RLHF where they can underperform less accurate models.Best-of-N samples independent candidates and returns the one with highest reward-model score; perfect accuracy preserves the ground-truth ordering.Source: B Perfectly Accurate Reward Models Are… · p. 20

C Deferred Proofs

  • Appendix C organizes proofs of the main theoretical results, including gradient bounds, reward-increase bounds, and conditions for fast ground-truth reward improvement.The proofs cover general autoregressive and tabular policies across Appendices C.3–C.9.Source: C Deferred Proofs · p. 21

C.1 Notation

  • The notation defines the KL-regularized reward as RM(x, y; θ) := rRM(x, y) − λ · ln πθ(y|x) πθref(y|x).It also specifies Euclidean, ℓ1, and spectral norms used in subsequent proofs.Source: §C.1 · p. 21

C.2 Useful Lemmas: General

  • The general lemmas provide differential-inequality bounds and KL-divergence bounds used throughout the deferred proofs.They include upper and lower growth estimates for continuously differentiable functions and an upper bound on KL(p||q).Source: §C.2 · pp. 21–22

C.3 Low Reward Variance Implies Vanishing Gradient

  • For general autoregressive policies, the KL-regularized RLHF gradient-norm upper bound depends on the variance of the KL-regularized reward.This extends Theorem 1 in [64], which treated only a non-KL-regularized objective.Source: §C.3 · p. 22
  • As KL-regularized reward deviations become small, the proof bounds the gradient using atypical-output probability and reward-deviation terms, yielding a vanishing-gradient conclusion.The argument partitions outputs by deviation from expected reward, applies Chebyshev’s inequality, and combines the resulting bounds.Source: §C.3 · pp. 23–24

C.4 Proof of Theorem 4 · C.5 Useful Lemmas: Tabular Policies

The proof of Theorem 4 combines bounds on policy movement and KL-regularized rewards to show how low reward variance delays reward improvement. The tabular-policy lemmas provide gradient, probability-evolution, subset-mass, and KL bounds used in later theorem proofs.

C.4 Proof of Theorem 4

  • Low induced reward variance makes the RLHF gradient vanish or remain small, keeping policy parameters near initialization and delaying any increase in expected reward.Lemma 5 links low initial variance and bounded KL-regularized rewards to a lower bound on the time tγ needed to improve expected reward by γ.Source: §C.4 · p. 25
  • The proof controls variance growth by differentiating VarKL(θ(t)) and bounding its rate, which yields an upper bound on policy displacement over time.The argument treats the zero-variance critical-point case separately and otherwise uses calculus and norm inequalities to control the trajectory.Source: §C.4 · pp. 25–26
  • Lemma 6 supplies the bounded-reward condition required by Lemma 5 by tracking the KL-regularization term until it could reach the boundary ±8.Although KL-regularized rewards can become unbounded when policy probabilities approach zero, the proof shows they remain bounded for a sufficiently long interval.Source: §C.4 · pp. 27–28
  • Combining the two lemmas converts trajectory bounds into a lower bound on tγ, establishing slow reward improvement when initial KL-regularized reward variance is low.The final step bounds expected-reward change through policy displacement and uses the reward range [−1, 1].Source: §C.4 · p. 29

C.5 Useful Lemmas: Tabular Policies

  • For tabular policies, the RLHF objective gradient for one prompt depends only on that prompt’s parameter block, while gradients for distinct prompts are zero.This locality follows from the tabular softmax parameterization and the log-derivative trick.Source: §C.5 · p. 30
  • The tabular lemmas characterize gradient-flow changes in individual output probabilities and bound the evolution of probability mass assigned to any output subset.The subset-mass bound combines bounded rewards, KL terms, continuity, and Grönwall’s inequality.Source: §C.5 · pp. 30–32
  • Additional tabular bounds control KL divergence using the least-likely reference-policy output, providing finite estimates for KL-related terms in later analyses.The proof uses 0 ≤ KL(θ) ≤ −ln πθref(ym|x), where ym minimizes the reference-policy probability.Source: §C.5 · p. 33

C.6 Improved Lower Bound on Reward Increase Rate for Tabular Policies · C.7 Sufficient Conditions for Fast Ground Truth Reward Increase

The appendix strengthens the reward-increase lower bound for tabular policies and identifies sufficient conditions under which ground-truth reward increases quickly. These results rely on prompt-wise parameter independence and favorable reward-model, initialization, and regularization conditions.

C.6 Improved Lower Bound on Reward Increase Rate for Tabular Policies

  • Proposition 3 gives a stronger lower bound on the time for any reward to increase by an additive constant under tabular-policy gradient flow.The bound applies to any prompt x ∈ S, any reward function, and any initial policy.Source: §C.6 · pp. 33–34
  • Unlike Theorem 4, Proposition 3 depends on variance for the individual prompt rather than average training-set variance because tabular prompt parameters evolve independently.The reward-variance exponent also improves from −1/3 to −1/2, though this improvement is not essential for Theorems 5 and 6.Source: §C.6 · p. 34
  • If the prompt-specific KL variance reaches zero, gradient flow remains at a critical point and the prompt’s expected reward never increases from its initial value.This follows from uniqueness of the gradient-flow solution and the existence of a stationary solution for that prompt’s parameters.Source: §C.6 · p. 34
  • For tabular policies, the expected reward for a prompt is 1-Lipschitz in its prompt-specific parameters, enabling parameter-deviation bounds to yield the lower bound on reward-increase time.The argument uses the fact that the reward variance is at most 1 for rewards bounded in [−1, 1].Source: §C.6 · p. 36

C.7 Sufficient Conditions for Fast Ground Truth Reward Increase

  • Proposition 4 establishes sufficient conditions on the reward model, initial policy, and KL coefficient that ensure a desired ground-truth reward increase during tabular-policy gradient flow.For a prompt x, outputs in Y+ must exceed the initial ground-truth and reward-model expectations, while the reward-model scores on Y+ must not be too dispersed.Source: §C.7 · p. 37
  • The proof reduces fast ground-truth improvement to increasing the policy mass on Y+ until it reaches threshold ρ, which then guarantees an improvement of at least γ.The threshold is defined so that πθ(t)(Y+|x) ≥ ρ implies VG(θ(t); x) ≥ VG(θ(0); x) + γ.Source: §C.7 · pp. 37–38
  • Under the assumed bounds on KL regularization and initial bad-output probability, the probability of Y+ increases monotonically and reaches ρ by the derived time bound.This contradicts any assumption that tγ exceeds the bound, proving that the ground-truth reward increase occurs by time T.Source: §C.7 · pp. 39, 41

C.8 Proof of Theorem 5 · C.9 Proof of Theorem 6

Theorem 5 constructs reward models with the same attainable accuracy but contrasting optimization speeds, while Theorem 6 extends the contrast across policy classes through reward-variance conditions. The proofs use low-variance bounds for slow improvement and Proposition 4 for fast improvement.

C.8 Proof of Theorem 5

  • Theorem 5 constructs reward models with prescribed attainable accuracies, including a model ranking yγ first while assigning adjustable rewards to other outputs.The construction chooses {by} to achieve accuracy α and {cy} to achieve accuracy α′ subject to yγ being ranked first.Source: §C.8 · p. 41
  • Under a tabular policy, reward values for prompts other than x can be arbitrary because πθ(t)(·|x) does not depend on their parameters.The proof defines expected rewards VRM and V′RM for the prompt x.Source: §C.8 · p. 41, §C.9 · p. 42
  • Theorem 5’s construction can instead make variance low only for the initial policy, high for other policies, and rewards can be shifted by any constant.Thus, low variance need not hold uniformly across policies, and near-zero reward levels are not essential.Source: §C.8 · p. 41
  • Theorem 5’s low-variance reward model yields slow ground-truth reward increase under gradient flow, whereas the contrasting model yields fast increase via Proposition 4.The slow bound follows from low initial-policy reward variance; the fast proof uses yγ as the sole favorable output and verifies Proposition 4’s conditions.Source: §C.8 · pp. 41–42

C.9 Proof of Theorem 6

  • For πθ(0) ∈ Π, Theorem 6 proves fast ground-truth reward increase under rRM by assigning reward 1 to yγ and −1 to every other output.The proof verifies Proposition 4 using the favorable set {yγ}, the bound δ ≥ 2(1 − ρ)^2, and an empty Ybad set.Source: §C.9 · p. 43
  • For πθ(0) ∈ Π, r′RM induces low reward variance and therefore slow ground-truth reward increase through Proposition 3.The variance bound uses πθ(0)(y′γ|x) ≤ T^−2, bounded reward differences, and subsequent simplification with T ≥ 1.Source: §C.9 · pp. 43–44
  • For πθ(0) ∈ Π′, the proofs establish the corresponding fast result for r′RM and slow result for rRM by steps analogous to the earlier cases.The argument explicitly transfers the fast and slow proof patterns to the alternate policy class Π′.Source: §C.9 · p. 44

D Additional Experiments · D.1 More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1) · D.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)

Additional experiments test whether the reward-variance relationship persists across policy initialization, supervised fine-tuning data, ground-truth reward models, normalization schemes, model scale, and policy-gradient methods. They also provide expanded variance and accuracy measurements for reward-model–policy combinations.

D.1 More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1)

  • Across alternative reward normalization schemes, the relationship between reward variance and optimization persists when reward scale is accounted for.The experiments normalize by the square root of average initial reward variance rather than jointly computed reward standard deviation.Source: §D.1 · p. 44
  • The D.1 experiments replicate the core analyses with a Pythia-1B initial policy, including variants using UltraFeedback-based rather than AlpacaFarm supervised fine-tuning data.The same experiment families are also repeated with GRM-gemma2-2B-rewardmodel-ft replacing ArmoRM as the ground-truth reward model.Source: §D.1 · p. 44
  • Figure 4 supplements the main reward-increase analysis by plotting reward gains against KL divergence from the initial policy instead of training epochs.This provides an alternative view of optimization progress for the D.1 experiments.Source: §D.1 · p. 44

D.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)

  • For each reward-model and initial-policy combination in the D.2 experiment, Table 10 reports reward variance and both on-policy and off-policy accuracy.These measurements directly expand the comparison underlying Figure 3.Source: §D.2 · p. 44
  • The D.2 comparison is extended to language models of roughly 3B parameters, with Table 11 reporting variance and on-policy and off-policy accuracy for each combination.Figure 13 supplies the larger-model experiment, while Table 11 provides its accompanying measurements.Source: §D.2 · p. 45
  • The D.2 experiment is also reproduced with GRPO [69] instead of RLOO, with Table 12 reporting reward variance and both accuracy types for every reward-model–policy pairing.Figure 14 presents the GRPO-based results and Table 12 supplies the corresponding measurements.Source: §D.2 · p. 45

E Additional Implementation Details · E.1 More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1) · E.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)

The appendix specifies the data, training, normalization, evaluation, and optimization procedures used to reproduce Sections 4.1–4.2. Across language models and experimental variants, reward variance is consistently important for reward maximization, while accuracy alone does not reliably identify the best teacher.

E.1 More Accurate Reward Models Are Not Necessarily Better Teachers (Section 4.1)

  • The experiments use binarized UltraFeedback [75], filtered at 512 tokens, yielding 41419 training and 1329 test samples, with 33135 samples for reward-model training and 8284 for policy-gradient training.Because training and test results were nearly identical and the focus was optimization, reported quantities were computed over training prompts.Source: §E.1 · p. 45
  • Reward models were trained with Bradley-Terry loss, rewards were normalized using statistics from 500 prompts and 5000 generated outputs, and accuracy and variance were evaluated on the same prompt set.The perfectly accurate low-variance model scaled ground-truth rewards by 0.001 for 50% of policy-gradient prompts; changing these choices did not significantly alter Section 4.1 trends.Source: §E.1 · pp. 45–46

E Additional Implementation Details

  • Policy-gradient experiments used RLOO and GRPO with Adam, learning rate 1e-7, batch size 32, two minibatches, KL coefficient 0.05, and two sampled outputs per prompt; excessive learning rates caused instability, whereas overly small rates or high KL penalties slowed progress.E.2 reused the E.1 setup but used publicly available reward models and ran policy gradient for two epochs.Source: §E.1 · p. 46, §E.2 · p. 46

E.2 For Different Language Models, Different Reward Models Are Better (Section 4.2)

  • Accurate reward models are typically more KL-efficient: at a given ground-truth reward, a perfectly accurate reward model requires significantly less KL divergence than a less accurate one.The comparison plots reward increase against KL divergence from the initial policy rather than training epochs.Source: Figure 4 · p. 47, Figure 6 · p. 47, Figure 8 · p. 48
  • When the initial policy is SFTed on the reward-model training set, reward variance remains positively correlated with reward increase but becomes less indicative because reward models induce similar variances.The reduced diversity arises because SFT makes initial on-policy and off-policy output distributions more similar; combining reward variance with accuracy remains informative.Source: Table 5 · p. 48, Figure 7 · p. 48, Table 6 · p. 49
  • With GRM-gemma2-2B-rewardmodel-ft [87] as ground truth, on-policy accuracy has the highest Spearman correlation with reward increase, while combining reward variance with on-policy accuracy reaches 0.809 Pearson and 0.828 Spearman.These correlations are reported for the Pythia-1B experiments; the combined measure excludes off-policy accuracy.Source: Table 7 · p. 49, Figure 9 · p. 49, Table 8 · p. 50
  • For different language models, different reward models are better, and reward variance strongly correlates with reward increase whereas accuracy alone may not.This pattern is reported for Pythia-1B settings, alternative SFT data, a different ground-truth reward model, altered normalization, larger language models, and GRPO experiments.Source: Table 4 · p. 48, Figure 13 · p. 51, Figure 14 · p. 51

Related papers

Loading 2503.15477v4…