PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Generative Reward Models

Angel Raychev, Nathan Lile, Rafael Rafailov

arXiv:2610.30765v1cs.AI

TL;DR

This paper asks whether stronger static generative reward-model judges provide better online training signals as policy distributions change. It compares judge-training pipelines and capacity under online DPO, finding that the static accuracy advantage often reverses in policy outcomes under a learned Gold reference. The evidence is bounded by that reference and by an inconclusive limited follow-up using another learned judge.

  • Problem

    Static reward-model accuracy can be a poor predictor of downstream optimization, leaving open how judge-training recipe and judge and policy capacity affect online preference optimization as policy distributions change.

    Source: §1 · p. 1

  • Method

    The study sweeps Qwen3 GenRM judge format and size, policy size, training duration, and checkpoint selection in online DPO, comparing trajectories under proxy and Gold evaluators.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Answer-Only rewarders yield higher peak policy Elo in 7 of 8 matched configurations in the shared global Gold arena.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    In these experiments, static in-distribution accuracy alone does not predict training effectiveness, and intermediate GenRM checkpoints can outperform fully trained models as rewarders.

    Source: §7 · p. 16

    The downstream ranking is relative to a learned Gold reference, and a limited audit using another learned judge is inconclusive overall.

    Source: §2 · p. 2, §5 · p. 14

Abstract

from arXiv · show

We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.

1 Introduction

Generative reward models can provide flexible preference feedback, but stronger static evaluation does not necessarily translate into better online training signals. This study compares judge-training recipes and judge and policy capacity, finding that the static ranking often reverses during optimization.

1 Introduction

  • Static reward-model accuracy can be a poor predictor of downstream optimization, motivating tests of GenRMs as rewarders while policy distributions change.The study uses creative writing, where correctness cannot be programmatically checked and reward hacking and proxy–Gold divergence remain salient.Source: §1 · p. 1
  • The study sweeps judge format, judge size, policy size, training duration, and checkpoint selection under online DPO.It traces policy trajectories under proxy and Gold evaluators using within-arena comparisons.Source: §1 · p. 2
  • Thinking GenRMs have higher static validation accuracy, yet Answer-Only rewarders usually yield higher peak Gold Elo, with some competitive Thinking exceptions.Additional judge capacity remains valuable even above policy size.Source: §1 · p. 2
  • Table 1 distinguishes GenRMs as evaluators on a fixed distribution from GenRMs as rewarders inside an online optimization loop.This is the paper’s central empirical distinction.Source: Table 1 · p. 2

2 Methodology

The methodology compares generative reward models trained and evaluated in distinct roles, then uses their preferences to optimize policies online. Its results are interpreted relative to a learned Gold reference, not as direct human judgments.

Gold evaluator

  • The frozen Gold evaluator is trained on vote-derived implicit preferences, so agreement with its labels does not establish reliable preferences on later policy-generated responses.The study therefore measures proxy behavior relative to a learned reference, not an independently established measure of human writing quality.Source: §2 · p. 2

Training objectives

  • Answer-Only judges use SFT to supervise verdicts, while Thinking judges use GRPO with normalized outcome rewards; this compares practical pipelines, not reasoning alone.The 4B Thinking ablation supports the chosen GRPO recipe but is not an exhaustive comparison of training methods.Source: §2 · p. 3

Effect of normalization

  • In the retained binary-reward GRPO setup, within-prompt standardization cancels the positional-consistency bonus apart from the numerical stabilizer.The bonus changes raw reward statistics but supplies no independent positional-consistency training signal in this configuration.Source: §2 · p. 3

Policy optimization

  • Online DPO repeatedly samples response pairs from the current policy, so its preference-supervision distribution evolves as the policy changes.This lets the study examine GenRM behavior under policy-induced distribution shift.Source: §2 · p. 3

Evaluation and interpretation

  • Static agreement on the fixed validation set measures neither proxy–Gold disagreement’s effects on gradients nor which sampled pairs reach updates after verdict filtering.Aggregate Elo comparisons do not reconstruct trustworthy response-level online agreement or a measured gradient-level explanation.Source: §2 · pp. 3–4

3 Experimental Setup

The setup separates vote-derived source supervision, Gold-derived proxy labels, and fresh policy-generated pairs used for online DPO. Qwen3 judges and policies are evaluated with separate proxy and Gold arenas, while the retained artifacts do not recover every historical training configuration.

Dataset construction

  • Source labels come from higher-voted Reddit story pairs, proxy labels come from Gold relabeling, and fresh proxy-labeled policy responses drive online DPO.Policy validation reuses the first 1,000 LitBench pair prompts and their higher-voted stories as fixed response references.Source: §3 · pp. 4–5

Model families

  • All GenRMs and policies use Qwen3, enabling matched mode and size comparisons within one model family without isolating scale from every model-specific difference.The same family also supports study of judge–policy capacity interaction.Source: §3 · p. 5

Proxy GenRM training

  • Proxy judges span 0.6B–14B: Answer-Only models use SFT, while Thinking models use GRPO with a separate actor KL loss.The trained proxies are evaluated against the frozen Gold reference.Source: §3 · p. 5

Policy training

  • Policies span 0.6B–8B and use Online DPO with two current-policy responses per prompt, judged in both orderings; zero-sum pairs are discarded.The retained launcher and unrecovered per-run configurations mean nominal token caps are not verified as run-wide settings.Source: §3 · p. 5

Evaluation and artifact reconstruction

  • Proxy and Gold Elo ratings use separate fitted evaluator arenas, with principal exports covering about 460 steps and usually 47 snapshots including step zero.Later continuations are not spliced into these curves; retained artifacts also lack every original preparation invocation and resolved run configuration.Source: §3 · p. 5

Experimental design

  • Table 2 covers matched and fixed-axis online-DPO sweeps rather than a full factorial grid.Its design separates fixed-distribution accuracy from in-loop rewarder behavior.Source: Table 2 · p. 6

4 Experiments and Results

Across these experiments, static Gold-label accuracy and online rewarder performance do not rank judge pipelines consistently: Thinking judges score higher statically, while Answer-Only rewarders usually produce higher policy Elo. Judge capacity, policy capacity, checkpoint choice, and DPO regularization all shape outcomes under the learned Gold evaluator.

4.1 GenRM Training

  • Thinking GenRMs have higher static Gold-label accuracy at every studied size, while trained Answer-Only SFT and Thinking GRPO models substantially exceed off-the-shelf baselines.In the 4B Thinking ablation, the retained GRPO recipe reaches 83.2%, versus 64.7% for thinking-trace distillation, 60.7% for STaR, and 60.0% off-the-shelf.Source: §4.1 · pp. 6–7, Figure 1 · p. 6

4.2 Trained GenRMs vs. Baseline Models in Policy Training

  • Trained 4B rewarders yield higher peak Gold ratings than off-the-shelf rewarders in both modes, but the Thinking comparison reverses at matched step 460.At step 460, the off-the-shelf Thinking rewarder’s policy has 542 Gold Elo versus 501 for the trained rewarder; Answer-Only peaks are 961 versus 476 for trained and off-the-shelf rewarders.Source: §4.2 · pp. 7, 9, Figure 2 · p. 7

4.3 Answer-Only vs Thinking

  • Answer-Only rewarders achieve higher peak Gold Elo in 7 of 8 matched configurations, with the same 7-of-8 pattern for top-five averages.The exception is the 0.6B policy with a 4B judge, where Thinking peaks at 1048 versus 1013 for Answer-Only; mean gaps are 126 Elo at peak and 107 for top-five averages.Source: §4.3 · p. 9, Table 3 · p. 10

4.4 Effect of GenRM Size

  • With a 4B policy, scaling rewarders from 0.6B to 14B improves Gold Elo especially for Answer-Only, while Thinking scaling is less monotone.The 8B Thinking judge slightly exceeds the 14B judge in peak Gold Elo (1507 vs. 1451) and top-5 average Gold Elo (1488 vs. 1437); the 14B Answer-Only judge is strongest in the global arena.Source: §4.4 · p. 11, Figure 5 · p. 11, Table 4 · p. 12

4.5 Effect of Policy Size

  • Larger policies achieve higher Gold Elo under Answer-Only supervision, while larger Thinking-supervised policies peak earlier in the recorded trajectories.Under Thinking supervision, the 8B policy peaks around step 130, the 4B around step 320, and the 0.6B and 1.7B policies near the end of the recorded window.Source: §4.5 · p. 12, Figure 6 · p. 12

4.6 Effect of the β Coefficient in Online DPO

  • In the tested Online-DPO range, β = 0.002 reaches the highest Gold Elo; the exported-KL subset’s peak rises from 357 at β = 0.1 to 1122 at β = 0.002.The retained KL coordinates are smoothed export coordinates rather than raw per-snapshot measurements, and the sweep establishes neither a precise KL-constrained frontier nor an optimum outside the tested range.Source: §4.6 · p. 13, Figure 7 · p. 13, Figure 8 · p. 14

5 Limitations and Discussion

The reported rankings are conditional on a learned evaluator, a creative-writing domain, and stored Elo comparisons rather than independent human judgments or broad model-family tests. Several analyses are exploratory or hypothesis-generating, with limits on what they establish.

Reference supervision and validation reuse

  • Gold is a learned 32B evaluator trained on Reddit vote-derived preferences, so downstream results measure optimization toward that reference rather than direct human adjudication.Gold, proxy, and policy validation reuse the same 1,000 source pair records, so the measurements are not independent source-set replications.Source: §5 · p. 14

Evaluator dependence and response quality

  • The alternate-judge audit was inconclusive: Answer-Only won 42 of 80 comparisons (52.5%; 95% Wilson interval 41.7–63.1%), and it does not establish the Gold-ranked advantage.The audit used another Qwen3 Answer-Only judge, Gold-selected checkpoints, and a different rubric; it was not human evaluation or cross-family replication.Source: §5 · p. 14

Distribution-dependent influence of judge fine-tuning

  • The proposed shifted-output preference-sharing explanation remains a hypothesis; retained Elo trajectories do not test whether shared model priors explain the measured advantage.Even agreement between Gold and its unfine-tuned parent would not by itself establish human quality or its decline.Source: §5 · pp. 14–15

Domain scope

  • The trained-rewarder sweeps concern creative writing and do not systematically test safety, helpfulness, math, or code; the ancillary OpenOrca experiment does not replicate the central result.The ancillary experiment used off-the-shelf rewarders and a different reference evaluator.Source: §5 · p. 15

Relation to prior creative-writing studies

  • The downstream reversal is specific to the complete training and measurement loop, not a general claim that thinking is unsuitable for creative-writing evaluation.Prior work reports chain-of-thought verification degradation alongside stronger trained Thinking judges on its benchmark [Fein et al., 2025].Source: §5 · p. 15

Family and algorithm scope

  • All models use Qwen3 backbones, and the study does not compare PPO-style optimization, alternative Thinking-judge recipes, or Best-of-N selection.The findings concern online-DPO training signals, not a compute-matched ranking of every way to use a judge.Source: §5 · p. 15

Elo anchoring and schedule

  • Elo differences depend on the evaluator, contestants, anchor, and schedule, while bootstrap intervals cover stored comparisons rather than prompt, generation, retraining, or tournament uncertainty.Rows sharing prompts or responses may be dependent, so the intervals are not universal confidence statements.Source: §5 · p. 15

Post-hoc fits and KL provenance

  • The size fits describe the stored grid rather than seed-averaged laws or out-of-sample predictions, and historical KL exports are not used to establish a constrained frontier.KL preprocessing is incomplete for some panels, and the original logger-to-step mapping is not independently validated.Source: §5 · p. 15

6 Related Work

Prior work studies reward overoptimization, GenRM designs, and the distinction between static accuracy and downstream policy performance. This study focuses on online-DPO outcomes for practical Answer-Only SFT and Thinking GRPO pipelines, without isolating reasoning format as a causal factor.

Prior evaluations

  • Reward overoptimization has been studied in RLHF and direct methods, while prior evaluations distinguish static reward-model accuracy from downstream policy performance [Ouyang et al., 2022, Gao et al., 2022, Rafailov et al., 2023, 2024, Wen et al., 2025, Frick et al., 2024, Kim et al., 2025].The generic distinction between static accuracy and downstream performance is not new to this work.Source: §6 · p. 16

Generative reward models

Study scope

  • This study instead measures generative-rewarder capacity in online DPO using direct pairwise preferences and evaluator-specific Elo arenas, without testing newer interventions or isolating a causal reasoning-format effect [Chiang et al., 2024].It does not establish the proposed mechanisms as explanations of its results.Source: §6 · p. 16

7 Conclusion

In the studied domain, Thinking GenRMs lead on static evaluation, but their advantage usually reverses during policy optimization under the learned Gold reference. The findings offer practical guidance about rewarder scale, checkpoint choice, and the limits of static accuracy for the compared SFT/GRPO pipelines.

7 Conclusion

  • Under the learned Gold reference, Thinking GenRMs outperform Answer-Only variants on static evaluation, yet this advantage usually reverses during policy optimization.The comparison concerns the studied SFT/GRPO pipelines, not an isolated causal effect of reasoning or a universal ranking of writing quality.Source: §7 · p. 16
  • Rewarder scale remains valuable even above policy size, and intermediate GenRM checkpoints can outperform fully trained models as rewarders.These findings are practical guidance within the reported experiments.Source: §7 · p. 16
  • In-distribution accuracy alone does not predict training effectiveness, so selecting GenRMs only by static benchmarks can choose checkpoints that are suboptimal under the learned reference.This conclusion is scoped to the experiments described in the paper.Source: §7 · p. 16

A.1 Trained and Off-the-Shelf Rewarders over Training

The trained-rewarder comparison tracks four rewarders against a common Gold evaluator and their own proxies. Trained Thinking reaches a higher Gold peak than its off-the-shelf counterpart, but its later trajectory crosses over while its own-proxy rating rises and Gold rating falls.

A.1 Trained and Off-the-Shelf Rewarders over Training

  • The figure distinguishes trained from off-the-shelf rewarders and Answer-Only from Thinking; its upper panel shares one Gold arena, while lower panels use separate own-proxy fits.Heights across the lower panels are not comparable; outlined points mark step 460.Source: Figure 9 · p. 19
  • Thinking’s trained Gold peak is 802.8 versus 542.3 for the baseline, but at shared step 460 the baseline leads 542.3 to 501.5.At step 450 the ordering is opposite, and only the baseline continues to step 470; the late crossover is not a significance result or uniform baseline superiority.Source: §A.1 · p. 19, Figure 9 · p. 19
  • From steps 320 to 460, the trained Thinking policy’s own-proxy rating rises from 1808.3 to 1971.5 while its Gold rating falls from 802.8 to 501.5.This is proxy–reference divergence, not an identified mechanism or evidence of human-quality decline.Source: §A.1 · p. 19

A.2 All Eight Matched Conditions in One Gold Arena

Figures 10 and 11 present the stored Gold-rated trajectories across eight matched policy–judge conditions in one common arena. They show observed trajectories rather than averaged seed curves or monotonic scaling claims.

A.2 All Eight Matched Conditions in One Gold Arena

  • Figures 10 and 11 show complete Gold-rated trajectories for all eight matched conditions on a shared arena and vertical scale.The trajectories expose early gains and late reversals that best-checkpoint summaries can hide.Source: §A.2 · p. 20
  • A falling trajectory tail does not by itself measure verdict agreement or reversion toward Gold’s training distribution.The curves show measured behavior, not monotonic improvement at every step or across every rewarder size.Source: §A.2 · p. 20, Figure 11 · p. 21
  • Each plotted line is one retained policy trajectory, not a mean across seeds.Figure 10 varies policy size with a fixed 4B rewarder; Figure 11 adds rewarder-size conditions at a fixed 4B policy.Source: Figure 10 · p. 20, Figure 11 · p. 21

A.3 The Same Gold Observations against Exported KL Coordinates

This section replots the common-arena Gold observations against retained policy-KL export coordinates, matching exported rows to trajectory records by exact prefix and Gold rating. The coordinates are ancillary descriptive exports, not reconstructed raw KL or a validated quantitative budget.

A.3 The Same Gold Observations against Exported KL Coordinates

  • 752 exported rows were uniquely matched to common-arena Gold observations using exact trajectory prefixes and Gold ratings within 10−7.Four additional global-arena snapshots lacked matching KL coordinates and were omitted rather than assigned inferred values.Source: §A.3 · p. 21
  • The exported coordinates are not fully reconstructed raw per-step KL, so these panels do not establish distance from Gold’s training distribution or a validated KL budget.Negative exported values are retained without interpreting them as raw KL measurements; the original preprocessing path is not fully reconstructed.Source: §A.3 · p. 21, Figure 12 · p. 22, Figure 13 · p. 23
  • Figures 12 and 13 show retained KL export coordinates: Figure 12 uses Gold common-arena observations from Figure 10 (47 coordinates per mode), excluding two 1.7B-policy continuation snapshots without retained KL coordinates; Figure 13 matches Figure 11.Figure 13 shows only uniquely matched exported observations, excluding two extra continuation snapshots in the 0.6B-rewarder cell that lack KL exports.Source: Figure 12 · p. 22, Figure 13 · p. 23

A.4 Rewarder and Policy Capacity Sweeps

The capacity sweeps summarize observed Gold-arena outcomes while varying rewarder size at fixed policy size and policy size at fixed rewarder size. Matched parameter-count swaps show that the preferred allocation differs by response format, without identifying a compute-optimal allocation.

A.4 Rewarder and Policy Capacity Sweeps

  • At a 12B summed parameter count, Answer-Only top-five Gold Elo is 1645.0 for the 8B-policy/4B-rewarder condition versus 1513.1 for the 4B-policy/8B-rewarder condition.Thinking reverses the ordering: 1402.2 versus 1488.1, respectively.Source: §A.4 · p. 23
  • A summed parameter count is not an inference- or training-compute budget, especially when a rewarder generates reasoning traces.The observed swaps therefore do not identify a universal compute-optimal allocation.Source: §A.4 · p. 23
  • Figure 14 summarizes each observed sweep cell by the average of its five highest Gold-rated snapshots, with rewarder size varied at fixed 4B policy and policy size varied at fixed 4B rewarder.These are descriptive summaries, not compute-normalized optima, predictive fits, or independent-seed confidence intervals.Source: §A.4 · p. 23, Figure 14 · p. 24
  • The intermediate-Thinking-checkpoint ablation covers judge steps 0, 50, 100, 150, 200, 250, and 300 with a 40,000-policy-prompt budget.The preserved figure does not support inferring exact static accuracies or checkpoint identities from pixels.Source: §A.4 · p. 24
  • The six appendix figures introduce no model runs, new labels, or significance tests.The analysis scripts and provenance ledgers record source hashes, row counts, coordinate joins, omissions, and condition summaries.Source: §A.4 · p. 24

B Evaluation and Analysis Details

The appendix details evaluator-specific Elo estimation, descriptive size fits, and possible reasons static accuracy may not predict online updates. It also reports a limited different-judge diagnostic and an ancillary OpenOrca comparison, while documenting the prompt templates used.

B.1 Elo Estimation and Conditional Uncertainty

  • Each evaluator and arena receives a separate Bradley–Terry fit, and ratings from different arenas cannot simply be subtracted.The reported 95% intervals resample stored winner/loser rows 200 times and describe sensitivity under an independent-row approximation, not fresh generations, seeds, or prompt-aware uncertainty.Source: §B.1 · pp. 24–25

B.2 Descriptive Scaling Fits

  • The post-hoc size model fits peak or top-five Gold Elo across 16 matched mode/size configurations, allowing size slopes to differ between pipelines.Both policy and judge scale have substantial fitted associations, but the sparse in-sample fit is not a universal scaling law or out-of-sample prediction.Source: §B.2 · p. 25, Table 5 · p. 25

B.3 Why Static Accuracy Is Not an Online-Update Guarantee

  • Online updates can differ despite equal average rewarder error because mistakes may occur on response pairs with different gradient directions or magnitudes sampled from the evolving policy.Filtering can also change which nonzero-sum pairs are retained; this explanation is possible, not a measured decomposition, and the artifacts do not establish which judge’s errors better track human preferences.Source: §B.3 · p. 26

B.4 Follow-Up Audit with a Different Learned Judge

  • The different-learned-judge diagnostic is inconclusive overall and does not establish a human evaluation result.Its two settings, selected checkpoints, changed rubric, shared Qwen3 family, and single verdict per pair limit interpretation.Source: §B.4 · p. 26, Table 6 · p. 26

B.5 Ancillary Policy Optimization on OpenOrca

  • The ancillary OpenOrca comparison uses a 1.7B SFT-initialized policy, online DPO with off-the-shelf Qwen3 rewarders, and a separate 32B Thinking reference evaluator.Its plot reports per-step net preference rates, not Elo, and is not a trained-GenRM creative-writing replication.Source: §B.5 · p. 27, Figure 15 · p. 27

C.1 Policy Prompt

  • The creative-writing policy prompt asks for an original short story that directly answers the instruction and outputs only story text.The QUESTION placeholder is replaced with the specific creative-writing instruction.Source: §C.1 · p. 28

C.2 GenRM Pairwise Evaluation Prompt

  • The GenRM prompt presents two replies to the same instruction and requests one structured verdict, A or B, about which better fulfills it.The placeholders are filled with the full policy prompt and candidate responses.Source: §C.2 · p. 28
Loading 2610.30765v1…