RewardBench 2: Advancing Reward Model Evaluation

• Most prompts are previously unused human prompts from WildChat, and the benchmark uses a best-of-4 format with one chosen and three rejected completions.The additional rejected samples are intended to reduce the distance between strong reward models and the random baseline.Source: Figure 1 · p. 2, §1 · p. 2
• Six domains combine new datasets for focus, math, and safety with factuality, precise instruction following, and ties for equivalently valid answers.Source: §1 · p. 2
• Leading RewardBench models score 20 or more points lower on REWARDBENCH 2, with leading models below 40% accuracy on Precise Instruction Following and below 70% on Math.Source: §1 · p. 2
• Controlled experiments find that post-trained base models within the same lineage and family yield different reward-model capabilities, and training beyond one epoch can be beneficial.The latter finding contrasts with the stated accepted best practice.Source: §1 · p. 2
• RewardBench 2 correlates strongly with best-of-N outcomes and provides a useful PPO signal, but RLHF reward models should match policy lineage because a mismatch can significantly degrade downstream performance.Source: §1 · p. 2

Reward Models

• Reward models learn from chosen–rejected completion pairs, using a Bradley–Terry model to predict preference probabilities from scalar reward scores fitted by maximum likelihood.For each prompt, the better completion is labeled chosen and the other rejected.Source: §2 · p. 3

Reward Model Benchmarking

• Reward-model benchmarks test general downstream performance, specific attributes such as multilinguality or agentic skills, and different modalities or reward-model structures.The passage also notes that pairwise human or language-model preferences can prescribe subjective optimization targets, motivating accuracy-based tests focused on downstream skills.Source: §2 · pp. 3–4
• RewardBench 2 uses unseen human prompts, unlike most prior benchmarks that reuse downstream-evaluation prompts, helping address potential contamination with downstream targets.Its construction prioritizes new human prompts and robust subset-specific completion generation and verification; the benchmark contains 1,865 prompts and completions from 20 models or human-written completions.Source: §2 · p. 3, Table 2 · p. 4

Prompt Sourcing

• About 70% of prompts are unseen, consented real-use queries, with the final 1,865 prompts manually filtered and checked for overlap with 20 downstream evaluations.The workflow includes prompt sourcing, classifier-based quality and domain annotation, diverse right and wrong completion generation, and filtering; code and data are slated for release under Apache 2.0 and ODC-By, respectively.Source: §3 · p. 4

Constructing RewardBench 2’s Domains

• The six domains include new Math, Safety, and Focus datasets inspired by earlier benchmarks, plus Factuality, Precise IF, and Ties for additional reward-model capabilities.Factuality pairs natural and subtly error-inducing completions, labels them only when two LLM judges agree, and spot-checks examples.Source: §3 · pp. 4–5

Precise Instruction Following

• Precise Instruction Following tests constraints such as answering without a specified letter, using one compliant and three noncompliant responses verified for constraint adherence and response quality.Constraints come from IFBench’s taxonomy (Pyatkin et al., 2025), are manually checked for relevance, and are evaluated with verifier functions.Source: §3 · p. 5

Math

• Math covers open-ended prompts from middle-school through college-level topics, with majority voting to select one correct and three incorrect completions and manual verification of every sample.Manual verification addresses the brittleness of answer extraction.Source: §3 · p. 5

Safety

• Safety evaluates compliance or refusal on harmful-use prompts using modified CoCoNot taxonomies and GPT-4o rubrics, with one noncompliant and three compliant completions manually verified per instance.The construction aims to be conservative where users may disagree about appropriate model behavior (Brahman et al., 2024).Source: §3 · p. 5

Focus

• Focus tests whether reward models prefer on-topic answers by pairing a natural completion with three off-topic, incorrect, or unresponsive alternatives created through prompt rewrites.The method follows LLMBar (Zeng et al., 2024).Source: §3 · p. 5

Ties

• Ties tests whether models avoid arbitrary preferences among equivalent correct answers while still preferring correct over incorrect ones; its weighted score also measures reward-margin alignment with quality differences.Samples are created manually with AI assistance; the distribution-aware metric rewards correctness and confidence ordering, supporting a stronger RLHF signal for correctness than for reducing diversity among correct responses.Source: §3 · p. 5

Scoring RewardBench 2

• Accuracy is measured by choosing the sole correct response among four completions, giving a 25% random baseline; the final score is the unweighted mean across six domains.Ties is the exception to accuracy-only scoring.Source: §3 · p. 5

Existing Reward Models

• Top existing reward models are challenged by RewardBench 2, particularly on instruction following, math, and factuality.For generative models, the reported score uses whichever of two prompting strategies performs better: choosing among four options or rating one option absolutely.Source: Table 3 · p. 6, §4 · p. 6

Newly Trained Reward Models

• The authors trained Bradley-Terry reward models in a controlled setup varying hyperparameters, base models, and preference-data mixtures.They used the Open Instruct library and tested Tulu and Skywork preference mixes.Source: §4 · pp. 6–7

Existing Reward Models

• RewardBench 2 scores are much lower than RewardBench scores, and scores on the two benchmarks are less correlated for external models than for the authors’ trained models.The authors state that they did not tune RewardBench 2 development to their trained models.Source: Figure 2 · p. 6, §4 · p. 6

Newly Trained Reward Models

• Llama 3.1 Instruct-based models perform strongly at both 8B and 70B, and larger reward models perform better; combining Tulu and Skywork data improves average performance across all base models.Skywork data particularly helps focus and safety, while Tulu data is better for factuality.Source: §4 · p. 7
• Base-model capabilities strongly affect some domains: Qwen 2.5 7B Instruct-based models excel at math, while post-training stage differences appear to carry over to reward-model performance.The Qwen-based models outperform even 70B reward models trained on Llama 3.1 70B Instruct and Tulu 3 70B SFT for math.Source: §4 · p. 7
• Eight of the eighteen best models in Appendix Table 5 were trained for two epochs, and training beyond one epoch can help performance in some cases.The paper also reports that multiple-epoch training does not inherently hurt downstream performance.Source: §4 · p. 7

5 ANALYSIS OF DOWNSTREAM EVALUATIONS

• The analysis tests reward-model prediction for best-of-N inference and RLHF; prior work also finds that score variance, beyond overall accuracy, affects RLHF performance (Razin et al., 2025).The authors report that other benchmarks also correlate well with best-of-N outcomes but weakly with RLHF outcomes, attributing the gap to on-policy versus off-policy effects.Source: §5 · pp. 7–8, §5.2 · p. 9

5.1 INFERENCE-TIME SCALING WITH BEST-OF-N SAMPLING

• 0.87 Pearson correlation linked average REWARDBENCH 2 scores with average downstream scores under best-of-N sampling across 113 reward models.The Factuality domain had the highest correlation; the benchmark’s math subset was especially predictive on GSM8K, MATH, and HumanEval+.Source: Figure 3 · p. 7, §5.1 · p. 8
• IFEval and PopQA had relatively lower benchmark correlations, while Focus and Ties were less correlated because their skills were not directly tested by the downstream evaluations.The authors note that lower Focus and Ties correlations do not make those reward-model capabilities less valuable.Source: §5.1 · p. 8

5.2 PREFERENCE FINETUNING WITH RLHF

• Best PPO exceeded Tulu 3 8B DPO’s average score of 60.3 (versus 54.1 for SFT), but performance saturated near DPO for reward models scoring 49.8–68.5 on REWARDBENCH 2.Scores averaged nine Tulu 3 Evaluation Suite tasks; the best PPO model outperformed DPO, the best comparable model in the suite.Source: §5.2 · p. 8, Figure 4 · p. 9
• PPO performance dropped significantly when the reward model’s base-model lineage or training-prompt distribution was misaligned with the policy or PPO prompts.Different-starting-point reward models had the strongest effect; top REWARDBENCH 2 scorers often failed to improve the policy, and the gap persisted at β = 0.0325.Source: §5.2 · pp. 8–9, Figure 4 · p. 9

6 CONCLUSION

• RewardBench 2 provides a strong signal of reward-model accuracy and Best-of-N use, but benchmark accuracy is necessary rather than sufficient for strong RLHF training.The paper highlights additional training-context-specific factors that affect RLHF performance and are not captured by general benchmark accuracy.Source: §6 · p. 9
• For PPO, benchmark results should be interpreted in the context of the training setup; researchers should integrate a top model’s recipe into their workflow rather than simply use its checkpoint.The benchmark may guide off-the-shelf model choice in settings such as Best-of-N sampling, but PPO results depend on the training setup.Source: §6 · p. 9
• Reward-model evaluation frameworks will need to evolve to provide more contextual and situational insight as model capabilities improve and post-training uses diversify.Source: §6 · p. 9

B ADDITIONAL BACKGROUND

• Reward models support post-training uses from data curation to online reinforcement learning, where they rank sampled completions or score policy completions.The passage gives rejection sampling and PPO and group relative policy optimization as examples.Source: B ADDITIONAL BACKGROUND · p. 17
• Best-of-N sampling is an inference-time scaling method that leaves the generating model’s weights unchanged, though it can be compared with online training methods such as PPO in some contexts.The passage notes that KL distance can be measured for Best-of-N relative to another policy.Source: B ADDITIONAL BACKGROUND · p. 17

C DETAILED COMPARISON WITH OTHER BENCHMARKS

• 20 points lower scores on RewardBench show that REWARDBENCH 2 is a harder evaluation, with three new subsets and unseen prompts from downstream evaluations.Its new Factuality, Precise IF, and Ties subsets capture capabilities and robustness.Source: C DETAILED COMPARISON WITH OTHER… · p. 18
• Unlike RewardMATH’s math focus and RM-Bench, ReWordBench, and M-RewardBench’s targeted variations on RewardBench, REWARDBENCH 2 evaluates six domains broadly using a different prompt source.Those variations focus on data style and length, surface perturbations, or multilinguality, respectively.Source: C DETAILED COMPARISON WITH OTHER… · p. 18
• For PPE, REWARDBENCH 2 avoids subjective preference prescriptions and downstream-evaluation prompts, while PPE’s correctness set uses five downstream evaluations and risks development-pipeline contamination.PPE also has a human-preference set sourced from ChatBot Arena interactions.Source: C DETAILED COMPARISON WITH OTHER… · p. 18
• Unlike RMB’s Wildchat-train prompts, model-generated responses, and LLM-as-a-judge preferences, REWARDBENCH 2 uses an objective accuracy-based approach.Source: C DETAILED COMPARISON WITH OTHER… · p. 18

C.1 CORRELATION AND DIVERGENCE WITH OTHER BENCHMARKS

• Across 17 reward models, RewardBench 2 scores are fairly correlated with other benchmarks, more so for accuracy-based benchmarks than PPE’s human-preference component.Source: §C.1 · p. 18, Table 4 · p. 18
• RewardBench 2 is significantly harder than RewardBench, with score ranges comparable to RMBench and PPE.Source: §C.1 · p. 18

D TRAINING REWARD MODELS

• The authors trained 120 Bradley–Terry reward models with Open-Instruct, varying epochs, learning rates, base-model stages and families, and preference-data mixtures.They tested 1, 2, and 3 epochs and learning rates of 1 × 10−6, 3 × 10−6, and 2 × 10−5, with selected configurations also run on 70B models.Source: D TRAINING REWARD MODELS · p. 19
• Combining the full Tulu 8B and Skywork preference mixes benefits performance, while subsampling degrades it; flipping Tulu preferences produces low-performing controls.The Tulu mix contains 270K pairwise GPT-4o-as-a-judge preferences, and the Skywork mix contains 80K preferences.Source: D TRAINING REWARD MODELS · p. 19
• The study leaves detailed experimentation with newer reward-model approaches—including generative or reasoning-generating models and inference-time scaling—for future work.The surveyed directions also include aspect-conditioned models, human datasets, scaling, and data debiasing.Source: D TRAINING REWARD MODELS · p. 19

E ANALYSIS OF OUR NEW TRAINED REWARD MODELS

• Llama 8B Instruct-based reward models perform best among the tested Llama lineages, and all tested post-trained Llama models outperform Llama 8B Base.The same base-versus-Instruct trend appears for Qwen 7B; Table 6 compares post-training stages by model family.Source: E ANALYSIS OF OUR NEW TRAINED REWARD… · pp. 19–20, Table 6 · p. 20
• Among Tulu 8B SFT-, DPO-, and RL-based reward models, average scores are similar, but most domains improve while Safety declines from SFT to DPO and RL.The authors describe per-domain differences as matching capabilities of the corresponding post-trained models.Source: E ANALYSIS OF OUR NEW TRAINED REWARD… · p. 20

F REWARD MODELS HAVE A PREFERENCE FOR THEIR BASE MODEL’S OUTPUTS

• The base-model-preference analysis evaluates reward models on completions from eight models across 977 prompts, comparing their scores relatively rather than controlling for completion quality.The prompts were reused from the initial unfiltered Chat subset.Source: F REWARD MODELS HAVE A PREFERENCE FOR… · pp. 20–21
• Reward models show a statistically significant lean toward completions from their own base model or family, as reflected in higher average inverse ranks.The trend appears for Tulu-, Llama-, and Qwen-based reward models; the authors note that a diverse completion pool is important for fair comparison.Source: Figure 5 · p. 21, F REWARD MODELS HAVE A PREFERENCE FOR… · p. 21
• Tulu-based reward models rank Tulu completions higher than Llama- and Qwen-based reward models do, with analogous trends for Llama and Qwen completions.Figure 5 reports average inverse rank by reward-model base type, with error bars showing standard deviation across models in each base-model group.Source: Figure 5 · p. 21, F REWARD MODELS HAVE A PREFERENCE FOR… · p. 21
• The base-model preference remains when reward models are separated by training-data source, although Tulu-preference-trained models show a stronger preference for Tulu completions than Skywork-trained models.The authors attribute this stronger Tulu preference to Tulu data containing on-policy Tulu SFT completions and having been used to train Tulu DPO.Source: F REWARD MODELS HAVE A PREFERENCE FOR… · p. 21

G ADDITIONAL DATASET CREATION DETAILS; H MODEL POOL

• Reward models showed slight preference for completions from their own base models, so dataset creation used a diverse model pool; Table 7 lists pools by subset except for manually constructed Ties.Reward scores are converted to per-prompt ranks before aggregation because score ranges differ across reward models.Source: G ADDITIONAL DATASET CREATION DETAILS · p. 21, Figure 6 · p. 22, H MODEL POOL · p. 23

G.1 FACTUALITY; G.3 MATH

• Factuality examples use GPT-4o labels and Claude Sonnet 3.7 verification, while math examples use majority-voted gold answers and one correct plus three incorrect completions.Claude disagreement led to discarding about 30% of factuality datapoints; natural-only rejected responses were harder, so the subset combines 213 natural and 269 system-prompted completions, with selection methods correlating above 0.85. All math examples were manually verified after Llama 3.1 8B Instruct graded answer matches.Source: §G.1 · p. 22, §G.3 · p. 23

G.2 PRECISE INSTRUCTION FOLLOWING

• Precise instruction-following examples use completions from the same model to control response quality, and exclude especially stringent word-level constraints.Constraints that do not fit a prompt, such as camelCase requirements for non-coding queries, are filtered out.Source: §G.2 · p. 22

G.4 SAFETY

• The Safety subset adapts CoCoNot (Brahman et al., 2024) with nuanced compliance rubrics, excludes debated categories, and manually verifies examples.Because most recent models followed the taxonomy, the authors used a wide model pool and augmented natural completions with rejected responses elicited by simple system-prompt jailbreaks.Source: §G.4 · p. 23

I EVALUATING GENERATIVE MODELS

• Generative-model judging used rankings or 1–10 ratings, but ratings often lacked granularity; the authors explored both and report the better-performing approach.Some Safety requests may have been content-moderated by API models’ safety filters.Source: I EVALUATING GENERATIVE MODELS · p. 23

J EPOCHS EXPLORATION

• Three training epochs brought no strong benefits, while two epochs improved accuracy in some configurations, so later experiments tested one and two epochs.The sweep varied training data and base models; three epochs occasionally helped, particularly at lower learning rates and in the Ties subset.Source: J EPOCHS EXPLORATION · p. 24, Table 8 · p. 25

K.1 CHOICE OF GENERATOR

• Tulu 3 8B SFT produced more informative Best-of-N comparisons and higher downstream correlation than stronger generators in this setup.With stronger generators, more of 16 sampled responses were correct, reducing the quality range among options and weakening the signal from reward scores.Source: §K.1 · pp. 24–25, Figure 7 · p. 25

K.2 CORRELATION WITHIN DOWNSTREAM TASKS

• IFEval and PopQA are relatively less correlated with RewardBench 2, mirroring their lower correlation with other downstream tasks.Source: §K.2 · p. 25

L FULL PPO EXPERIMENT RESULTS

• Off-policy or out-of-distribution reward models score highly on RewardBench 2 and downstream best-of-N, but lag behind in PPO.Table 9 presents the full PPO results alongside on-policy models trained with in-distribution prompts, including models intentionally trained on flipped preference data.Source: L FULL PPO EXPERIMENT RESULTS · p. 25, Table 9 · p. 26
• Table 9 compares on-policy reward models trained on in-distribution prompts, including intentionally flipped-preference models, with off-policy or out-of-distribution models.Source: Table 9 · p. 26

M DOWNSTREAM CORRELATION OF OTHER BENCHMARKS

• On- and off-policy trends identified for RewardBench 2 extend to RMBench, PPE, and RewardBench, which provide further context for correlations with RLHF outcomes.Table 10 adds results for post-RLHF models on these benchmarks; the authors identify on- and off-policy considerations as important alongside benchmark scores.Source: M DOWNSTREAM CORRELATION OF OTHER… · p. 26, Table 10 · p. 27

N MODEL LICENSES

• The project used the Apache 2.0-licensed Open-Instruct library and reports that its model-pool generations were used permissibly under the models’ licenses.The pool includes open-weight and proprietary models, with licenses listed in Appendix Table 7.Source: N MODEL LICENSES · p. 27

19. GPT-4o (Outputs produced by GPT-4 are subject to OpenAI’s terms of use)

• The model-license list states that Claude 3.5 Sonnet outputs are subject to Anthropic’s terms of service and usage policy.Source: §19 · p. 28

O BROADER IMPACT

• The authors say Safety-subset completions may expose sensitive text, while jailbreak prompts elicit random outputs for PII or copyright prompts and the benchmark uses ground-truth correct and incorrect responses.They state that this accuracy-based design avoids biasing the benchmark toward particular groups’ preferences.Source: O BROADER IMPACT · p. 28

P COMPUTE USAGE

• The project used 55,000 GPU hours across evaluation, reward-model training, and PPO experiments.Evaluation covered 160 models and took around 8 minutes for an average 8B model or 30 minutes for an average 70B model; the project trained around 120 8B and five 70B reward models and ran 17 PPO experiments.Source: P COMPUTE USAGE · p. 28