PaperlayerView exact revision on arXiv

Source-linked AI summary

RewardBench 2: Advancing Reward Model Evaluation

Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert

arXiv:2506.01937v2cs.CL

TL;DR

Reward models need evaluations that measure difficult skills and predict downstream usefulness without relying on reused prompts. RewardBench 2 addresses this with a multi-domain, accuracy-based benchmark built largely from unseen human prompts and evaluates its relationship to inference-time scaling and RLHF. The benchmark is challenging and useful for downstream prediction, but RLHF model selection also depends on training context.

  • Problem

    Reward-model benchmarks need to assess multiple skills and downstream usefulness while avoiding reliance on prompts reused from downstream evaluations.

  • Method

    RewardBench 2 uses mostly unseen human prompts, six skill domains, four-completion accuracy tests, and downstream analyses involving best-of-N sampling and PPO.

  • Results

    20 or more points lower: leading RewardBench models score at least 20 points lower on RewardBench 2, which strongly correlates with best-of-N performance and provides a helpful PPO signal.

  • Takeaways & Limitations

    Benchmark accuracy is a prerequisite for strong RLHF training but is not sufficient; reward-model selection should account for the specific training setup.

  • Takeaways & Limitations

    For PPO, benchmark results must be interpreted in the context of the training setup rather than used to select the top-scoring checkpoint directly.

Abstract

from arXiv · show

Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target for optimization across instruction following, reasoning, safety, and more domains. The community has begun establishing best practices for evaluating reward models, from the development of benchmarks that test capabilities in specific skill areas to others that test agreement with human preferences. At the same time, progress in evaluation has not been mirrored by the effectiveness of reward models in downstream tasks -- simpler direct alignment algorithms are reported to work better in many cases. This paper introduces RewardBench 2, a new multi-skill reward modeling benchmark designed to bring new, challenging data for accuracy-based reward model evaluation -- models score about 20 points on average lower on RewardBench 2 compared to the first RewardBench -- while being highly correlated with downstream performance. Compared to most other benchmarks, RewardBench 2 sources new human prompts instead of existing prompts from downstream evaluations, facilitating more rigorous evaluation practices. In this paper, we describe our benchmark construction process and report how existing models perform on it, while quantifying how performance on the benchmark correlates with downstream use of the models in both inference-time scaling algorithms, like best-of-N sampling, and RLHF training algorithms like proximal policy optimization.

1 INTRODUCTION

RewardBench 2 introduces a challenging, multi-domain accuracy benchmark using mostly unseen human prompts and a four-completion evaluation format. Its evaluations examine reward-model performance and downstream usefulness in inference-time scaling and RLHF.

  • Benchmark design: The benchmark uses one chosen and three rejected completions, adding distractions compared with pairwise evaluation and lowering the random baseline to 25%.The format increases headroom for distinguishing strong reward models from chance performance.
  • Benchmark scope: RewardBench 2 measures reward models across six domains, including focus, math, safety, factuality, precise instruction following, and ties.Ties tests calibration between equivalently valid answers.
  • Data construction: Most prompts are previously unused human prompts from the WildChat pipeline, filtered through manual, programmatic, and language-model-based procedures.The construction prioritizes evaluation data independent of downstream benchmarks.
  • Benchmark results: 20 or more points lower: leading RewardBench models score at least 20 points lower on RewardBench 2, with leading models below 40% on Precise Instruction Following and 70% on Math.These results indicate a more challenging evaluation than the first RewardBench.
  • Downstream use: RewardBench 2 correlates with inference-time scaling performance and provides a useful signal for PPO training, while RLHF performance depends on training context.For RLHF, reward models from the policy model’s lineage are favored; selecting solely by benchmark score may degrade downstream performance.

2 BACKGROUND

Reward models learn scalar quality signals from preference data, while reward-model benchmarks differ in whether they test accuracy, preference agreement, or downstream performance. RewardBench 2 emphasizes unseen prompts and accuracy-based evaluation to reduce contamination concerns and better target downstream skills.

  • Reward models: Reward models are trained on preference pairs to output scalar values that predict which completion is preferred.The canonical setup compares a chosen completion with a rejected completion for each prompt.
  • Applications: Reward models support RLHF, online direct alignment, data filtering, and inference-time scaling methods such as best-of-N sampling.Their uses extend beyond training a policy with reinforcement learning.
  • Evaluation landscape: Existing reward-model evaluations vary across accuracy, human or LM-as-a-judge agreement, domains, and downstream-performance analysis.Prior work includes classification benchmarks and evaluations connected to best-of-N sampling and RLHF.
  • Benchmark rationale: RewardBench 2 uses mostly unseen human prompts rather than repurposed downstream-evaluation prompts, addressing potential contamination when measuring downstream correlation.The prompts are intended to remain independent of evaluations used for downstream post-trained models.
  • Benchmark rationale: Because pairwise preference labels can be subjective, RewardBench 2 uses accuracy-based tests focused on downstream skills.This choice targets domains with known correct answers rather than preference agreement alone.

3 BUILDING THE BENCHMARK AND MEASURING PERFORMANCE

RewardBench 2 combines mostly unseen human prompts with domain-specific completion generation, verification, and filtering across six skill areas. It scores four-completion selection accuracy, while Ties additionally evaluates confidence ordering among valid and invalid answers.

  • Construction pipeline: The benchmark construction pipeline comprises prompt sourcing, domain annotation, diverse completion generation, and filtering for prompt and completion quality.Most prompts are unreleased human-written queries obtained with user consent from WildChat.
  • Domains: The six domains include improved datasets for Math, Safety, and Focus, plus new Factuality, Precise Instruction Following, and Ties domains.The new domains target hallucination detection, precise constraints, and calibration among multiple correct answers.
  • Domain construction: Factuality uses natural and intentionally error-containing completions labeled accurate only when two independent language-model judges agree.The resulting labels are spot-checked for integrity.
  • Domain construction: Precise Instruction Following combines one constraint-satisfying completion with three violating completions, using verifier functions and manual checks.The constraints are drawn from the IFBench taxonomy and appended to prompts when relevant.
  • Scoring: 25%: standard accuracy selects the correct response from four completions, producing a 25% random baseline and an unweighted average across domains.Accuracy is used for every subset except Ties.
  • Scoring: Ties combines correctness with whether reward margins distinguish incorrect answers from the score range among correct answers.The metric evaluates both accuracy and whether confidence ordering reflects quality differences.

4 ANALYSIS OF PERFORMANCE ON REWARDBENCH 2

REWARDBENCH 2 challenges both top existing and newly trained reward models, revealing domain- and training-dependent performance patterns. Comparisons with RewardBench also suggest that external-model scores may be affected by metric capture.

  • Existing Reward Models: Top existing reward models remain challenged by Instruction Following, Math, and Factuality, leaving room for improvement across several domains.The benchmark evaluates generative models with both best-of-four selection and absolute-rating prompts, reporting each model’s better setting.
  • Comparison with RewardBench: External-model scores are less correlated across the two benchmarks than scores for the newly trained models, indicating potential metric capture in RewardBench version 1.This pattern is reported for the benchmark comparison involving both existing and newly trained models.
  • Newly Trained Reward Models: Llama 3.1 Instruct-based reward models perform strongly at both 8B and 70B, with larger reward models performing better overall.The analysis attributes the size trend to stronger underlying base models.
  • Newly Trained Reward Models: Different training data sources benefit different domains, while combining the Tulu and Skywork mixtures improves average performance across all base models.Skywork is particularly helpful for focus and safety, whereas Tulu is better for factuality.
  • Newly Trained Reward Models: Base-model choice can dominate domain performance: Qwen 2.5 7B Instruct-based models excel on math, outperforming some 70B reward models.This matches the strong math capabilities of the underlying Qwen Instruct models.

5 ANALYSIS OF DOWNSTREAM EVALUATIONS

RewardBench 2 scores strongly predict reward-model performance in best-of-N sampling, while PPO outcomes depend on model and prompt alignment and saturate for decent-to-good reward models.

  • Motivation: A good reward-model benchmark should predict downstream applications, reducing the need for costly full downstream experiments.The analysis examines best-of-N inference-time sampling and RLHF training as downstream use cases.
  • 5.1 Inference-Time Scaling with Best-of-N Sampling: 0.87 Pearson correlation links RewardBench 2 scores with average downstream scores under best-of-N sampling across 113 reward models.The evaluation spans GSM8K, MATH, IFEval, AlpacaEval 2, BBH, PopQA, and HumanEval+.
  • 5.1 Inference-Time Scaling with Best-of-N Sampling: RewardBench 2’s math subset strongly signals downstream math and coding performance, while factuality shows the highest domain correlation.The related downstream tasks are GSM8K, MATH, and HumanEval+ for the math-subset comparison.
  • 5.1 Inference-Time Scaling with Best-of-N Sampling: IFEval and PopQA correlate relatively less with RewardBench 2, mirroring their lower correlations with other downstream tasks.Focus and Ties also correlate less with downstream performance because their skills are not directly represented in the downstream evaluations.
  • 5.2 Preference Finetuning with RLHF: 54.1 is the starting Tulu 3 8B SFT score, 60.3 is the Tulu 3 8B DPO score, and the best PPO model exceeds DPO.PPO performance quickly saturates near DPO for reward models scoring 49.8 to 68.5 on RewardBench 2.
  • 5.2 Preference Finetuning with RLHF: Misalignment between the policy and reward model’s base model or prompt distribution significantly reduces downstream PPO performance.Reward models initialized from different starting points can score highly on RewardBench 2 yet often fail to improve downstream policy metrics.

6 CONCLUSION

REWARDBENCH 2 offers broad, multi-domain accuracy evaluation that correlates with downstream use, while benchmark scores remain insufficient for selecting PPO reward models without training-context considerations.

  • REWARDBENCH 2 provides a broad, multi-domain accuracy-based evaluation that can translate into downstream use.The conclusion frames the benchmark as a strong signal for reward-model accuracy and Best-of-N use.
  • For PPO, benchmark results should be interpreted within the specific training setup rather than used to select the top checkpoint directly.The authors recommend integrating the selected model’s training recipe into the workflow instead.
  • Reward-model evaluation frameworks will need more contextual and situational performance insights as capabilities and post-training uses diversify.

C.1 CORRELATION AND DIVERGENCE WITH OTHER BENCHMARKS

REWARDBENCH 2 scores correlate with other reward-model benchmarks, with especially strong relationships to accuracy-based evaluations, while remaining substantially harder than RewardBench.

  • REWARDBENCH 2 scores are fairly correlated with scores from other reward-model benchmarks.The correlation is higher for other accuracy-based benchmarks than for the human-preference component of PPE.
  • REWARDBENCH 2 has a higher correlation with other accuracy-based benchmarks than with PPE’s human-preference benchmark component.
  • REWARDBENCH 2 is significantly harder than RewardBench while having score ranges comparable to RM-Bench and PPE.

D TRAINING REWARD MODELS

The authors train reward models in a controlled setup that varies training duration, learning rate, base model, and related training choices, while leaving newer reward-modeling methods for future work.

  • The study trains 120 Bradley-Terry reward models in a controlled setup using the Open-Instruct library.
  • Training experiments vary one, two, and three epochs alongside learning rates of 1 × 10−6, 3 × 10−6, and 2 × 10−5.
  • The paper leaves experimentation with generative reward models, reasoning-generating reward models, and newer inference-time methods to future work.

E ANALYSIS OF OUR NEW TRAINED REWARD MODELS

Reward-model performance varies with the post-trained base model and training data, including domain-specific differences and preferences for completions from the model’s own lineage.

  • Base-model effects: Post-trained base models within the same lineage produce different reward-model capabilities, with Llama 8B Instruct-based models performing best among the tested Llama variants.
  • Base-model effects: Tulu SFT, DPO, and RL reward models have similar average scores but separate by domain, with Safety decreasing from SFT to DPO and RL.
  • Base-model preference: Reward models show a statistically significant preference for completions from their own base model or base-model family.
  • Base-model preference: Training data also affects this preference: models trained on Tulu preference data prefer Tulu completions more than models trained on Skywork Preference data.

G ADDITIONAL DATASET CREATION DETAILS

RewardBench 2 combines varied model-generated and human-judged data procedures to test reward models across challenging domains. Its construction emphasizes fairness, calibration, and meaningful downstream correlation.

  • Data creation: RewardBench 2 uses a diverse model pool because reward models show slight preference for completions generated by their own base model.This controls for self-preference across training data sources.
  • Factuality: Factuality examples compare natural and intentionally error-inducing completions, with GPT-4o labeling accuracy and Claude Sonnet 3.7 validating the most accurate response.Disagreements between judges lead to discarding datapoints.
  • Factuality: Natural-only rejected responses are hardest for reward models in the factuality subset, suggesting models readily detect induced errors.This setting limits the number of available data instances.
  • Precise instruction following: Precise Instruction Following requires all completions for a prompt to come from one model because response quality and constraint satisfaction create a dual objective.Mixing completion models would make comparisons unfair when those objectives conflict.
  • Safety: Safety data evaluates nuanced compliance or refusal for harmful-use prompts using modified CoCoNot taxonomies, conservative exclusions, manual verification, and a broad model pool.The model pool is needed because most recent language models already produce correct refusals.
  • Generative evaluation: Generative-model evaluation compares rankings and 1–10 ratings, but judges often assign identical ratings to multiple candidates, limiting granularity.The benchmark reports the better-performing prompting strategy because best practices for LM judges remain unsettled.
  • Epoch exploration: Training for three epochs provides no strong benefit, whereas two epochs improve accuracy in some configurations and are therefore retained for further experiments.The epoch sweep compares Tulu 3 8B SFT and Qwen 2.5 7B Instruct configurations.
  • Inference-time evaluation: A weaker generator such as Tulu 8B SFT produces enough variation across 16 responses to support meaningful Best-of-N scoring and downstream correlations.Stronger instruction-tuned generators were too high-performing for this setup.

K.2 CORRELATION WITHIN DOWNSTREAM TASKS

Within downstream evaluations, IFEval and PopQA correlate less with RewardBench 2 than other tasks, matching their lower correlations across downstream tasks. PPO results also distinguish reward models by training-distribution setting.

  • Task correlations: IFEval and PopQA are relatively less correlated with RewardBench 2, mirroring their lower correlations with other downstream tasks.The comparison is based on the downstream-task correlation analysis shown in Figure 7.
  • Training-distribution settings: RewardBench 2 downstream analyses compare on-policy reward models with in-distribution prompts, intentionally flipped preferences, and off-policy or out-of-distribution training.The table reports both Best-of-N and PPO-related comparisons.
  • Training-distribution settings: Reward models trained off policy or on out-of-distribution prompts perform highly on RewardBench 2 and Best-of-N, yet lag behind in PPO.This separates benchmark and inference-time performance from PPO outcomes for this model category.

M DOWNSTREAM CORRELATION OF OTHER BENCHMARKS

Additional benchmark comparisons show that the downstream trends identified for RewardBench 2 extend to several other accuracy-based reward-model benchmarks. These comparisons provide broader context for interpreting correlations with RLHF outcomes.

  • Broader benchmark trends: The on-policy and off-policy trends identified for RewardBench 2 extend to RMBench, PPE Human Preference and Correctness, and RewardBench.These comparisons evaluate trained post-RLHF models on additional accuracy-based benchmarks.
Loading 2506.01937v2…