PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Evaluate Reward Models for RLHF

Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica

arXiv:2410.14872v2cs.LGcs.AIcs.CL

TL;DR

Testing reward models by training and evaluating a new LLM is prohibitively expensive. PPE evaluates reward models on human-preference and verifiable-correctness proxy tasks and links those metrics to outcomes from end-to-end RLHF experiments. The authors report a 77% Pearson correlation with downstream performance and find granular measures such as accuracy are the best predictors.

  • Problem

    The direct test of a reward model is whether it produces good post-RLHF language-model performance, but training and evaluating a new LLM for each reward model is prohibitively expensive.

    Source: §1 · p. 1

  • Method

    PPE evaluates reward models on human-preference and verifiable-correctness datasets, then correlates those metrics with human preferences for models produced by end-to-end RLHF experiments.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    77% Pearson correlation with downstream performance; across the board, granular measurements such as accuracy are the best predictors.

    Source: §9 · p. 10

  • Takeaways & Limitations

    The results suggest that granular measurements and lower-bound performance may be more indicative of expected reward-model performance in the RLHF pipeline.

    Source: §9 · p. 10

    End-to-end testing covers nine selected models and uses DPO; downstream performance is therefore specific to the base model and RLHF algorithm used.

    Source: §8.2 · p. 10

Abstract

from arXiv · show

We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback). The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance. However, this process is prohibitively expensive. To address this, we build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. These proxy tasks consist of a large-scale human preference and a verifiable correctness preference dataset, in which we measure 12 metrics across 12 domains. To investigate which reward model metrics are most correlated to gold-standard RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowdsourced human preference platform to view real reward model downstream performance as ground truth. Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we open-source for public use and further development. Our code and evaluations can be found at https://github.com/lmarena/PPE .

1 INTRODUCTION

Evaluating a reward model by training and testing a new LLM is prohibitively expensive, slowing reward-model improvement. The paper introduces PPE, using proxy evaluations and end-to-end RLHF experiments to connect reward-model metrics with downstream human preferences.

1 INTRODUCTION

  • Directly testing downstream LLM performance requires training a new LLM for each reward model, making evaluation prohibitively expensive and time-consuming.The long development-feedback cycle is described as limiting achievable reward-model quality and the effectiveness of RLHF.Source: §1 · p. 1
  • The authors run selected reward models through full RLHF and measure the resulting models’ human-preference scores on Chatbot Arena.This end-to-end experiment identifies which metrics across domains correlate most strongly with post-RLHF performance.Source: §1 · p. 2
  • PPE evaluates reward models with human-preference and verifiable-correctness datasets, measuring 12 metrics across 12 domains.The released resources include 16,038 labeled human-preference pairs and 2,555 prompts with 81,760 responses grounded in correctness labels.Source: §1 · p. 2
  • The paper analyzes reward-model metric correlations with post-RLHF human preference and open-sources PPE for further use.The introduction presents PPE as the only reward-model benchmark directly linked to downstream RLHF outcomes.Source: §1 · p. 2

2 RELATED WORK

Prior work uses reward models and LLM judges as proxies for human preference, while research also studies reward models as RLHF training signals. Existing reward-model benchmarks cover relevant tasks, but their relationship to RLHF success remains an open question.

2.1 HUMAN PREFERENCE AND REWARD MODELS

  • Reward models approximate human preferences for training, while LLMs have also been used as judges for evaluating other LLMs.The RLHF training family first trains a reward model as a preference proxy, then uses it as the signal for reinforcement-learning fine-tuning.Source: §2.1 · p. 2
  • The paper frames selecting reward models for downstream performance as a critical problem in the RLHF process.Its stated question is how to evaluate reward models and select the best one for downstream performance.Source: §2.1 · p. 2

2.2 REWARD MODEL BENCHMARKS

  • RewardBench evaluates chat, hard chat, safety, and reasoning using preference pairs, adversarial examples, coding pairs, and safety pairs.The paper describes it as a crucial first step, while noting that further research is needed to understand how benchmark scores correlate with RLHF success.Source: §2.2 · p. 2
  • Table 1 compares PPE with existing work, and the paper identifies a gap between benchmark scores and downstream RLHF performance.The supplied text does not report the table’s cell values.Source: §2.2 · p. 3, Table 1 · p. 3

3 SOURCING GROUND TRUTH PREFERENCE LABELS

The paper argues that common ways of sourcing preference labels can introduce biases or produce response pairs unlike those encountered during RLHF. It therefore combines crowdsourced human preferences with correctness labels verified by existing benchmarks.

3 SOURCING GROUND TRUTH PREFERENCE LABELS

  • LLM-judge labels with manual verification may introduce preference bias, while synthetically perturbed rejected responses may create unrepresentative errors.The paper says such pairs do not reflect the response distribution reward models encounter when providing RLHF learning signals.Source: §3 · p. 3
  • The authors ground preference labels in crowdsourced prompts and responses with human votes, alongside benchmark responses with verifiable correctness checks.These are the two stated components of their ground-truth methodology.Source: §3 · p. 3
  • The 16,038 crowdsourced preference labels aggregate diverse preferences to mitigate individual label noise and avoid overfitting to one person’s preferences.The paper characterizes this approach as an unbiased estimate of real-world human preference.Source: §3 · p. 3
  • Verifiable correctness labels provide an objective signal intended to be naturally unbiased by response style.The sampled LLM responses and errors are meant to resemble rollouts or best-of-k exploration in RLHF, yielding more naturally occurring preference pairs.Source: §3 · p. 3

4 HUMAN PREFERENCE METRICS

The human-preference evaluation uses Chatbot Arena votes on pairwise comparisons between anonymized model responses and can be refreshed as preferences and available models change. It scores reward models through pairwise accuracy, aggregate ranking and win-rate correlations, and confidence-related metrics, including domain-specific evaluations.

4 HUMAN PREFERENCE METRICS

  • Chatbot Arena users vote on pairwise comparisons between responses from two anonymized, randomly selected models; the dataset covers 16,038 comparisons between 20 selected top models.The selected models were chosen for strong Arena performance and popularity, with an emphasis on models that had undergone RLHF.Source: §4 · p. 3
  • The human-preference set can be recollected to reflect current models and changing preferences, and updated data may mitigate leakage from the previous set.The paper presents refreshability as a way for the metric to remain current with fresh, relevant data.Source: §4 · p. 3

4.1 CURATION

  • Curation samples and filters Arena battles before subsampling 16,038 pairs for the human-preference dataset.The process begins with 50,000 battles weighted to represent selected models uniformly, then deduplicates, removes samples containing P.I.I. or potentially harmful conversations, and subsamples the remaining battles.Source: §4.1 · p. 4
  • The curated preference data includes instruction-following, hard, and math prompts, queries in over 121 languages, and preferences from 6,120 individuals.The listed prompt counts are 4,583 instruction-following, 5,195 hard, and 2,564 math prompts; categories may overlap.Source: §4.1 · p. 4

4.2 SCORING

  • Scoring includes pairwise accuracy against human labels, Spearman and Kendall ranking correlations, and row-wise Pearson correlation of model win-rate matrices.Accuracy excludes battles labeled as ties; the correlation metrics compare reward-model rankings or win-rate matrices with ground truth.Source: §4.2 · p. 4
  • Separability with Confidence Interval, Confidence Agreement, and Brier Score assess uncertainty and overconfidence against ground-truth rankings.The metrics follow those proposed in Arena-Hard-Auto.Source: §4.2 · p. 4
  • The scores can also be computed on prompt subsets to evaluate performance across seven domains, including math and instruction following.The paper expects strong reward models to score highly regardless of the selected domain.Source: §4.2 · p. 4

5 CORRECTNESS METRICS

The correctness metrics test whether reward models can distinguish correct from incorrect responses sampled from the same LLM distribution. The section describes how these responses are curated and evaluates reward models with best-of-K curves, ROC AUC, and pairwise accuracy.

5 CORRECTNESS METRICS

  • Correctness labels from verifiable benchmarks test whether reward models prefer correct responses among samples from the same LLM distribution, including similar or adversarial outputs and naturally occurring errors.The approach assumes expert humans would prefer correct answers over incorrect ones.Source: §5 · pp. 4–5

5.1 CURATION

  • Five correctness benchmarks cover general knowledge, mathematics, STEM, coding, and instruction following; responses are sampled across four models and filtered to retain prompts with meaningful correctness variation.The procedure samples 500 prompts per benchmark, repeating each 32 times; it removes rows with all-correct or all-incorrect responses and generally retains rows with 10%–90% correct responses.Source: §5.1 · p. 5

5.2 SCORING

  • The scoring metrics target reward-model robustness, granularity, and theoretical roof-line performance.Source: §5.2 · p. 5

5.2.1 BEST OF K CURVES

  • Best-of-K curves track the ground-truth score of the reward-selected response as K increases, against the ground-truth best-response curve; at K = 32, greater exploration also raises the chance of reward exploitation.The example figure shows average performance across benchmarks and MBPP Plus, where many reward models over-optimize.Source: §5.2.1 · pp. 5–6, Figure 2 · p. 6

5.2.2 AREA UNDER RECEIVER OPERATOR CHARACTERISTICS (ROC) CURVE

  • ROC AUC evaluates binary correctness classification after min-max normalizing reward scores within each response row.Source: §5.2.2 · p. 6

5.2.3 ACCURACY

  • Pairwise accuracy measures how often a reward model or LLM judge prefers the correct response in five sampled correct-versus-incorrect pairs per row.Source: §5.2.3 · p. 6

6 VALIDATING PPE ON POST-RLHF OUTCOMES

The paper tests whether PPE metrics relate to real-world human preferences for LLMs trained with different reward models. It trains a common base model with DPO and evaluates the resulting models in Chatbot Arena, collecting 12,190 human votes.

6 VALIDATING PPE ON POST-RLHF OUTCOMES

  • The validation experiment uses human preference rankings of RLHF-trained LLMs as a measure of the downstream performance of their reward models, while noting the experiment uses one base model and off-policy DPO.Source: §6 · p. 7

6.1 TRAINING PROCEDURE

  • Nine general-purpose reward models were selected, and each built a preference dataset from 8,000 prompts with 16 base-model responses per prompt, using its top-scoring response as chosen and a seeded rank-sampled response as rejected.The prompt set combined 7,000 human-preference prompts with 500 each from MMLU-Pro and MATH; the resulting 8,000 chosen/rejected pairs were used to train a DPO model per reward model.Source: §6.1 · pp. 7–8

6.2 EVALUATION ON REAL-WORLD HUMAN PREFERENCE

  • Blind Chatbot Arena evaluation collected 12,190 votes over six days, and the resulting Arena Scores were calculated with the Bradley-Terry model.The evaluation cohort included the trained models and three unaltered baselines; Table 3 marks the best non-base model score in bold.Source: §6.2 · p. 8, Table 3 · p. 8

7 STUDYING CORRELATION WITH DOWNSTREAM PERFORMANCE

The section compares offline reward-model metrics with post-RLHF human-preference rankings. Accuracy is the strongest predictor on human-preference data, while correlations vary across datasets and aggregation strategies.

7 STUDYING CORRELATION WITH DOWNSTREAM PERFORMANCE

  • On correctness metrics, mean performance across domains correlates well across metrics, with higher correlations for AUC and Accuracy; Math is the strongest individual domain.ROC AUC correlates more strongly across benchmarks, including benchmarks that are otherwise uncorrelated.Source: §7 · p. 8, Figure 3 · p. 9
  • Reward-model accuracy is the best predictor of fine-tuned LLM preference scores; Spearman and Kendall correlations are nearly zero.Row-wise Pearson Correlation, Confidence Agreement, and Separability show some correlative power but do not exceed accuracy.Source: §7 · p. 8, Figure 3 · p. 9
  • Human-preference accuracy correlates more strongly with downstream scores than correctness metrics, while style control slightly lowers human-preference correlations but leaves correctness correlations unchanged.The authors note that correctness and human preference do not necessarily align.Source: §7 · p. 9
  • For off-policy DPO, Loss, Max score, and End score may be more useful when the RLHF algorithm has higher over-optimization risk.The authors report these measures may be helpful in that setting.Source: §7 · p. 9
  • Lower-quantile aggregation increases downstream-score correlation in nearly every metric, and accuracy reaches 0.80 correlation at low quantile aggregation.The authors suggest this pattern reflects the need for reward models to remain robust across input distributions, since an LLM can exploit domain weaknesses during training.Source: §7 · p. 9, Figure 5 · p. 10

8 LIMITATIONS

The authors identify benchmark leakage as a possibility and limit their downstream claims to a small set of models trained with a specific base model and algorithm.

8.1 BENCHMARK LEAKAGE

  • Benchmark leakage remains a possibility, though the human-preference data can be refreshed and correctness datasets can be extended to new standard benchmarks.The authors describe these updates as mitigating factors, not eliminating leakage.Source: §8.1 · p. 10

8.2 LIMITS ON TESTING DOWNSTREAM PERFORMANCE

  • End-to-end evaluation covered only nine reward models, used DPO rather than PPO, and therefore does not establish outcomes for all models, base models, or RLHF algorithms.The authors note that DPO may differ in over-optimization or reward-model requirements and call for testing other RLHF methods.Source: §8.2 · p. 10

9 CONCLUSION

PPE links reward-model evaluations to post-RLHF human preference outcomes. The authors report that granular metrics and lower-bound measurements are predictive, with an overall 77% Pearson correlation.

9 CONCLUSION

  • 77% Pearson correlation with downstream performance was achieved, and granular measurements such as accuracy were the best predictors across the evaluated metrics.The results also suggest lower-bound performance may better indicate expected reward-model performance in the RLHF pipeline.Source: §9 · p. 10
  • PPE is a reward-model benchmark explicitly tied to post-RLHF outcomes based on real human preferences.The authors open-source the dataset-creation, experimental-validation, and reward-model evaluation code and methods.Source: §9 · p. 10

A.1 DETAILED SCORES FOR THE HUMAN PREFERENCE EVALUATION DATASET

The appendix catalogs human-preference evaluation results by prompt and language subsets, describes benchmark prompt modifications, and provides detailed benchmark and downstream-analysis materials.

A.1 DETAILED SCORES FOR THE HUMAN PREFERENCE EVALUATION DATASET

  • Tables report reward-model and LLM-judge performance on overall, hard, easy, instruction-following, coding, and math subsets of the human-preference dataset.Additional tables cover shorter-winner, similar-response, and English and non-English subsets, including language-specific results.Source: Table 4 · p. 14, Table 5 · p. 14, Table 6 · p. 15

A.2.1 SMALL BENCHMARK MODIFICATIONS

  • Verifiable-benchmark prompts were modified to encourage chain-of-thought responses, with further changes to MATH answer checking, MBPP-Plus sampling, and IFEval instruction emphasis.MATH uses symbolic equality checking; MBPP-Plus prompts request explanatory code comments and add standard test cases as examples.Source: §A.2.1 · p. 23

A.2.2 DETAILED SCORES

  • The detailed-score materials include tables for best-of-k and AUC performance across benchmarks and sample models, alongside figures comparing benchmark performance and correlations.Figure 8 describes correlations with post-RLHF human-preference Arena scores across quantile aggregations; all metrics except separability show higher correlation at lower quantiles.Source: Table 23 · p. 25, Table 24 · p. 25, Table 25 · p. 25

A.3 DPO CONFIGURATION

  • The appendix lists DPO configuration settings for training Meta-Llama-3.1-8B-Instruct, including a constant learning-rate schedule, batch size, sequence lengths, optimizer, and implementation.The listed implementation is TRL DPOTrainer, with AdamW and DeepSpeed ZeRO2.Source: §A.3 · p. 28

A.4 CROWDSOURCED HUMAN PREFERENCE VOTE DETAILS

  • A table presents participation and vote-distribution statistics for the crowdsourced human-preference labels.Source: Table 27 · p. 28

A.5 ADDITIONAL ANALYSIS ON DOWNSTREAM PERFORMANCE

  • The downstream analysis includes style-controlled post-DPO Arena performance and correlations between reward-model metrics and human-preference scores.The style control accounts for length and markdown; the appendix also compares RewardBench model-ranking correlations with style-controlled post-DPO rankings.Source: Table 28 · p. 29, Figure 9 · p. 29, Figure 10 · p. 29

Related papers

Loading 2410.14872v2…