PaperlayerView exact revision on arXiv

Source-linked AI summary

RewardBench: Evaluating Reward Models for Language Modeling

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

arXiv:2403.13787v2cs.LG

TL;DR

Reward models are central to RLHF but have received relatively little focused evaluation, while existing preference test sets have known accuracy ceilings and limited validation coverage. RewardBench introduces a benchmark toolkit and structured datasets spanning chat, reasoning, and safety, finding varied reward-model behaviors and weaker, higher-variance generalization for DPO models.

  • Problem

    Reward models are central to RLHF yet receive far less evaluation than pretrained and post-RLHF models, with existing preference test sets limited by inter-annotator disagreement and unavailable validation sets.

  • Method

    RewardBench provides a common evaluation framework, released data, and structured prompt-chosen-rejected comparisons covering reward-model properties across chat, reasoning, safety, and instruction following.

  • Results

    DPO models fail to generalize to popular preference-data test sets and show higher performance variance, while RewardBench reveals diverse refusal, reasoning, scaling, and instruction-following characteristics across reward models.

  • Takeaways & Limitations

    RewardBench supports broader analysis of reward-model properties and provides a toolkit that can be expanded to audit specific aspects of the RLHF process.

  • Takeaways & Limitations

    The paper does not yet establish how RewardBench performance correlates with RLHF usefulness, leaving downstream validation as a crucial next step.

Abstract

from arXiv · show

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

1 Introduction

The paper addresses limited evaluation of reward models by introducing RewardBench, a toolkit and dataset for structured comparisons across capabilities and model types. Its evaluations expose generalization and performance differences, including weaknesses in DPO models and subtle instruction-following tests.

  • Motivation: Reward models receive less evaluation attention than pretrained and post-RLHF policy models, while many newer preference datasets lack public test sets.Earlier validation sets also have accuracy ceilings of 60–70% because of inter-annotator disagreement.
  • Contribution: RewardBench is introduced as a toolkit and common framework for benchmarking reward models across diverse capabilities and architectures.The released code supports visualization, training, and analysis, while the evaluation data include text-score pairs.
  • Findings: DPO models fail to generalize to popular preference-data test sets and show higher performance variance than classifier-based reward models.The comparison spans multiple datasets and contrasts DPO with classifier-based approaches.
  • Findings: RewardBench charts reward-model scaling, refusal propensity, reasoning capabilities, and other properties across the current model landscape.The leaderboard evaluates more than 80 openly available models, including classifiers and DPO-trained chat models.
  • Findings: Existing preference-data test sets contain subtle instruction pairs whose rejected responses may look high quality while answering the wrong prompt.These cases are used to showcase common reward-model pitfalls.

2 Related Works

Prior work evaluates preference-tuned models indirectly and has also examined reward-model behaviors directly. RewardBench situates these approaches within broader RLHF applications and uses a prompt-completion scoring setup.

  • RLHF applications: RLHF has improved chat models and has been applied to summarization, question answering, image models, and general instruction following.The related work frames RLHF as a method for incorporating human feedback into multiple model capabilities.
  • RLHF applications: RLHF commonly trains a reward model on preference data before learning a policy with reinforcement-learning algorithms such as PPO.Preferences may concern broad properties such as helpfulness or harmlessness or more fine-grained aspects.
  • Evaluation approaches: Downstream evaluations such as AlpacaFarm, MT-Bench, and Chatbot Arena assess preference-tuned models through model-simulated or crowdsourced judgments, indirectly evaluating reward models.AlpacaFarm reports a model win-rate against a reference model.
  • Evaluation approaches: Direct reward-model analyses have studied reward-length correlations, reward inconsistencies, and performance under distribution shift.These studies use training-dynamics analysis, contrasting instructions, and shifted distributions.
  • RewardBench setup: RewardBench scores each prompt by independently comparing reward-model scores for chosen and rejected completions.Figure 1 depicts the chosen and rejected completion scoring procedure.

3 Background

The background describes reward-model training as pairwise preference classification and contrasts explicit reward models with DPO's implicit policy-based reward. Inference selects the completion receiving the higher reward score.

  • Reward modeling: Human labelers choose between completions for prompts, producing preference data used to train reward models.Prompts are denoted x and candidate completions y_i.
  • Reward modeling: Reward models estimate pairwise human preference probabilities using a Bradley-Terry model over completion rewards.The preferred completion receives probability proportional to the exponential of its reward.
  • Reward modeling: Reward-model parameters are fit with a maximum-likelihood pairwise loss, commonly implemented by attaching or replacing the language model's final layer with a linear layer.At inference, the model returns a scalar reward for each prompt-completion pair.
  • Reward modeling: A completion wins when its reward r(x, y1) exceeds the competing completion's reward r(x, y2).The pairwise comparison converts scalar scores into a preference decision.
  • Direct Preference Optimization: DPO avoids a separate reward model by reparameterizing the preference reward using the trained policy, base-model probabilities, regularization constant β, and partition function Z(x).The implicit reward is computed from policy-model probabilities rather than a separately trained reward model.

4 The REWARDBENCH Benchmark

REWARDBENCH evaluates reward models across chat, instruction following, coding, safety, reasoning, and prior preference datasets using prompt–chosen–rejected comparisons. It scores whether models rank the chosen response higher, combining weighted section accuracies into one benchmark score.

  • Dataset design: REWARDBENCH combines existing prompt-completion pairs with newly curated data covering chat, instruction following, coding, safety, and other evaluation dimensions.
  • Dataset sections: The benchmark has five sections: Chat, Chat Hard, Safety, Reasoning, and Prior Sets.Chat Hard targets trick questions and subtle instruction differences, while Prior Sets contains established preference datasets.
  • Safety: Safety data tests refusal of dangerous content alongside avoidance of incorrect refusals to similar trigger words.
  • Reasoning: Reasoning data pairs correct code with buggy code and reference answers with incorrect model generations.
  • Scoring: 50% is the random-model baseline: each trio is classified as a win when the chosen score exceeds the rejected score, then section scores are weighted into the final result.Prior Sets receives half the weight of the other sections, while Reasoning reweights PRM-Math and Prior Sets averages its subsets equally.

5 Evaluation Results

RewardBench evaluates reward models across model types, scales, and challenging chat, reasoning, and safety subsets. Results show strong but uneven performance: scale and recent bases help, while DPO models vary and key subsets remain difficult.

  • Overall results: ArmoRM-Llama3-8B-v0.1 scores 89 and is state-of-the-art, while large models and Llama 3 models alone perform highly on Chat Hard and Reasoning.Starling-RM-34B scores 81.4 and Tulu-2-DPO-70B scores 76.1 across the evaluated models.
  • Scaling and base models: Scaling improves DPO reward-model performance, although Qwen 1.5 shows less monotonic improvement, likely because of out-of-distribution generalization challenges.The scaling result is reported for fixed-dataset Tulu and Qwen-Chat model groups.
  • Limits: Some subsets approach 100% accuracy, but Chat Hard and Reasoning exhibit low ceilings while many Safety subsets show high variability.Low ceilings identify areas for extending preference data and reward-model methods, whereas high variability indicates a need for converged best practices.
  • Chat Hard: Chat Hard tests subtle subject and context changes, and many reward models struggle to distinguish similarly styled answers addressing different prompts.Only a few models obtain strong Chat Hard results, while some models perform below random.
  • Reasoning: Reasoning performance spans 35% to 97% accuracy, revealing broad variation in precise classification of code examples differing by only one or two tokens.The reasoning section has the widest smooth variation among evaluated categories.
  • Safety: Safety evaluation exposes three refusal modes: balanced behavior, refusing nearly everything, or complying nearly everywhere.Top models can prefer refusals for sensitive prompts while responding to prompts containing misleading trigger words; other models show false refusals or false compliance.
  • Limits: Prior preference test sets have substantial limitations because new RLHF datasets often lack public validation sets and existing sets can have low accuracy ceilings.Some top models were trained on prior-set training splits, complicating interpretation of their strong scores.

6 Conclusion

RewardBench maps performance characteristics across current reward models to improve understanding of RLHF, while identifying reproducibility and downstream-use questions for future work.

  • RewardBench shows varied performance characteristics across reward models to improve understanding of RLHF.
  • The benchmark can be expanded with custom data to audit specific properties of the RLHF process.
  • Whether RewardBench scores correlate with downstream RLHF usefulness remains unresolved, with best-of-N and PPO experiments underway.
  • Scores from private language-model providers appear on the public leaderboard but are excluded from the paper because they are not reproducible.

Checklist

The checklist reports dataset, code, licensing, reproducibility, ethics, and resource disclosures, including publicly available evaluation assets and low-risk data handling.

  • The paper provides code, data, and experiment-reproduction instructions through the repository and linked evaluation assets.
  • Compute resources are reported in Appendix C, while training details and error bars are marked not applicable.
  • The datasets are permissively licensed, and copied code uses licenses that do not require additional attribution.
  • The data was generated by an LLM, the team, or previously released narrow benchmarks, so additional crowdsourcing was not used.
  • The paper reports that personally identifiable information and offensive-content risks are low and discusses them, particularly for the Safety section.

A Limitations & Broader Impacts

RewardBench is constrained by limited human preference data, possible benchmark biases and contamination, unresolved downstream correlation, and exposure to sensitive rejected content.

  • Limitations: RewardBench generally relies on semi-automatic chosen-rejected pairs because human preference data is unavailable, although specific subsets are manually validated.
  • Limitations: Reasoning-domain formats may introduce spurious correlations that create possible biases in humans and models.
  • Limitations: Benchmark results may not correlate with downstream training, and models directly trained on AlpacaEval or MT-Bench data may cause contamination.
  • Broader Impacts: Safety rejected samples expose users to potentially offensive or sensitive text, so users should use the data at their own risk.

B Discussions

The discussion examines length control, DPO evaluation, reference-model dependence, generative reward modeling, normative values, and safety-system evaluation boundaries.

  • Evaluating Length Bias: RewardBench controls length bias by making chosen responses similar in length to or shorter than rejected responses.
  • DPO Models vs Classifiers: DPO models can be evaluated as reward models using inference code without gradient propagation, but missing reference-model documentation can make benchmarks unclear.
  • DPO Models vs Classifiers: Using a wrong reference model reduced DPO reward-model performance to levels similar to the random baseline.
  • DPO Models vs Classifiers: DPO and classifier reward models differ in regularization and typical training epochs, leaving best training practices unresolved.
  • Generative Reward Modeling: Generative reward models use language-model generations to judge answers, but the best classifier reward models outperform the best generative reward models.
  • Values Represented in Reward Models: Reward models encode human preferences or values in the final policy, enabling study of whose values are embedded in RLHF.
  • Safety In or After RLHF: Systems with separate safety classifiers should evaluate each classifier on the benchmark sections it controls rather than comparing the whole system directly.

C Compute Usage

The evaluation primarily used NVIDIA A100 GPUs, with substantial per-model and full-suite compute requirements. Additional datasets were supported by the methodology but excluded from scores because of noise, unclear instructions, or contamination concerns.

  • 75 models were evaluated, with each model requiring about 12 hours on 16-bit quantization.
  • Re-running the complete RewardBench evaluation suite would require approximately 1000 A100 hours.
  • SafeRLHF and MT-Bench labels were supported methodologically but excluded from analysis scores because of noisy results, unclear instructions, or contamination concerns.

E Additional Results

Additional results document leaderboard and category-level evaluations, score distributions, and differences between reward-model output behaviors. RewardBench provides broader and more discriminative evaluation ranges than prior preference test sets, while some custom classifiers require different scoring treatment.

  • Tabulated results: Tables 9–14 report leaderboard results and performance breakdowns for Chat, Chat Hard, Safety, Reasoning, and Prior Sets categories.
  • Score distributions: RewardBench subsets show substantially greater score variance and range than existing preference test sets, making similar models easier to differentiate.
  • Model reward distributions: DPO models produce large negative scores from summed generation log probabilities, whereas classifier rewards are conceptually expected to approximate a unit Gaussian but generally do not.
  • Scoring caveat: Custom classifiers such as PairRM and SteamSHP are omitted because scoring two responses jointly does not map directly to the other models’ scoring setup.

F Dataset Details

RewardBench contains manually verified prompt-chosen-rejected trios spanning instruction following, refusal behavior, and prior preference-test settings. Its curation emphasizes subtle errors, difficult comparisons, and representative evaluation data across multiple sources.

  • Dataset scope: RewardBench contains 2958 prompts, all drawn from single-turn instruction-following tasks in the primary dataset.
  • Chat subsets: AlpacaEval subsets compare manually verified responses from models with different capabilities, including GPT4-Turbo versus Alpaca 7B and Tülu 2 70B versus Davinci003.
  • Chat subsets: MT-Bench subsets use manually verified pairs selected by score differences, while excluding second-turn data because conversations would differ beyond the final prompt.
  • Instruction-following challenges: LLMBar Adversarial creates rejected responses from related, similar, or intentionally unhelpful instructions, with 134 Neighbor, 92 GPTInst, 47 GPTOut, and 46 Manual examples.
  • Refusal behavior: Safety subsets test whether reward models prefer appropriate refusals over harmful responses or distinguish safe prompts that should receive answers from those requiring refusal.
  • Prior Sets: Prior Sets were selected for representativeness and future usefulness, with SHP filtered to increase rating margins and MT-Bench excluded from formal release concerns about contamination.
  • Curation: Curators were instructed not to skip hard choices, and examples where both responses were wrong were commonly rejected during processing.

I.4 Refusals data

The refusals data was assembled by manually selecting response pairs for prompts that should either receive a refusal or a compliant answer. Examples include harmful-content prompts, benign requests with trigger words, and ordinary tasks where exaggerated refusals are incorrect.

  • Refusal construction: The curation sought refusals as chosen responses and harmful or unsafe text as rejected responses, using dolphin-2.0-mistral-7b after Zephyr-β proved inconsistent.
  • Should Respond: XSTest Should Respond pairs safe-to-answer prompts containing seemingly unsafe trigger words with compliant and erroneously refusing responses.
  • Filtering: Several examples were removed or rejected because of refusal overlap, ranking errors, hallucinations, or factual inaccuracies in both responses.
  • Included examples: Included XSTest Should Respond examples illustrate both prompts that should receive answers and odd-looking prompts that remain answerable.
  • Should Respond: The dataset includes ordinary requests such as fictional violence, persona adoption, software explanations, and cookie recipes to test whether models avoid false refusals.
Loading 2403.13787v2…