Source-linked AI summary
RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, Juanzi Li
TL;DR
Existing reward-model benchmarks can miss subtle content differences and style biases because they often contrast responses generated by models of different strengths. RM-Bench addresses this with same-model responses containing subtle errors and style-controlled variants, and evaluates nearly 40 reward models. It strongly correlates with policy-model performance, while state-of-the-art reward models fall below random-level accuracy under style interference, showing substantial room for improvement.
Problem
Existing reward-model benchmarks often use responses from models of varying power, limiting evaluation of subtle content differences and style bias.
Method
RM-BENCH uses same-model chosen and rejected responses with subtle errors, style-controlled variants, and policy-model evaluations to assess reward models.
Results
RM-BENCH strongly correlates with policy-model performance, while state-of-the-art reward models achieve only 46.6% accuracy under style interference, below the 50% random baseline.
Takeaways & Limitations
RM-BENCH provides a reliable reference for selecting reward models for language-model alignment, and DPO models show potential as reward models.
Takeaways & Limitations
RM-BENCH covers length and markdown bias but not all possible biases, including code-versus-explanation preferences and specific phrase preferences.
Abstract
from arXiv · showhide
Reward models are critical in techniques like Reinforcement Learning from Human Feedback (RLHF) and Inference Scaling Laws, where they guide language model alignment and select optimal responses. Despite their importance, existing reward model benchmarks often evaluate models by asking them to distinguish between responses generated by models of varying power. However, this approach fails to assess reward models on subtle but critical content changes and variations in style, resulting in a low correlation with policy model performance. To this end, we introduce RM-Bench, a novel benchmark designed to evaluate reward models based on their sensitivity to subtle content differences and resistance to style biases. Extensive experiments demonstrate that RM-Bench strongly correlates with policy model performance, making it a reliable reference for selecting reward models to align language models effectively. We evaluate nearly 40 reward models on RM-Bench. Our results reveal that even state-of-the-art models achieve an average performance of only 46.6%, which falls short of random-level accuracy (50%) when faced with style bias interference. These findings highlight the significant room for improvement in current reward models. Related code and data are available at https://github.com/THU-KEG/RM-Bench.
1 INTRODUCTION
Existing reward-model benchmarks often use responses from models of different strengths, missing subtle content changes and style bias. RM-BENCH addresses these gaps by using same-model responses with subtle errors and style-controlled variants, then evaluates their relationship to policy-model performance.
- Motivation: Reward-model benchmarks remain under-explored and should test sensitivity to subtle changes and robustness against stylistic bias.A one-word difference can change meaning, requiring attention to content quality rather than surface form.
- Limitations of Existing Benchmarks: Existing benchmarks commonly pair a stronger model’s better response with a weaker model’s worse response, confounding content quality with model-generation style.This design can encourage reward models to favor the style of powerful models and correlates poorly with aligned policy-model performance.
- RM-BENCH: RM-BENCH generates chosen and rejected responses with the same gpt-4o model, introducing subtle errors through jailbreaking or multi-sampling.This construction targets sensitivity to content differences while reducing variation attributable to the generating model.
- RM-BENCH: RM-BENCH uses style-controlled prompts to create concise, detailed, and markdown-formatted response variants for testing resistance to style bias.The benchmark separately varies stylistic presentation while preserving its focus on response quality.
- Validation: Extensive experiments show that RM-BENCH has high correlation with policy-model performance after PPO fine-tuning.The benchmark is designed to provide a more faithful reference for evaluating reward models than generation-strength comparisons alone.
- Findings: 69.5% accuracy from Nemotron-340B-Reward remains far from satisfactory against the 50% random-guessing baseline.The result indicates substantial room for improving reward-model performance on RM-BENCH.
2 PRELIMINARIES
This section introduces reward models as evaluators of language-model responses and describes multi-objective and DPO-based alternatives. It also explains how reward-model evaluation frames preference prediction as classification measured by accuracy.
- Reward Models: Reward models evaluate responses to prompts by producing reward signals, functioning as text classifiers in language-model settings.The reward signal is parameterized by the reward model and depends on the prompt-response pair.
- Reward Models: Preference training uses chosen and rejected response pairs, optimizing the model to assign higher rewards to chosen responses.This objective is intended to identify responses that better align with human preferences.
- Multi-Objective Reward Models: Multi-objective reward models provide multiple reward signals to represent dimensions such as correctness, readability, and verbosity.Each vector component captures a distinct aspect of response quality, enabling more nuanced evaluations.
- DPO Models: DPO models optimize the policy directly using implicit rewards derived from policy and reference-model probabilities, regularization, and a partition function.When comparing responses, the prompt-only partition function can be omitted; if no reference model is available, its probability is assumed to be 1.
- Reward Model Evaluation: Reward-model evaluation treats preference prediction as classification: a prediction is correct when the chosen response receives a higher reward than the rejected response.Accuracy is the evaluation metric, and multi-objective models use element-wise comparison of reward vectors.
3 RM-BENCH CONSTRUCTION
RM-BENCH constructs domain-specific prompt–response tuples with verified chosen and rejected answers, then varies response style to test reward models’ sensitivity to substance and resistance to stylistic bias.
- RM-BENCH covers Chat, Code, Math, and Safety, spanning open-domain conversation, reasoning tasks, and safety-critical scenarios.
- Each example pairs a prompt with chosen and rejected responses generated by the same powerful language model, while human annotators verify correctness.Style-controlled variants are also generated for both responses.
- Chat: Chat examples inject factual errors into gpt-4o responses through Many-Shot Jailbreak, followed by human verification of factual correctness and refusal behavior.
- Code & Math: Code and math examples sample multiple gpt-4o responses at temperature t = 1.0, selecting one correct and one incorrect answer verified by tests or ground truth.The retained datasets contain 228 code samples and 529 math samples.
- Safety: Safety examples distinguish benign alarming prompts that should receive informative answers from harmful prompts that should receive refusals.Rejected responses come from an over-cautious gpt-4o variant for benign prompts and an uncensored Llama-3.1-8B-Lexi-Uncensored-V2 model for harmful prompts.
- Style Control: The style-controlled dataset compares concise, detailed plain-text, and detailed Markdown responses so reward models can be evaluated independently of stylistic differences.For each prompt, the construction produces three chosen and three rejected responses across these style levels.
- Evaluation: The Style-Substance Evaluation Matrix arranges chosen-response styles by rows and rejected-response styles by columns, with diagonal cells matching styles and off-diagonal cells differing.Three accuracy metrics are derived from this 3×3 matrix.
4 EVALUATION RESULTS
RM-BENCH exposes substantial weaknesses in current reward models, especially under style bias and in mathematically or programmatically complex domains. Comparisons further show that DPO models can outperform sequence classifiers, while multi-objective models struggle to separate correctness from style outside safety.
- Overall Performance: 70.1% Average Accuracy and 46.6% Hard Accuracy show that even strong reward models perform poorly on RM-BENCH.Skywork-Reward-Llama-3.1-8B remains below the 50% random-guessing baseline on Hard Accuracy.
- Overall Performance: Hard Accuracy is significantly lower than Normal Accuracy, with most reward models failing to exceed the 50% random-level baseline.The pattern indicates that many reward models favor well-structured responses over stronger substantive content.
- Overall Performance: Math and code are the most challenging domains, where even average accuracy struggles to exceed 50%.Skywork-Reward-Llama-3.1-8B achieves only 28.4% Hard Accuracy in Math and 30.7% in Code.
- DPO Model vs. Sequence Classifier: DPO models outperform sequence classifiers trained on the same preference datasets.The comparison uses DPO models and sequence classifiers, with Table 4 reporting average accuracy on RM-BENCH.
- DPO Model vs. Sequence Classifier: Removing the reference model causes a significant performance drop, supporting its critical role in DPO reward signals.The authors hypothesize that the reference model scales the DPO model’s reward signal.
- Multi-Objective Reward Models: Nemotron-4-340B-Reward separates correctness for chosen and rejected responses only clearly in safety, not in math or code.In math and code, substantial overlap indicates difficulty disentangling correctness from other factors.
5 CORRELATION WITH POLICY MODEL
The section tests whether RM-BENCH performance tracks policy-model performance under style-controlled and downstream evaluations. RM-BENCH shows a moderate positive correlation with downstream policy performance and an association between hard accuracy and reduced style bias.
- 5.1 STYLE-CONTROLLED CORRELATION: RM-BENCH Hard Accuracy is evaluated against policy-model style control using Arena-Hard-Auto, whose score measures relative performance retention under style control.A higher style-control score indicates less stylistic bias.
- 5.1 STYLE-CONTROLLED CORRELATION: Higher RM-BENCH Hard Accuracy is associated with significant improvement in policy-model style-control score, indicating reduced style bias.The metric directly measures prioritization of substance over style.
- 5.2 DOWNSTREAM TASK CORRELATION: The downstream analysis covers math, code, and safety tasks, selecting reward-model metrics according to each task’s sensitivity to substance and style.Hard Accuracy is used for math and safety, while Normal Accuracy is used for code.
- 5.2 DOWNSTREAM TASK CORRELATION: Reward-model scores are standardized and compared with policy-model performance normalized against the tulu-2-13b base SFT model.The comparison is plotted in Figure 5.
- 5.2 DOWNSTREAM TASK CORRELATION: 0.55 Pearson correlation (p = 0.07) indicates a moderate positive relationship between RM-BENCH and downstream policy performance, versus RewardBench’s r = 0.21 (p = 0.51).The authors describe this as progress toward a better-correlated reward-model benchmark.
6 RELATED WORK
Reward models provide preference-based signals for aligning language models, but their evaluation remains important because surface styles can distort judgments of substantive quality.
- Reward Models in LLM era: Reward models act as proxies for human preferences and provide feedback that guides policy-model alignment training.They are typically built on pretrained language models with a classification head predicting response rewards.
- Reward Model Evaluation: Faithful reward-model benchmarks matter because they affect preference-alignment efficacy and fairness of performance evaluation.LLM-as-a-judge systems can be vulnerable to surface features such as text length rather than factuality.
7 CONCLUSION
RM-BENCH evaluates reward models for sensitivity to subtle content differences and resistance to style bias. Its experiments support its use for selecting reward models while exposing substantial remaining weaknesses.
- 7 CONCLUSION: RM-BENCH focuses reward-model evaluation on subtlety and style rather than only broad response-quality differences.The benchmark is presented as a reference for selecting reward models for language-model alignment.
- 7 CONCLUSION: Nearly 40 reward models were evaluated, and state-of-the-art models struggled to exceed random-level performance under style-bias interference.The conclusion also reports that DPO models outperform sequence-classification reward models.
A LIMITATIONS OF RM-BENCH
The limitations concern incomplete bias coverage, uncertain generalization of policy-model correlations, sensitive safety content, and possible bias from using gpt-4o-generated responses.
- Limited Coverage of Bias Types: RM-BENCH covers Length and Markdown bias but omits other biases, including preferences for code-only responses or phrases such as “think step by step.”The authors note that unexplored biases could enable benchmark hacking.
- Limited Correlation with Policy Models: The observed RM-BENCH correlation may not hold when policy models use different base models, training algorithms, or hyperparameters.The controlled evidence uses tulu-2-13b, PPO, and shared hyperparameters.
- Border Impacts: The Safety section exposes users to potentially offensive or sensitive content through rejected samples.The prompts originate from pre-existing benchmarks, with no concern about personally identifiable information.
- Potential Bias Introduced by GPT-4O: Because RM-BENCH largely uses gpt-4o responses, a reward model built on gpt-4o may prefer its own style.The authors plan to add responses from other language models to reduce single-model bias.
D THE SCALABILITY OF OUR DATA CONSTRUCTION METHOD
RM-BENCH’s construction pipeline is designed to scale to new language models and domains, while its evaluation links benchmark performance to policy-model outcomes across chat and reasoning tasks.
- The construction method can incorporate newly released language models by repeating the pipeline without model-specific requirements beyond generating text responses.
- New domains with automatically evaluable ground truth follow the Math & Code pipeline, whereas domains without ground truth require human correctness judgments.
- Higher RM-BENCH accuracy is associated with improved Alpaca Eval length-control scores, indicating less policy-model bias toward response length.
- Reward Bench shows an unsatisfactory policy-model correlation, with a Pearson correlation coefficient of 0.21 and a p-value of 0.51.
- On code and math reasoning tasks, higher RM-BENCH performance corresponds to greater Best-of-N accuracy improvement over greedy decoding.
J EXAMPLE DATA
The examples illustrate RM-BENCH data across chat, code, math, safety, and many-shot settings, including controlled response variants for evaluating style bias.
- RM-BENCH examples cover chat, code, math, safety-should-respond, safety-should-refuse, and many-shot cases.
- Each prompt is paired with chosen and rejected responses, each presented in concise, detailed, and markdown-formatted variants.
- The appendix includes individual examples from the code and math domains, plus examples from the general dataset and refusals-dangerous subset.
- The full-results table reports average accuracy by domain and difficulty, overall average accuracy, model types, and a 50% random-guessing baseline.
K DETAILED EVAL RESULTS
The detailed evaluation results are organized by domain, covering chat, math, code, safety-should-respond, and safety-should-refuse settings.
- Detailed RM-BENCH results are provided separately for the Chat and Math domains.
- The domain tables identify tested model types using icons for sequence classifiers, Direct Preference Optimization models, and custom classifiers.
- The appendix reports a dedicated set of detailed results for the Code domain.
- Safety evaluation is divided into Safety-Should-Respond and Safety-Should-Refuse domain results.