PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward Reasoning Model

Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei

arXiv:2505.14674v1cs.CL

TL;DR

Reward models lack an adaptive way to spend test-time compute on complex judgments, limiting their ability to scale reward estimation effectively. RRMs reason before assigning rewards and learn this capability through reinforcement learning without explicit reasoning traces. The paper reports stronger reward-model benchmark performance across domains and adaptive use of test-time compute.

  • Problem

    Current reward models use nearly uniform compute across inputs, despite complex judgments sometimes requiring extensive reasoning.

    Source: §1 · p. 2

  • Method

    RRMs generate reasoning before rewards and use rule-based reinforcement learning to develop this capability without explicit reasoning traces.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    RRMs consistently outperform strong baselines across reasoning, general knowledge, safety, and human-preference alignment benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RRMs leverage test-time compute through parallel and sequential scaling and are evaluated in practical reward-guided applications.

    Source: §5 · p. 13

Abstract

from arXiv · show

Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing test-time compute to enhance reward model performance. In this work, we introduce Reward Reasoning Models (RRMs), which are specifically designed to execute a deliberate reasoning process before generating final rewards. Through chain-of-thought reasoning, RRMs leverage additional test-time compute for complex queries where appropriate rewards are not immediately apparent. To develop RRMs, we implement a reinforcement learning framework that fosters self-evolved reward reasoning capabilities without requiring explicit reasoning traces as training data. Experimental results demonstrate that RRMs achieve superior performance on reward modeling benchmarks across diverse domains. Notably, we show that RRMs can adaptively exploit test-time compute to further improve reward accuracy. The pretrained reward reasoning models are available at https://huggingface.co/Reward-Reasoning.

1 Introduction

Reward models support broad post-training alignment, but current approaches lack a way to allocate more test-time compute to complex reward judgments. RRMs reason before assigning rewards and are trained to develop this capability without explicit reasoning traces.

1 Introduction

  • Existing reward models apply nearly uniform compute across queries, limiting evaluation of responses needing nuanced or multi-step reasoning.The challenge matters because accurate, broadly applicable reward models are critical to post-training alignment.Source: §1 · p. 2
  • RRMs generate chain-of-thought reasoning before final rewards, enabling adaptive compute allocation for complex evaluations.The approach also introduces ELO and knockout strategies for applying RRMs to multiple responses.Source: §1 · p. 2
  • RRMs consistently outperform strong baselines across reasoning, general knowledge, safety, and human-preference alignment benchmarks.Experiments also examine best-of-N inference, post-training with RRM feedback, and adaptive test-time compute use.Source: §1 · p. 2
  • Reward Reasoning via Reinforcement Learning lets RRMs self-evolve reward reasoning without explicit reasoning traces as training data.The framework uses a rule-based reward environment.Source: §1 · pp. 2–3

2 Related Work

Prior reward-model work varies in both reward formulation and scoring scheme, while generative models offer flexible judgments but retain reliability concerns. Inference-time scaling methods adapt compute through parallel sampling or longer reasoning traces.

Reward Models

  • Reward models vary by numeric-only or generative formulation and by absolute scoring or discriminative comparison of responses.These dimensions characterize reward formulation and scoring scheme.Source: §2 · p. 3

Generative Reward Models

  • Generative reward models provide interpretable feedback across single-instance and multi-response evaluation, but may produce biased or hallucinated judgments.They can address open-ended tasks and support diverse inputs, including multimodal ones.Source: §2 · p. 3

Inference-Time Scaling

  • Inference-time scaling adjusts compute to input complexity through parallel aggregation or longer reasoning traces.The cited approaches include multi-sampling, reward-model-guided aggregation, and horizon-based scaling.Source: §2 · p. 3

3 Reward Reasoning Model

RRMs treat reward modeling as text completion: they reason over a query and two responses before issuing a preference judgment. Rule-based reinforcement learning develops this reasoning without explicit traces, while pairwise strategies extend reward assignment to multiple responses.

3.1 Input Representation

  • RRMs take a query and two responses, reason through evaluation criteria, and then output a preference judgment.The Qwen2-based decoder generates a thinking process followed by a final judgment; the input is limited to two candidates to reserve output length for reasoning.Source: §3.1 · pp. 3–4

3.2 Model Training with Reinforcement Learning

  • Reward Reasoning via Reinforcement Learning trains RRMs to self-evolve reasoning in a rule-based reward environment without explicit reasoning traces.The reward checks whether the model prefers the ground-truth response, rather than scoring its own output; the framework uses Deepseek-R1 distilled models and GRPO.Source: §3.2 · p. 4

3.3 Multi-Response Rewarding Strategies

  • ELO rates candidates from pairwise round-robin results, with optional matchup sampling to reduce its O(n^2) comparison cost.The resulting ratings can serve as RLHF rewards, and experiments report successful LLM post-training with these ratings.Source: §3.3 · p. 4
  • Knockout tournament advances pairwise winners through a bracket, requiring n − 1 comparisons and O(log(n)) sequential rounds.Its stated complexity is O(n), and experiments apply it to best-of-N sampling.Source: §3.3 · p. 5
  • Both multi-response strategies can use repeated pairwise samples and majority voting to produce comparison results.The paper describes this combination as using additional test-time compute to enhance assessment robustness.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate RRMs on reward-model benchmarks, reward-guided inference, and downstream post-training. They report strong benchmark performance and show that additional test-time computation can improve reward accuracy.

4.1 Training Details

  • The evaluation covers RewardBench and PandaLM Test, with human-annotated preference judgments, and uses about 420K preference pairs to train the RRMs.The training data combines Skywork-Reward, Tülu preferences, and synthesized pairs; the models use DeepSeek-R1-Distill-Qwen as their base.Source: §4.2.1 · p. 5, §4.1 · p. 5

4.2.2 Results

  • RRM-32B reaches 98.6 accuracy in RewardBench’s reasoning category, while RRMs perform competitively against strong baselines on RewardBench and PandaLM Test.The experiments also compare RRMs with DirectJudge models trained on the same data and report a substantial gap in reasoning.Source: §4.2.2 · p. 6, Table 1 · p. 6

4.3.2 Results

  • In reward-guided best-of-N inference, RRMs surpass all reported baselines, and majority voting improves nearly all evaluated subsets except RRM-7B on GPQA.Each query uses the same 32 candidate responses; the evaluation includes MMLU-Pro and GPQA domain analyses.Source: §4.3.2 · p. 7, Table 2 · p. 7
  • RRM-32B achieves state-of-the-art accuracy on MMLU-Pro, MATH, and GPQA in binary preference classification, with voting@5 reaching peak results across benchmarks.The comparison follows Frick et al.’s protocol and includes reward models and instruction-tuned LLMs.Source: §4.3.2 · pp. 7–8, Table 3 · p. 7

4.4.1 Reinforcement Learning with Unlabeled Data

  • RL post-training with RRM-32B feedback on unlabeled WebInstruct queries improves downstream MMLU-Pro and GPQA performance steadily during training.The method uses GRPO and ELO rewards derived from pairwise judgments among groups of eight responses.Source: §4.4.1 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

4.4.2 Direct Preference Optimization

  • DPO-trained Qwen2.5-7B models all outperform the original model on Arena-Hard, and the model trained with RRM-32B labels scores highest.Preference labels come from RRM-7B, RRM-32B, or GPT-4o.Source: §4.4.2 · pp. 8–9, Table 4 · p. 9

4.5.1 Parallel Scaling

  • More pairwise comparisons and majority voting improve MATH best-of-N performance, while ELO scoring outperforms knockout tournaments for both RRM sizes.Knockout tournaments perform only slightly worse and require O(n) comparisons; the setup uses eight generated candidates per question.Source: §4.5.1 · pp. 9–10, Figure 5 · p. 9, Table 5 · p. 10

4.5.2 Sequential Scaling

  • Longer thinking horizons improve RewardBench accuracy across 7B, 14B, and 32B RRMs, and the reported gains do not require larger models or additional inference passes.The experiments limit the thinking budget and use a small post-thinking budget to guard against gains from simply increasing output length.Source: §4.5.2 · p. 10, Figure 6 · p. 10

5 Conclusion

Reward Reasoning Models add explicit reasoning before reward assignment and develop that capability through rule-based-reward reinforcement learning without reasoning-trace supervision. The paper reports stronger judgment accuracy across domains, test-time scaling, and effectiveness in two practical applications.

5 Conclusion

  • RRMs achieve superior judgment accuracy across diverse domains and competitive performance on established benchmarks.The conclusion reports parallel and sequential test-time scaling and practical use in reward-guided best-of-N inference and post-training with RRM feedback.Source: §5 · p. 13
  • Rule-based-reward reinforcement learning enables RRMs to develop reward reasoning without explicit reasoning traces as supervision.Source: §5 · p. 13
  • RRMs incorporate explicit reasoning before assigning rewards and leverage test-time compute through parallel and sequential scaling.Source: §5 · p. 13

A RRM Implementation Details

The implementation details specify how RRMs compare two responses and how evaluation prompts guide judgments. The prompts set evaluation criteria, address potential biases, and prescribe structured reasoning and output formats.

A.1 Prompt Template

  • The RRM prompt directs step-by-step comparison of two responses using instruction adherence, usefulness, accuracy, harmlessness, and level of detail.It also asks the model to ignore response order, length, and style unless they affect helpfulness.Source: §A.1 · pp. 20–21
  • For PPE evaluation, GPT-4o is prompted following Liu et al. [41] to select the best response from candidates using several criteria.The paper describes this standardized evaluation as supporting fair comparison between reward modeling methods.Source: §A.1 · p. 20
  • The comparison prompt requires identifying query-specific criteria and their weights, analyzing responses, and returning the selected response in a specified format.Source: §A.1 · p. 21

A.2 Hyperparameters for Training RRM

The appendix identifies a table of training hyperparameters and states that the parameters were selected to support reinforcement learning and the development of reasoning capabilities.

A.2 Hyperparameters for Training RRM

  • Table 7 presents the hyperparameters used to train RRMs, which the authors say were selected to support reinforcement learning and reasoning-capability development.Source: §A.2 · p. 21, Table 7 · p. 22

B.1 Detailed Results on Subsets of MMLU-Pro and GPQA

Across the reported subset results, RRM-32B is superior or comparable to Skywork-Reward-Gemma-2-27B-v0.2 on MMLU-Pro and performs strongly across GPQA subsets. RRM-7B shows stronger performance in some STEM or technical categories, with variation across other categories.

B.1 Detailed Results on Subsets of MMLU-Pro and GPQA

  • RRM-7B outperforms Skywork-Reward-Gemma-2-27B-v0.2 in several STEM-related MMLU-Pro categories despite having significantly fewer parameters.Source: §B.1 · p. 22, Figure 9 · p. 22
  • RRM-32B is consistently superior or comparable to Skywork-Reward-Gemma-2-27B-v0.2 across MMLU-Pro subsets and maintains excellent performance across GPQA subsets.Source: Figure 9 · p. 22, §B.1 · p. 22, Figure 10 · p. 23
  • On GPQA subsets, RRM-7B is stronger in certain technical categories but occasionally lags behind Skywork-Reward-Gemma-2-27B-v0.2 in more general-knowledge areas.Source: §B.1 · p. 22, Figure 10 · p. 23

B.2 Go Further into Knockout Tournament

On MATH, RRM models outperform the compared reward-model and non-verifier selection strategies. Accuracy rises through knockout elimination rounds while requiring only O(n) pairwise comparisons.

B.2 Go Further into Knockout Tournament

  • The evaluation selects the best of eight Qwen2.5-Math-7B-Instruct responses per MATH question, comparing RRM judges with PRMs, voting, and an oracle.The reward models are RRM-7B and RRM-32B; PRM baselines are Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B.Source: §B.2 · p. 23
  • On MATH, RRM-7B surpasses all comparison methods, including voting@8 and PRM judges, while RRM-32B significantly outperforms PRM-based baselines and further narrows the gap toward oracle-level accuracy.Source: Table 8 · p. 23, §B.2 · p. 23
  • Accuracy improves consistently across knockout elimination rounds, with only O(n) pairwise comparisons.The authors describe this as a practical balance between performance gains and computational requirements.Source: §B.2 · pp. 23–24, Figure 11 · p. 24

C Post-thinking Token Length Distribution

The authors examine post-thinking response lengths to set a fixed response budget for sequential scaling experiments. They use the budget to distinguish effects of thinking duration from output verbosity.

C Post-thinking Token Length Distribution

  • The response budget is calibrated so performance changes can be attributed to deeper reasoning rather than more verbose outputs.Source: C Post-thinking Token Length… · p. 24
  • All RRM-32B responses on RewardBench use fewer than 100 tokens for the final judgment after thinking.Figure 12 shows the post-thinking token-length distribution.Source: C Post-thinking Token Length… · p. 24, Figure 12 · p. 24
  • A fixed 100-token post-thinking budget is used in sequential scaling experiments to prevent further reasoning during the response phase.The fixed budget is intended to keep performance differences attributable to thinking-phase length rather than output verbosity.Source: C Post-thinking Token Length… · p. 25

D Reasoning Pattern Analysis

Table 9 identifies the pattern groups and keywords applied in reasoning pattern analysis.

D Reasoning Pattern Analysis

Loading 2505.14674v1…