PaperlayerView exact revision on arXiv

Source-linked AI summary

RM-R1: Reward Modeling as Reasoning

Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, Heng Ji

arXiv:2505.02387v4cs.CLcs.AIcs.LG

TL;DR

Accurate, grounded reward modeling often requires reasoning over evaluation criteria and trade-offs. RM-R1 combines task-specific rubrics or solutions with reasoning-focused training and achieves state-of-the-art average performance across three benchmarks, outperforming larger and commercial models by up to 4.9%.

  • Problem

    Accurate, grounded reward modeling often requires jointly reasoning and assigning rewards because preference judgments involve latent evaluation criteria and trade-offs.

    Source: §1 · p. 1

  • Method

    RM-R1 categorizes tasks, generates tailored chat rubrics or solves reasoning problems before evaluating responses, and uses reasoning distillation before reinforcement learning with verifiable rewards.

    Source: §1 · p. 2

  • Results

    Up to 4.9%: RM-R1 achieves state-of-the-art average performance across RewardBench, RM-Bench, and RMB, outperforming 70B, 340B, GPT-4o, and Claude models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablations identify task-type categorization, bootstrapping from high-quality reasoning traces, and RL fine-tuning as indispensable to RM-R1.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning a score or a judgment. Inspired by recent advances of long chain-of-thought on reasoning-intensive tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's interpretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (ReasRMs), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism -- self-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks on average, outperforming much larger open-weight models (e.g., INF-ORM-Llama3.1-70B) and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful ReasRM training.

1 INTRODUCTION

RM-R1 treats reward modeling as a reasoning task, using a staged training pipeline and task-specific evaluation to produce interpretable reward judgments. The paper reports strong benchmark performance and studies the effects of its training choices.

Introduction

  • Preference judgments require reasoning about latent evaluation criteria, trade-offs, and potential consequences, but scalar RMs are opaque and generative RM reasoning can be superficial.The paper argues that reliable reward modeling must jointly reason and assign rewards.Source: §1 · pp. 1–2
  • The authors find that RLVR alone does not fully elicit reward-model reasoning, while plain chain-of-thought misses fine-grained distinctions across question types.They also fine-tune existing reasoning models with RLVR without an additional distillation stage.Source: §1 · p. 2
  • RM-R1’s pipeline distills high-quality reasoning before RLVR, then uses Chain-of-Rubrics to tailor evaluation to chat or reasoning tasks.For chat, the model generates question-specific rubrics and justifications; for reasoning, it solves the problem before judging responses, prioritizing correctness.Source: §1 · p. 2
  • On average, RM-R1 achieves state-of-the-art performance across RewardBench, RM-Bench, and RMB, outperforming 70B, 340B, GPT-4o, and Claude models by up to 4.9%.The paper also reports coherent, interpretable reasoning traces from RM-R1.Source: §1 · p. 2
  • The paper studies training-recipe ablations, scaling, non-reasoning baselines, case studies, and training dynamics to analyze RM-R1 and its development.It also presents a systematic empirical study of how different training strategies affect final reward-model performance.Source: §1 · pp. 2–3

2 RM-R1

RM-R1 trains a generative reward model to produce reasoning and preference judgments through a two-stage pipeline of reasoning distillation followed by reinforcement learning. Its chain-of-rubrics rollout tailors evaluation to chat or reasoning tasks.

2 RM-R1

  • RM-R1 has two training stages: distillation of reasoning traces, followed by reinforcement learning to address overfitting and improve generalization.The first stage starts from an instruction-tuned model; the second produces the final RM-R1.Source: §2 · p. 3

2.1 TASK DEFINITION

  • A generative reward model maps a prompt and two responses to a textual judgment containing its prediction of which response is preferred.The task is defined over preference data with a ground-truth label identifying the preferred response.Source: §2.1 · pp. 3–4

2.2 REASONING DISTILLATION FOR REWARD MODELING

  • Reasoning distillation uses synthesized long traces to train instruction-tuned models whose judgments may otherwise be inconsistent, with oracle models generating justifications for preferred responses.The distillation objective maximizes the likelihood of the desired reasoning trace and response choice given the prompt.Source: §2.2 · p. 4

2.3 RL TRAINING

  • RL treats the reward model as a policy generating reasoning and judgments, and uses GRPO with a reference-model KL term to optimize it (Ouyang et al., 2022; Shao et al., 2024).The reference model is the pre-RL checkpoint, which may be an off-the-shelf or distilled model; RL is motivated by distillation’s overfitting and limits on generalizing reasoning (Chu et al., 2025).Source: §2.3 · p. 4

2.3.1 CHAIN-OF-RUBRICS (COR) ROLLOUT

  • In the CoR rollout, RM-R1 classifies samples as Chat or Reasoning, solves reasoning prompts before comparing responses, and generates grading rubrics for chat quality and safety.Chat rubrics can target textual qualities such as politeness, while reasoning evaluations emphasize logical coherence and answer correctness.Source: §2.3.1 · pp. 4–5

2.3.2 REWARD DESIGN

  • RM-R1’s RL reward focuses on correctness; adding a format reward did not significantly change task performance because distilled models already follow instructions and format responses properly.The correctness-based design follows prior approaches (Shao et al., 2024; Li et al., 2025), while rule-based rewards have supported reasoning (Guo et al., 2025).Source: §2.3.2 · p. 5

3 EXPERIMENTS

RM-R1 surpasses leading reward models on average, including models much larger than its 14B version, and its reasoning-oriented training produces strong RM-Bench results. The evaluation covers RewardBench, RM-Bench, and RMB, using preference, code, and math training data and comparing ScalarRMs, GenRMs, and ReasRMs.

3.1 EXPERIMENTAL SETUP

  • RM-R1 uses a reasoning-oriented pipeline across RewardBench, RM-Bench, and RMB, with ScalarRMs, GenRMs, and ReasRMs as baseline categories.Training uses a cleaned subset of Skywork Reward Preference 80K, 8K Code-Preference-Pairs examples, and the full Math-DPO-10K dataset.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • On average, RM-R1-DEEPSEEK-DISTILLED-QWEN-14B surpasses leading RMs including INF-ORM-Llama3.1-70B, Nemotron-4-340B-Reward, and GPT-4o; its 32B models extend the lead.RM-R1 also outperforms top-tier ScalarRMs, while prior critique-based GenRMs generally lagged them. The evaluation finds that larger models frequently underperform smaller ones on RewardBench, motivating more systematic RM assessment.Source: §3.2 · pp. 6–7, Table 1 · p. 6
  • Up to 8.7%, RM-R1 exceeds the RM-Bench top baseline; its 32B DeepSeek-distilled model reaches 91.8% math and 74.1% code accuracy.The previous best models scored 73% in math and 63% in code. The 32B model establishes a new state of the art on RM-Bench, while RM-R1-QWEN-INSTRUCT-14B surpasses the five-times-larger Self-taught-evaluator-llama-3.1-70B; the DeepSeek-distilled model also has the strongest reasoning performance among released models on RewardBench.Source: §3.2 · p. 7
  • RM-R1 Instruct models reach competitive performance using 8.7K distillation examples, compared with 800K examples used to train DeepSeek-Distilled.The paper attributes RM-R1's gains to its training methodology and model scaling; prior critique-based methods relied on rejection sampling and unstructured self-generated chain-of-thought reasoning.Source: §3.2 · pp. 6–7

4 ANALYSIS

The analysis finds that reasoning-based training improves reward-model performance, with gains shaped by distillation, rubric design, model scale, and inference compute. It also shows more stable warm-start training and more interpretable, content-grounded judgments.

4.1 TRAINING RECIPES

  • Distillation gives the strongest training-recipe results; CoR rubrics and query categorization improve performance, while RL alone is insufficient.The reported gains from rubrics are especially notable for chat and safety, and query categorization notably improves reasoning performance.Source: §4.1 · p. 7

4.3 EFFECTIVENESS OF REASONING TRAINING

  • Reasoning-based models consistently outperform SFT-only counterparts under equal-data comparisons, while distillation alone also yields gains.Table 3 reports results across RewardBench, RM-Bench, and RMB.Source: Table 3 · p. 8, §4.3 · p. 9

4.2.1 MODEL SIZES

  • Across RM-R1 model sizes, larger models achieve better absolute post-training performance and consistently greater gains over original performance, supporting a scaling law; some ScalarRMs show no scaling advantage.Relative improvement (%) follows an approximately linear trend with model size; linear regression extrapolates to hypothetical scales of 3B and 72B.Source: §4.2.1 · p. 8, §4.2 · p. 8

4.2.2 INFERENCE-TIME COMPUTATION

  • Increasing inference budgets from 512 to 8192 tokens improves average performance across three benchmarks when training rollout budgets are matched.The study uses DeepSeek-R1-Distill-Qwen-14B and matches each training rollout limit to its inference token budget.Source: §4.2.2 · p. 8, Figure 4 · p. 8

4.4 TRAINING DYNAMICS

  • Warm-start RL refines reasoning more stably, while cold-start RL increases response length but becomes unstable near the end with a sharp reward drop.Warm-start reward rises smoothly; cold-start instability may indicate overfitting.Source: §4.4 · p. 9, Figure 5 · p. 9

4.5 CASE STUDY

  • RM-R1 produces interpretable judgments with question-specific rubrics and evaluates response content faithfully, unlike cold-start RL’s focus on superficial features.Its rubric generation prioritizes accuracy for medical questions, and its structured evaluations identify factual inaccuracies.Source: §4.5 · pp. 9–10, Table 4 · p. 10

5 CONCLUSION AND FUTURE WORK

RM-R1 frames reward modeling as reasoning, generating explicit rubrics and rationales to produce interpretable judgments. It matched or surpassed commercial and open-source reward models across three public benchmarks, while the analysis identifies essential training components and future directions.

5 CONCLUSION AND FUTURE WORK

  • Across three public benchmarks, RM-R1 matched or surpassed commercial and open-source reward models while producing more interpretable judgments.The family generates explicit chains of rubrics and rationales and scales with model size and inference compute.Source: §5 · p. 10
  • Task-type categorization, bootstrapping from high-quality reasoning traces, and reinforcement-learning fine-tuning were all indispensable to RM-R1.Qualitative analyses found that it prioritizes high-impact rubrics, follows its criteria faithfully, and justifies judgments coherently.Source: §5 · p. 10
  • Future work includes using active learning to query human preferences when the current rubric set is insufficient for a new preference sample.The paper also proposes extending reward modeling to multimodal and agentic scenarios.Source: §5 · p. 10

ETHICS STATEMENT

RM-R1 is presented as fundamental research conducted on public benchmarks, without human subjects or sensitive/private data, and with no foreseeable ethical risks.

ETHICS STATEMENT

  • The authors report no foreseeable ethical risks, citing publicly available benchmarks, no human subjects or sensitive or private data, and a general optimization pipeline.They state that the work raises no concerns about fairness, bias, discrimination, privacy, or security and complies with research integrity standards.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT … F.1 BENCHMARKS

The appendix documents reproducibility resources, reasoning-chain generation and GRPO details, and the scope of three evaluation benchmarks, alongside related-work context and an LLM-use disclosure.

REPRODUCIBILITY STATEMENT

  • The authors provide a repository with training and evaluation materials, versioned model and service configurations, and configurations for deviations and ablations.The repository includes distillation-data curation and training, RL training, prompts, chain-of-rubrics implementation details, and evaluation scripts for RewardBench, RMB, and RM-Bench. Experimental setup and protocols are detailed in Sections 3, 4, and F.1; hyperparameters, rollout settings, and compute requirements are in Section G.Source: REPRODUCIBILITY STATEMENT · p. 11

B RELATED WORK

  • DeepSeek-GRM and JudgeLRM study reasoning models as generative reward models, focusing on inference-time computation; RM-R1 provides the first systematic empirical comparison of different reward-model training paradigms.Source: B RELATED WORK · p. 16
  • Most RLHF implementations still treat reward modeling and reasoning as separate stages, despite the emergence of verifiable supervision.DeepSeek-R1 (Guo et al., 2025) uses self-verification during RLHF to reward correct reasoning steps rather than only final-answer quality.Source: B RELATED WORK · p. 16

C USER PROMPT FOR DEEPSEEK-DISTILLED REASONING MODELS

D DETAILS OF REASONING CHAIN GENERATION

  • Claude-generated traces are corrected by O3, and about 12% of training data—slightly fewer than 9K examples—is selected for distillation before RL.Approximately 25% of initial Claude-3.7-Sonnet traces are incorrect, primarily on harder chat tasks; OpenAI-O3 receives the prompt, incorrect trace, and correct final answer to generate a corrected trace. The order reflects qualitative observations: Claude handles easier tasks and safety attention well, while O3 performs better on harder tasks but tends to overemphasize helpfulness over safety.Source: D DETAILS OF REASONING CHAIN GENERATION · p. 17

E GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

  • GRPO uses the average reward of multiple sampled outputs for a prompt as its baseline and avoids an additional value-function approximation.Its objective balances task-specific loss against KL divergence, and its advantage is computed from the rewards of responses within each group.Source: E GROUP RELATIVE POLICY OPTIMIZATION… · p. 17

F.1 BENCHMARKS

  • The three benchmarks cover preference ranking, subtle response differences and style robustness, and comprehensive helpfulness and harmlessness evaluation.RewardBench has 358 chat, 456 chat-hard, 740 reasoning, and 1,431 safety samples. RM-Bench has 129 Chat, 441 Safety, 529 Math, and 228 Code samples, with three difficulty-varying prompts per sample; it is described as the most reasoning-intensive benchmark considered. RMB has 25,845 instances across 37 helpfulness-alignment and 12 harmlessness scenarios, and supports pairwise and Best-of-N evaluation.Source: §F.1 · p. 17

F.2 PREFERENCE DATASETS · F.3 BASELINES

The paper trains on cleaned preference datasets for chat, safety, math, and code, and compares RM-R1 with scalar, generative, and reasoning-enhanced reward models.

F.2 PREFERENCE DATASETS

  • About 30% of Skywork Reward Preference 80K comes from a spurious magpie_ultra subset, which the authors exclude from training.Rejected magpie_ultra responses consistently contain “<im_start>” and are typically multi-turn, unlike accepted responses; the retained Skywork data is the cleaned subset. Skywork Reward Preference 80K is described as spanning chat, safety, math, and code (Liu et al., 2024).Source: §F.2 · p. 18
  • The authors subsample 8K examples from Code-Preference-Pairs, a coding preference dataset built by introducing and manipulating code bugs.Manipulations include swapping broken and corrected versions and removing error comments.Source: §F.2 · p. 18
  • The authors use the full Math-DPO-10K stepwise preference dataset for mathematical reasoning (Lai et al., 2024).Global training-dataset statistics are summarized in Table 5; the supplied passage does not report its cell values.Source: §F.2 · p. 18, Table 5 · p. 18

F.3 BASELINES

  • ScalarRMs predict preferences from single numeric scores without explicit reasoning traces, and can struggle with interpretability and fine-grained reasoning.Examples include Eurus-RM (Yuan et al., 2025), Internlm2 (Cai et al., 2024), SteerLM-RM (Wang et al., 2024c), and Skywork-RM (Liu et al., 2024).Source: §F.3 · p. 18
  • GenRMs provide free-form textual judgments, often without further training or intermediate reasoning, and use natural-language rationales to make evaluations more interpretable.Examples include LLM-as-a-Judge (Zheng et al., 2023), GPT-4o (Achiam et al., 2023; Hurst et al., 2024), and Skywork-Critic.Source: §F.3 · p. 18
  • ReasRMs incorporate reasoning before final judgments, often through critiques or chain-of-thought training.Examples include JudgeLRM (Chen et al., 2025), Critique-RM (Yu et al., 2025), DeepSeek-GRM (Liu et al., 2025b), Self-taught Evaluators (Wang et al., 2024b), and RM-R1; the passage associates them with rigorous reasoning, safety evaluation, and nuanced preference judgments.Source: §F.3 · pp. 18–19

G IMPLEMENTATION DETAILS · H FULL EXPERIMENT RESULT · I SUPPLEMENTARY INFORMATION FOR SECTION 4

The section specifies RM-R1’s training framework, data use, and implementation settings, and identifies tables containing full results for RewardBench, RM-Bench, and RMB.

G IMPLEMENTATION DETAILS

  • The training framework uses VERL and OpenRLHF; Instruct models use 8.7k distillation examples and 64k RLVR examples, while Deepseek-Distilled models use the full RLVR data.Source: G IMPLEMENTATION DETAILS · p. 19
  • Distillation uses OpenRLHF’s SFTTrainer for one epoch with batch size 128 and micro-batch size 1, plus gradient checkpointing, FlashAttention, Adam offloading, and size-specific learning rates.Learning rates are 5e−6, 3e−6, and 2e−6 for 7B, 14B, and 32B models, respectively.Source: G IMPLEMENTATION DETAILS · p. 19
  • RLVR uses VERL for GRPO with batch size 1024, mini-batch size 128, FSDP, seven candidate responses per prompt, KL coefficient 1e−3, and clip ratio 0.2.Rollouts use vLLM with tensor parallelism size 4 and GPU memory utilization capped at 0.4; sampling uses temperature 1.0 and top-p 1.0.Source: G IMPLEMENTATION DETAILS · p. 19
  • Training uses maximum input and response lengths of 4,096 and 8,192 tokens, respectively, and runs 7B, 14B, and 32B models on 1, 2, and 4 nodes with 8 H100 GPUs each.Source: G IMPLEMENTATION DETAILS · p. 19
  • For 7B, 14B, and 32B models, Instruct learning rates are 1e−6, 7e−7, and 5e−7, while reasoning-model rates are 1e−6, 1e−6, and 8e−7, respectively.Source: G IMPLEMENTATION DETAILS · p. 19

H FULL EXPERIMENT RESULT

  • Full experiment results and broader baseline coverage are provided for RewardBench, RM-Bench, and RMB in Tables 6, 7, and 8, respectively.Source: H FULL EXPERIMENT RESULT · p. 19

I.1 ABLATION SETTINGS

The ablation settings progressively add generated rubrics, chat-versus-reasoning task categorization, and teacher-trace distillation to a cold-start RL setup using answer correctness and format rewards.

Cold Start RL

  • Cold Start RL combines format and answer rewards, with answer reward equal to 1 for a matching answer and 0 otherwise, and uses a prompt without structured-reasoning guidance.The total reward is R = Ranswer + Rformat.Source: §I.1 · p. 19

Cold Start RL + Rubrics

  • Cold Start RL + Rubrics prompts the model to generate rubrics and evaluate against them, but treats chat and reasoning tasks uniformly.This setting uses Figure 7's prompt rather than the final system prompt in Figure 3.Source: §I.1 · pp. 19–20

Cold Start RL + Rubrics + Query Categorization (QC)

  • Adding query categorization prompts the model to classify tasks as chat or reasoning, with correctness-based evaluation emphasized for reasoning tasks.The setting uses distinct strategies by task type, reflecting the close link between reasoning-task answer quality and correctness.Source: §I.1 · p. 20

Distilled + RL + Rubrics + QC (RM-R1)

  • RM-R1 adds distillation from stronger teachers before RL to help weaker models learn high-quality chat rubrics and convincing reasoning chains.The distillation uses a small subset of data to mitigate RL-only exploration limitations.Source: §I.1 · p. 20

J COMPUTATIONAL OVERHEAD

Long chain-of-thought increases inference latency, but parallelizing rollout and reward computation can make practical overhead minimal when their runtimes are similar. The paper treats efficiency improvements as orthogonal to its core contribution.

J COMPUTATIONAL OVERHEAD

  • Despite higher inference cost, reasoning models offer interpretability, generalization, broader applicability, and dynamic compute allocation, while efficiency improvements remain orthogonal to RM-R1’s contribution.The paper frames its core contribution as recasting reward modeling as reasoning and demonstrating generative RMs can outperform scalar RMs on public benchmarks.Source: J COMPUTATIONAL OVERHEAD · p. 21
  • Parallelizing next-batch rollouts with current-batch reward processing makes total wait the maximum of the two stages’ latencies rather than their sum.In conventional cascaded pipelines, the wait time is the sum of rollout and reward computation times.Source: J COMPUTATIONAL OVERHEAD · pp. 21–22
  • When policy rollout and reward-model computation times are similar on complex tasks, this parallelism can make practical REASRM overhead minimal.Long chain-of-thought outputs naturally increase inference latency.Source: J COMPUTATIONAL OVERHEAD · pp. 21–22

K ANALYSIS FOR THE IMPORTANCE OF RL … K.2 PROOF OF PROPOSITION 1

The analysis argues that high-reward filtering can make disagreement rare in SFT data, while broader RL exploration reveals it, giving RL a stronger incentive to favor robust behavior over shortcuts. The proofs establish the filtering gap and the robust policy’s unique optimality under the stated assumptions.

Define the disagreement probabilities

  • The setup partitions trajectories into high- and low-reward sets at threshold τ and defines disagreement between robust and trivial features.The reward function assigns higher values to better performance, and H and L denote examples respectively at or above and below τ.Source: K ANALYSIS FOR THE IMPORTANCE OF RL · p. 27

K ANALYSIS FOR THE IMPORTANCE OF RL

  • High-reward filtering yields εtrain < δ when disagreement is more prevalent among low-reward than high-reward trajectories.The SFT distribution is the environment distribution conditioned on high reward; under the stated filtering and disagreement assumptions, its disagreement probability is strictly below the environment rate.Source: K ANALYSIS FOR THE IMPORTANCE OF RL · p. 27
  • A finite SFT sample can miss rare disagreements, while RL rollouts increasingly encounter them when the environment disagreement rate δ is positive.The probability of no SFT disagreement can remain close to 1 for moderate sample size N, whereas the probability RL encounters one approaches 1 as rollout count M grows.Source: K ANALYSIS FOR THE IMPORTANCE OF RL · p. 27

Assume the gap

  • RL improves reward by δ versus an εtrain SFT-loss improvement when switching from the trivial to the robust policy, with δ > εtrain.The robust policy is uniquely optimal for RL up to environment-null sets, whereas the trivial policy is strictly suboptimal when δ > 0.Source: Assume the gap · pp. 28–29

K.1 PROOF OF LEMMA 1

  • The Lemma 1 proof uses total probability over H and L to show that greater disagreement in L makes the filtered SFT disagreement rate strictly smaller.The argument relies on both reward regions having nonzero probability.Source: §K.1 · p. 29

K.2 PROOF OF PROPOSITION 1

  • The Proposition 1 proof shows that no deterministic policy can exceed the robust policy’s RL reward, with equality only when it agrees with the ground truth almost surely.Thus the robust policy is the unique RL maximizer up to environment-null sets, and the trivial policy is strictly suboptimal.Source: §K.2 · p. 30

Related papers

Loading 2505.02387v4…