PaperlayerView exact revision on arXiv

Source-linked AI summary

RM-R1: Reward Modeling as Reasoning

Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, Heng Ji

arXiv:2505.02387v4cs.CLcs.AIcs.LG

TL;DR

Reward models need interpretable reasoning to make accurate preference judgments, motivating RM-R1’s reasoning-based formulation. RM-R1 combines reasoning distillation, reinforcement learning, and Chain-of-Rubrics rollouts, and outperforms larger reward models across three benchmarks by up to 4.9%.

  • Problem

    Reward modeling must provide accurate, interpretable judgments for preference-based LLM post-training, but the paper asks whether it can be formulated as a reasoning task.

  • Method

    RM-R1 trains Reasoning Reward Models with reasoning distillation before reinforcement learning and uses Chain-of-Rubrics rollouts tailored to chat or reasoning tasks.

  • Results

    Across three public reward-model benchmarks, RM-R1 matches or surpasses commercial and open-source models while producing more interpretable judgments, with gains up to 4.9%.

  • Takeaways & Limitations

    Task categorization, high-quality reasoning-trace bootstrapping, and reinforcement-learning fine-tuning are all indispensable to RM-R1’s reported performance.

Abstract

from arXiv · show

Reward modeling is essential for aligning large language models with human preferences through reinforcement learning. To provide accurate reward signals, a reward model (RM) should stimulate deep thinking and conduct interpretable reasoning before assigning a score or a judgment. Inspired by recent advances of long chain-of-thought on reasoning-intensive tasks, we hypothesize and validate that integrating reasoning into reward modeling significantly enhances RM's interpretability and performance. We introduce a new class of generative reward models, Reasoning Reward Models (ReasRMs), which formulate reward modeling as a reasoning task. We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1. RM-R1 features a chain-of-rubrics (CoR) mechanism -- self-generating sample-level chat rubrics or math/code solutions, and evaluating candidate responses against them. The training of RM-R1 consists of two key stages: (1) distillation of high-quality reasoning chains and (2) reinforcement learning with verifiable rewards. Empirically, our models achieve superior performance across three reward model benchmarks on average, outperforming much larger open-weight models (e.g., INF-ORM-Llama3.1-70B) and proprietary ones (e.g., GPT-4o) by up to 4.9%. Beyond final performance, we perform thorough analyses to understand the key ingredients of successful ReasRM training.

1 INTRODUCTION

RM-R1 reframes reward modeling as a reasoning task because preference judgments require nuanced, multifaceted evaluation. The resulting models use reasoning-oriented training and achieve strong benchmark performance with interpretable judgments.

  • Reward modeling requires reasoning about latent criteria, trade-offs, consequences, and nuanced response quality.
  • The paper asks whether reward modeling can be cast as a reasoning task.
  • Reasoning Reward Models use long, coherent reasoning chains to assess and distinguish complex outputs.
  • RM-R1 combines reasoning distillation before reinforcement learning with a Chain-of-Rubrics evaluation process.
  • Up to 4.9%: RM-R1 outperforms larger open-weight and proprietary reward models across RewardBench, RM-Bench, and RMB on average.
  • The paper studies training recipes through ablations, scaling analyses, baseline comparisons, case studies, and training-dynamics analyses.

2 RM-R1

RM-R1 trains generative reward models to produce reasoning traces and judgments from preference data. Its pipeline combines reasoning distillation, reinforcement learning, and task-specific Chain-of-Rubrics rollouts.

  • 2.1 TASK DEFINITION: Generative reward modeling maps a prompt and two candidate responses to a textual judgment containing the predicted preferred response.
  • 2.2 REASONING DISTILLATION FOR REWARD MODELING: Reasoning distillation trains an instruction-tuned model on synthesized high-quality reward-modeling reasoning traces from oracle models.
  • 2.3 RL TRAINING: RL addresses distillation overfitting by treating the reward model as a policy and optimizing generated reasoning traces and judgments.
  • 2.3.1 CHAIN-OF-RUBRICS (COR) ROLLOUT: Chain-of-Rubrics first classifies samples as chat or reasoning, then uses rubrics for chat and self-solved correctness judgments for reasoning tasks.
  • 2.3.1 CHAIN-OF-RUBRICS (COR) ROLLOUT: The rollout prompt distinguishes reasoning tasks such as math and code from chat tasks such as safety and general helpfulness.
  • 2.3 RL TRAINING: Training uses correctness-based rewards, while distilled models are assumed to have learned instruction following and response formatting.

3 EXPERIMENTS

RM-R1 is evaluated on three public reward-model benchmarks against scalar, generative, and reasoning-model baselines. It surpasses leading larger models while using smaller model scales.

  • Experimental setup: RM-R1 is evaluated on RewardBench, RM-Bench, and RMB against ScalarRMs, GenRMs, and REASRMs.The training data includes cleaned Skywork Reward Preference 80K, 8K Code-Preference-Pairs examples, and Math-DPO-10K.
  • Main results: RM-R1-DEEPSEEK-DISTILLED-QWEN-14B surpasses INF-ORM-Llama3.1-70B, Nemotron-4-340B-Reward, and GPT-4o on average at smaller scale.The 32B RM-R1 models extend this lead by a notable margin.
  • Main results: Larger baseline models can underperform smaller ones, motivating more comprehensive and systematic reward-model assessment.This evaluation pattern is explicitly reported in the benchmark comparison discussion.

4 ANALYSIS

The analysis identifies reasoning distillation, rubrics, query categorization, and RL as important training components, while showing gains from model scale and inference compute. Warm-start RL is more stable than cold-start RL.

  • 4.1 TRAINING RECIPES: RL training alone is insufficient, while CoR prompting and distillation improve reward-model performance across hard, reasoning, chat, and safety tasks.High-quality reasoning traces before RL yield the strongest ablation results.
  • 4.2 SCALING EFFECTS: Larger RM-R1 models achieve greater final performance and greater training gains across the three evaluated benchmarks.The study examines Qwen-2.5-Instruct models at 7B, 14B, and 32B, with an approximately linear improvement trend.
  • 4.2 SCALING EFFECTS: Increasing inference-time compute from 512 to 8192 tokens improves average performance when training and inference budgets are matched.The experiment uses DeepSeek-R1-Distill-Qwen-14B with identical datasets and hyperparameters across settings.
  • 4.3 EFFECTIVENESS OF REASONING TRAINING: Reasoning-based training consistently outperforms SFT-only approaches under equal data budgets across the three benchmarks.High-quality distillation alone also provides notable gains despite using only a small subset of the data.
  • 4.4 TRAINING DYNAMICS: Cold-start RL lengthens responses but becomes unstable near training end, whereas warm-start RL produces smoother rewards and more stable reasoning refinement.Warm-start RL follows reasoning-chain distillation and begins with stronger reasoning abilities.
  • Qualitative analysis: RM-R1 generates interpretable judgments by articulating preference rationales, prioritizing question-specific rubrics, and grounding evaluations in response content.The qualitative analyses report faithful rubric adherence and systematic evaluation of rubric aspects.

5 CONCLUSION AND FUTURE WORK

The paper concludes that reasoning-based reward models can scale with model size and inference compute while producing interpretable judgments. It identifies task categorization, reasoning-trace bootstrapping, and RL fine-tuning as indispensable.

  • Conclusion: RM-R1 generates explicit chains of rubrics and rationales and matches or surpasses commercial and open-source reward models across three public benchmarks.The conclusion also reports more interpretable judgments than the evaluated alternatives.
  • Conclusion: Task-type categorization, high-quality reasoning-trace bootstrapping, and RL fine-tuning are all indispensable components of RM-R1.Qualitative analyses found that RM-R1 prioritizes high-impact rubrics, follows its criteria, and justifies judgments coherently.
  • Future work: Future work includes active preference collection and extensions to multimodal and agentic reward modeling.The proposed active-learning direction queries human preferences when the current rubric set is insufficient for a new sample.

ETHICS STATEMENT

The paper presents RM-R1 as fundamental research evaluated on public benchmarks without human subjects, sensitive personal data, or private information. It reports no foreseeable ethical risks.

  • Ethics statement: RM-R1 is developed and evaluated entirely on publicly available benchmarks without human subjects, sensitive personal data, or private information.The authors describe the training pipeline as a general optimization technique.
  • Ethics statement: The authors report no foreseeable concerns regarding fairness, bias, discrimination, privacy, or security.They state that the study fully complies with their ethical requirements.

REPRODUCIBILITY STATEMENT

The paper situates RM-R1 among reward-modeling approaches, benchmarks it across three evaluations, and details procedures intended to support reproduction. It also describes teacher-trace distillation followed by reinforcement learning and the GRPO objective.

  • REPRODUCIBILITY STATEMENT: The authors report that experimental settings, datasets, baselines, protocols, hyperparameters, rollout settings, and compute requirements are documented for independent reproduction.A supplementary repository is also provided.
  • RELATED WORK: RM-R1 is compared with scalar, generative, and reasoning-enhanced reward models, including approaches using chain-of-thought or multiple sampled rationales.Reasoning-enhanced models are distinguished by explicit reasoning before final judgments.
  • REPRODUCIBILITY STATEMENT: Claude-3.7-Sonnet generates initial reasoning traces, while OpenAI-O3 corrects approximately 25% of incorrect traces, producing a higher-quality distillation set.The teachers are selected for complementary strengths on easier versus harder tasks and safety considerations.
  • REPRODUCIBILITY STATEMENT: The training pipeline selects approximately 12% of the training data, slightly fewer than 9K examples, for distillation before reinforcement-learning training.The distillation stage precedes RL training.
  • REPRODUCIBILITY STATEMENT: GRPO samples groups of outputs for each prompt and uses their average reward as the baseline, avoiding additional value-function approximation.The objective includes a task-specific loss balanced against KL divergence.
  • EXPERIMENTS: The paper evaluates reward models on RewardBench, RM-Bench, and RMB, covering chat, safety, reasoning, mathematics, code, helpfulness, and harmlessness.RM-Bench is identified as the most reasoning-intensive benchmark.

F.2 PREFERENCE DATASETS

The paper combines cleaned preference data from chat, safety, mathematics, and code with specialized coding and mathematical reasoning datasets. It removes a biased Skywork subset before training and summarizes the resulting corpus in Table 5.

  • F.2 PREFERENCE DATASETS: Skywork Reward Preference 80K spans chat, safety, mathematics, and code, but the authors exclude its approximately 30% magpie_ultra subset because of spurious token and turn-structure correlations.The retained data are the cleaned subset used in experiments.
  • F.2 PREFERENCE DATASETS: Code-Preference-Pairs contains fine-grained coding preference pairs created by introducing bugs and manipulating original and corrected examples; 8K examples are subsampled.The manipulations include swapping broken and corrected versions and removing error comments.
  • F.2 PREFERENCE DATASETS: Math-DPO-10K is a stepwise preference dataset focused on mathematical reasoning, and the full dataset is used.
  • F.2 PREFERENCE DATASETS: The paper states that global training-dataset statistics are summarized in Table 5.Table 5 identifies Skywork-derived sources with an asterisk.
  • F.2 PREFERENCE DATASETS: The study compares RM-R1 with scalar reward models, generative reward models, and reasoning-enhanced reward models.These categories distinguish direct numeric scoring, free-form textual judgments, and explicit reasoning before final judgments.

G IMPLEMENTATION DETAILS

RM-R1 uses a distillation stage followed by GRPO-based reinforcement learning, with separate training configurations for instruct and reasoning models. The implementation reports memory, rollout, sequence-length, learning-rate, and hardware settings, and presents results in three tables.

  • G IMPLEMENTATION DETAILS: The framework uses VERL and OpenRLHF; instruct models receive 8.7K distillation and 64K RLVR examples, while DeepSeek-distilled models use the full RLVR data.
  • G IMPLEMENTATION DETAILS: Distillation uses one epoch with batch size 128 and micro-batch size 1, plus gradient checkpointing, FlashAttention, and Adam offloading.Learning rates are 5e−6, 3e−6, and 2e−6 for 7B, 14B, and 32B models, respectively.
  • G IMPLEMENTATION DETAILS: RLVR uses GRPO with batch size 1024, mini-batch size 128, FSDP, vLLM tensor parallelism of 4, KL coefficient 1e−3, and clip ratio 0.2.Rollout sampling uses temperature 1.0 and top-p 1.0.
  • G IMPLEMENTATION DETAILS: The maximum input and response lengths are 4,096 and 8,192 tokens, respectively, with learning rates specified separately for instruct and reasoning variants.
  • G IMPLEMENTATION DETAILS: The 7B, 14B, and 32B models are trained on 1, 2, and 4 nodes, respectively, each equipped with 8 H100 GPUs.
  • G IMPLEMENTATION DETAILS: Results for RewardBench, RM-Bench, and RMB are reported in Tables 6, 7, and 8, respectively.

I.1 ABLATION SETTINGS

The ablation settings progressively add structured reasoning, task categorization, and teacher-model distillation to reinforcement learning. They compare answer-and-format rewards with rubric-based and task-specific evaluation strategies.

  • I.1 ABLATION SETTINGS: Cold Start RL uses pure reinforcement learning with format and answer rewards, and its prompt provides no guidance on structured reasoning.The total reward is R = R_answer + R_format.
  • I.1 ABLATION SETTINGS: Cold Start RL + Rubrics prompts the model to generate rubrics and evaluate accordingly, while treating chat and reasoning inputs uniformly.
  • I.1 ABLATION SETTINGS: Table 6 reports RewardBench results, while Table 7 reports RM-Bench results with domain- and difficulty-level accuracies.The RM-Bench caption identifies Chat, Math, Code, Safety, Easy, Normal, and Hard breakdowns.
  • I.1 ABLATION SETTINGS: Cold Start RL + Rubrics + Query Categorization first classifies each task as reasoning or chat, then applies different handling strategies.For reasoning tasks, the setting emphasizes correctness-based evaluation.
  • I.1 ABLATION SETTINGS: RM-R1 adds distillation from stronger teacher models before RL, providing a warm start for high-quality rubrics and convincing reasoning chains.The paper motivates this addition by weaker models’ difficulty exploring such chains during RL alone, especially at smaller scales.

J COMPUTATIONAL OVERHEAD

RM-R1’s reasoning-based design increases interpretability and performance but also raises inference latency. The paper argues that parallel rollout and reward computation can make the practical overhead small.

  • J COMPUTATIONAL OVERHEAD: Long chain-of-thought outputs increase inference latency, creating a computational trade-off for reasoning reward models.The paper treats efficiency improvements as separate from its main contribution.
  • J COMPUTATIONAL OVERHEAD: Parallelizing rollout and reward computation changes total waiting time from the sum of stage latencies to their maximum.The next rollout batch can begin while the reward model processes the current batch.
  • J COMPUTATIONAL OVERHEAD: When rollout and reward-model latencies are similar for complex tasks, parallelism can make REASRM overhead practically minimal.The relevant stage times are denoted T1 for policy rollout and T2 for reward-model computation.
  • J COMPUTATIONAL OVERHEAD: The ablation materials compare Chain-of-Rubrics prompts with and without task-type categorization and with and without rubrics.The supplied captions identify these as cold-start RL ablations and distinguish cold-start judges from RM-R1 RL judges.

K ANALYSIS FOR THE IMPORTANCE OF RL

The analysis formalizes why reinforcement learning can overcome shortcuts that supervised distillation may preserve. High-reward filtering makes disagreement rare in SFT data, while broader RL exploration exposes it and gives the robust policy a larger objective advantage.

  • K ANALYSIS FOR THE IMPORTANCE OF RL: High-reward filtering produces an SFT distribution where disagreement is less prevalent than in the environment, yielding εtrain < δ.The formal setup conditions the environment distribution on high-reward examples and assumes disagreement is more common among low-reward trajectories.
  • K ANALYSIS FOR THE IMPORTANCE OF RL: A finite distilled SFT dataset can miss rare harmful disagreement patterns, whereas sufficiently broad RL rollouts eventually encounter them.The probability of seeing disagreement during RL approaches 1 as M grows when δ > 0.
  • K ANALYSIS FOR THE IMPORTANCE OF RL: SFT distinguishes trivial and robust policies by the small margin εtrain, while RL distinguishes them by the larger reward gap δ.The analysis states that δ > εtrain, making the robust strategy uniquely optimal for the RL objective.
  • K ANALYSIS FOR THE IMPORTANCE OF RL: The formal result explains why SFT can overfit to a teacher shortcut even when RL has a stronger incentive to adopt robust behavior.The paper models the teacher as a trivial feature and the distilled dataset as samples labeled by that teacher.
  • K.2 PROOF OF PROPOSITION 1: The robust policy is the unique expected-reward maximizer up to environment-null sets, while the trivial policy is strictly suboptimal when δ > 0.Optimality requires matching the ground-truth label almost surely under the environment distribution.
Loading 2505.02387v4…