PaperlayerView exact revision on arXiv

Source-linked AI summary

Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models

Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, Tuo Zhao

arXiv:2505.16265v1cs.LG

TL;DR

Reward models face a choice between BT models that are sensitive to data coverage and GenRMs whose shallow reasoning can limit complex-task evaluation, while pairwise outputs do not directly fit standard RLHF. Think-RM trains long-horizon reasoning and uses pairwise rewards to optimize policies directly; it reports an 8% RM-Bench gain over BT RMs and vertically scaled GenRMs, alongside stronger end-policy performance.

  • Problem

    BT RMs are sensitive to dataset size and coverage, while existing GenRMs often reason shallowly on complex tasks and produce pairwise outputs incompatible with standard RLHF’s pointwise reward requirements.

    Source: §1 · pp. 1–2

  • Method

    Think-RM warms GenRMs with long-CoT SFT, refines reasoning through rule-based RL, and directly optimizes policies from pairwise preference rewards.

    Source: §1 · p. 2, §3.3 · p. 5, §3.4 · p. 6

  • Results

    8% on RM-Bench over BT RMs and vertically scaled standard GenRMs; Think-RM also outperforms these baselines on ID and OOD tasks and yields stronger end-policy performance with pairwise RLHF.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Think-RM offers a depth-oriented approach to GenRM design and a pairwise-reward route for preference-based policy optimization in RLHF.

    Source: §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) has become a powerful post-training paradigm for aligning large language models with human preferences. A core challenge in RLHF is constructing accurate reward signals, where the conventional Bradley-Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer a more robust alternative by generating chain-of-thought (CoT) rationales followed by a final reward. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRMs by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model's long-horizon abilities by rule-based reinforcement learning (RL). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies using pairwise preference rewards, eliminating the need for pointwise reward conversion and enabling more effective use of Think-RM outputs. Experiments show that Think-RM achieves state-of-the-art results on RM-Bench, outperforming both BT RM and vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance compared to traditional approaches.

1 Introduction

Think-RM addresses limits in reward modeling by extending GenRM reasoning within a single trajectory and directly using pairwise preferences for policy optimization. It outperforms BT RMs and vertically scaled standard GenRMs on ID and OOD tasks, including an 8% gain on RM-Bench.

1 Introduction

  • BT RMs can overfit training patterns, depend sensitively on dataset size and coverage, and contribute to reward over-optimization and reward hacking.RLHF depends on accurate reward signals to align models with human preferences.Source: §1 · pp. 1–2
  • Existing GenRMs typically generate shallow reasoning paths that can miss complex context, long-term dependencies, or a false statement embedded in a fluent response.Their pairwise preference outputs also do not directly fit standard RLHF algorithms requiring pointwise rewards.Source: §1 · p. 2
  • Think-RM models an internal thinking process, extending a single self-guided CoT trajectory from hundreds to thousands of tokens and supporting self-reflection, hypothetical, and divergent reasoning.The framework warms models with SFT on selected long CoTs, then refines reasoning with rule-based RL.Source: §1 · p. 2
  • The proposed pairwise RLHF pipeline optimizes policies directly from pairwise preference rewards, avoiding pointwise reward conversion.The authors report stronger end-policy performance than traditional pointwise RLHF with BT RMs.Source: §1 · p. 3
  • 8% on RM-Bench over BT RMs and vertically scaled standard GenRMs, with state-of-the-art performance among publicly available reward models under 10B using 6K training examples.Think-RM also outperforms these baselines on both in-distribution and out-of-distribution tasks.Source: §1 · p. 3

2 Related Work

Prior work spans discriminative BT reward models, generative models that produce rationales, direct LLM judging, and long-horizon reasoning. Think-RM applies extended reasoning horizons to reward modeling, where existing GenRMs often remain shallow on complex tasks.

2 Related Work

  • BT reward modeling trains a discriminative model to predict scalar rewards from preference data, but requires high-quality data and remains sensitive to coverage and reward hacking.The passage notes its continued adoption in systems such as GPT-4 and Qwen-2.5.Source: §2 · p. 3
  • GenRMs generate explanatory rationales before preference decisions, with reported data-efficiency, OOD-generalization, and interpretability benefits, but their reasoning paths are often shallow.Prior work also uses majority voting over multiple paths for test-time compute.Source: §2 · p. 3
  • LLM-as-a-judge work uses off-the-shelf models with prompt engineering, while reported limitations include position and verbosity biases and difficulty judging tasks beyond the model’s own solving ability.These constraints are noted for complex reasoning problems requiring detailed analysis or long-term dependency tracking.Source: §2 · p. 3
  • Long-horizon reasoning work, including o1/o3 and DeepSeek R1, motivates extending reasoning chains to preference learning and evaluation tasks.The cited work describes extended multi-step reasoning and reinforcement-learning training of long-horizon capabilities.Source: §2 · pp. 3–4

3 Method

Think-RM equips generative reward models with long-horizon reasoning through long-CoT supervised warm-up and accuracy-based rule reinforcement learning, then uses pairwise preferences directly to optimize policies.

3 Method

  • Think-RM uses long-CoT supervised fine-tuning followed by rule-based reinforcement learning to develop long-horizon reasoning in generative reward models.The method flow starts from a pretrained LLM and proceeds through task specification, preference data creation, and the two training stages.Source: §3 · p. 4

3.1.1 Generative Reward Modeling

  • GenRMs evaluate responses through natural-language reasoning; Think-RM focuses on preference judgments using five HelpSteer attributes and safety as criteria.A pairwise GenRM takes a prompt and two responses, then generates internal reasoning and a preference; multiclass outputs encode preference strength and direction.Source: §3.1.1 · p. 4

3.1.2 Reinforcement Learning from Human Feedback

  • RLHF optimizes a policy language model against scalar rewards, with PPO as one optimization option and KL regularization and token-level advantages in its surrogate objective.The advantage estimates may use generalized advantage estimation or group relative policy optimization.Source: §3.1.2 · pp. 4–5

3.2 Warm-up Supervised Fine-Tuning

  • Warm-up SFT selects the longest reasoning trajectory that predicts the preference label correctly, then jointly trains preference prediction and reasoning generation.The candidate trajectories are generated by a pretrained reasoning model for each preference-dataset instance.Source: §3.2 · p. 5

3.3 Rule-based Reinforcement Learning

  • Rule-based RL refines verbose SFT reasoning with GRPO using accuracy-only rewards, assigning full credit to exact predictions and partial credit to correct multiclass preference direction.For multiclass outputs, exact matches receive 1.0, matching preference direction receives 0.5, and other predictions receive 0.0.Source: §3.3 · p. 5

3.4 Pairwise RLHF with GenRMs

  • Pairwise RLHF fine-tunes a target policy directly from GenRM comparisons, forming a preference-strength matrix for response pairs and using it to estimate advantages.This avoids recovering pointwise rewards for individual prompt-response pairs; the described advantage method is for GRPO, with similar principles for GAE.Source: §3.4 · p. 6

4 Experiments

The experiments compare reward models under matched training-data conditions across in-distribution, shifted, and out-of-distribution tasks, and compare pairwise with pointwise RLHF. Think-RM outperforms baselines on reasoning-intensive OOD benchmarks, while pairwise RLHF with generative reward models yields stronger end-policy performance than pointwise RLHF with BT reward models.

4.1 Experiment Setup

  • The setup uses HelpSteer2-Preference and QwQ-32B-generated long internal CoTs, retaining 6K binary and 4K multiclass examples after filtering.Head-to-head comparisons use the same training data within each category; the authors do not directly compare binary and multiclass GenRMs because their training sets differ.Source: §4.1 · p. 6

4. Experiments

  • The experiments evaluate reward models on ID, moderate-shift, and OOD benchmarks, and assess end-policy performance using AlpacaEval2 with GPT-4-as-a-judge.The setup describes HelpSteer2 validation as ID evaluation, HelpSteer3 as moderate shift, and RewardBench and RM-Bench as OOD benchmarks.Source: §4 · p. 6, §4.1 · p. 7

4.2.1 Evaluation on In-Distribution and Moderately Shifted Tasks

  • Binary Think-RM outperforms all baselines on in-distribution and moderately shifted tasks, and leads in HelpSteer3-Preference’s reasoning-heavy code domain.For multiclass tasks, Think-RM performs slightly below ground-truth CoT-GenRM but matches or exceeds model-generated CoT-GenRM, including with vertical scaling.Source: §4.2.1 · p. 7, Table 1 · p. 8

4.2.2 Evaluation on Out-of-Distribution Tasks

  • Up to 8% average improvement on RM-Bench: Think-RMs outperform all baselines on reasoning-intensive OOD tasks, including vertically scaled CoT-GenRMs.The reported average improvements reach 5% on RewardBench and 8% on RM-Bench.Source: §4.2.2 · p. 8, Table 3 · p. 9

4.3 Ablation Study

  • Selecting the longest CoT per instance for warm-up data yields higher preference accuracy across benchmarks than selecting the shortest, at the cost of longer responses.Figure 3 compares preference accuracy and average response length for the two warm-up data-selection strategies.Source: §4.3 · p. 9, Figure 3 · p. 9

4.4 Pairwise vs. Pointwise RLHF

  • Pairwise RLHF policies using generative reward models outperform pointwise RLHF policies using BT reward models; Think-RM also has a higher overall win rate than CoT-GenRM.Think-RM and CoT-GenRM have comparable HH-RLHF test preference accuracy (65.03 vs. 64.42), which the authors associate with the prompts’ relative ease.Source: §4.4 · pp. 9–10, Table 4 · p. 10

5 Conclusion

Think-RM trains generative reward models for long-horizon reasoning through internal thinking, and pairs them with a pipeline that directly uses pairwise preference rewards. The authors report that Think-RMs outperform BT RMs and vertically scaled CoT-GenRMs, with superior end-policy performance when used in their pairwise RLHF approach.

  • Think-RMs outperformed BT RMs and vertically scaled CoT-GenRMs, and achieved superior end-policy performance with the pairwise RLHF approach.Source: §5 · p. 10
  • Think-RM equips generative reward models with long-horizon reasoning through an internally modeled thinking process.Source: §5 · p. 10
  • The framework uses supervised fine-tuning on long chain-of-thought data followed by rule-based reinforcement learning to elicit advanced reasoning skills.Source: §5 · p. 10
  • The pairwise RLHF pipeline directly uses pairwise preference rewards, removing the need for pointwise reward conversion.Source: §5 · p. 10

A Task Instruction Template

The task instruction templates direct a judge to compare two assistant responses using specified evaluation criteria. One template requests a graded preference, while another requests a binary winner.

B Additional Implementation Details

The implementation details specify training resources and configurations for Think-RM, its baselines, pairwise RLHF, and evaluation. Selected hyperparameters are summarized across binary and multiclass setups.

Loading 2505.16265v1…