PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference-Time Scaling for Generalist Reward Modeling

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu

arXiv:2504.02495v3cs.CLcs.AIcs.LG

TL;DR

Generalist reward modeling must produce accurate, flexible rewards for diverse queries beyond domains with explicit rules or ground truth. The paper combines pointwise generative reward modeling with SPCT, parallel sampling, and a meta RM, and reports improved reward quality and inference-time scalability across benchmarks. DeepSeek-GRM can also outperform training-time scaling by model size, although transparency and task-specific limitations remain.

  • Problem

    Generalist reward modeling lacks accurate, flexible reward signals for diverse domains where explicit reference answers or ground truth are often unavailable.

  • Method

    The paper trains pointwise generative reward models with SPCT through online RL, then scales inference with parallel sampling and a meta RM-guided voting process.

  • Results

    SPCT improves GRM reward quality and inference-time scalability across diverse RM benchmarks, with DeepSeek-GRM outperforming baseline methods and some strong public RMs.

  • Takeaways & Limitations

    Inference-time scalable GRMs could serve as versatile interfaces for generalist reward systems in LLM post-training and inference.

  • Takeaways & Limitations

    Self-bootstrapped reward generation may produce unfaithful principles and critiques, raising transparency and accountability concerns.

Abstract

from arXiv · show

Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that $\textit{proper learning methods could enable effective inference-time scalability}$. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the $\textbf{inference-time scalability of generalist RM}$, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in $\textbf{DeepSeek-GRM}$ models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope.

1 Introduction

Generalist reward modeling aims to provide accurate, flexible reward signals for diverse LLM queries where explicit rules or ground truth are often unavailable. The paper proposes SPCT-trained generative reward models and inference-time scaling with parallel sampling and a meta RM.

  • Motivation: General-domain reward generation is difficult because criteria are diverse and explicit references or ground truth are often absent.This motivates generalist reward modeling for both RL post-training and RM-guided inference.
  • Challenges: Generalist RMs must support different input types, generate accurate rewards across domains, improve rewards with more inference compute, and learn scalable behaviors.The paper identifies these as four requirements for effective inference-time scalability.
  • Approach: Pointwise generative reward modeling unifies scoring single, paired, and multiple responses in a language-based representation.This addresses input flexibility while preserving potential for inference-time scaling.
  • Approach: SPCT uses rule-based online RL to teach GRMs to generate input-adaptive principles and accurate critiques, producing DeepSeek-GRM models.The method targets scalable reward-generation behavior in general domains.
  • Inference-time scaling: Parallel sampling expands inference compute, while a meta RM guides voting to improve scaling performance.These components are introduced specifically to make inference-time scaling more effective.
  • Results: SPCT improves GRM quality and scalability across RM benchmarks, and inference-time scaling can outperform training-time scaling on model size.The authors also report competitive or superior performance without severe domain biases.

2 Preliminaries

Reward models differ along reward-generation and scoring dimensions that determine input flexibility and inference-time scalability. The paper focuses on pointwise generative scoring and uses principles to improve reward quality for general-domain evaluation.

  • RM taxonomy: Reward-generation paradigms are scalar, semi-scalar, and generative, while scoring patterns are pointwise or pairwise.These two design axes characterize the main RM approaches discussed in the paper.
  • Scoring patterns: Pointwise scoring assigns an individual reward to each response, whereas pairwise scoring selects a best response from candidate responses.Pairwise scoring is typically applied with n = 2 and cannot score a single response directly.
  • Inference-time scalability: Scalar methods generally have little inference-time scalability because their reward generation lacks diversity.The paper contrasts this limitation with generative approaches that can produce varied reward explanations or scores.
  • Generative reward modeling: Generative reward models can generate pointwise rewards for multiple responses in pure language representations, enabling input flexibility and inference-time scalability.The extracted rewards are usually discrete, with the default range set to 1 through 10.
  • Principle-guided rewards: Generalist RM replaces artificial rules with principles that guide reward generation when criteria are diverse and ground truth is unavailable.The preliminary experiment evaluates principle-guided reward quality on Reward Bench and PPE subsets.
  • Preliminary findings: Filtered principles significantly improve reward quality, whereas self-generated principles alone barely improve performance.The result suggests generated principles vary in suitability and that correctly selected principles can guide reward generation.

3 Self-Principled Critique Tuning (SPCT)

SPCT trains pointwise generative reward models to generate adaptive principles and accurate critiques through rejective fine-tuning and rule-based online RL. These behaviors support inference-time scaling by producing finer-grained rewards from multiple generated principles.

  • Unpinning Principles from Understanding to Generation: SPCT trains pointwise GRMs to generate adaptive principles that guide high-quality critiques for generalist reward generation.Principles are generated as part of reward generation rather than fixed as a preprocessing step.
  • Unpinning Principles from Understanding to Generation: Generated principles are conditioned on the query and responses, allowing post-training to improve the quality and granularity of principles and critiques.The principle-generation function shares the same language head and model with reward generation.
  • Training Procedure: SPCT combines rejective fine-tuning as a cold start with rule-based online RL to optimize principle and critique generation simultaneously.The online RL stage uses rule-based outcome rewards and compares predicted rewards with ground truth using accuracy rules.
  • Rejective Fine-Tuning: Rejective fine-tuning teaches GRMs to produce principles and critiques in the correct format across input types using a unified pointwise representation.Pointwise GRMs generate rewards for single, paired, and multiple responses in the same format.
  • Rejective Fine-Tuning: Hinted sampling appends the ground-truth best-response index to prompts, while online RL addresses shortcut-taking observed in some hinted critiques.Hinted trajectories are rejected only when incorrect, but reasoning-task critiques can still exploit shortcuts.

4 Inference-Time Scaling with SPCT

Inference-time scaling for DeepSeek-GRM uses parallel sampling and voting to aggregate multiple generated rewards. A meta reward model further guides voting, addressing bias and limited sensitivity to subtle response differences in majority-based aggregation.

  • Sampling-Based Scaling: Parallel sampling expands inference compute by generating multiple reward sets for the same query and responses before aggregation.Sampling-based methods are used to improve performance with more inference compute.
  • Voting with Generated Rewards: Semi-scalar RMs aggregate k sampled scalar rewards by averaging, whereas pairwise GRMs select the response identified as best most frequently.The pairwise procedure is majority voting over individually selected best responses.
  • Voting with Generated Rewards: Majority voting may be biased because ties are disallowed and may miss subtle response differences without quantitative scores.This limitation motivates alternatives that retain richer reward information during aggregation.
  • Voting with Generated Rewards: Pointwise GRMs sum rewards across sampled reward sets, expanding the effective reward space by k times when individual scores use a small discrete range.The expanded space supports more principles and can improve the granularity of final rewards.
  • Meta Reward Modeling Guided Voting: A pointwise scalar meta RM is trained to identify whether DeepSeek-GRM principles and critiques are correct, then guides the voting process.The meta RM uses binary cross-entropy with labels determined from the reward-ranking procedure.

5 Results on Reward Modeling Benchmarks

DeepSeek-GRM-27B performs strongly across generalist reward-modeling benchmarks, and SPCT enables further gains through inference-time scaling. Meta-RM-guided voting and principle generation are important components, while scaling inference compute can outperform scaling model size.

  • Benchmark performance: DeepSeek-GRM-27B outperforms reproduced baselines overall and becomes the best overall performer with inference-time scaling.It is competitive with Nemotron-4-340B-Reward and GPT-4o before scaling.
  • Benchmark performance: Scalar and semi-scalar reward models show domain-biased performance, whereas PairRM alleviates this problem and LLM-as-a-Judge performs below DeepSeek-GRM-27B.Scalar models perform especially well on PPE Correctness but fail on other benchmarks.
  • Inference-time scalability: With up to 8 samples, DeepSeek-GRM-27B achieves the largest performance increase over greedy decoding and sampling results.The model retains potential for further gains with inference compute up to 32 samples.
  • Inference-time scalability: The meta RM filters low-quality trajectories and improves scaling, while token-probability-weighted voting strengthens LLM-as-a-Judge beyond majority voting.For CLoud-Gemma-2-27B, gains remain limited because scalar reward generation has little variance.
  • Ablation study: Principle generation is crucial for both greedy decoding and inference-time scaling, and meta-RM-guided voting remains robust across different k_meta values.Online RL improves a general-instruction-tuned GRM from 66.1 to 68.7 without cold-start critique data.
  • Inference versus training scaling: Direct voting with 32 DeepSeek-GRM-27B samples matches the 671B MoE model, while meta-RM-guided voting achieves the best result with 8 samples.On a 300-example Reward Bench subset, DeepSeek-R1-0120 performs below the 236B MoE RFT model.

6 Related Work

Prior work includes scalar, semi-scalar, and generative reward models, alongside inference-time scaling methods based on sampling and reward-guided aggregation. Recent reasoning-oriented methods improve judging of difficult verifiable questions, but this paper focuses on scalable generalist reward modeling.

  • Reward modeling paradigms: Generative reward models represent rewards as textual feedback or scores, enabling richer representations and flexible judging of single and multiple responses.LLM-as-a-Judge methods support reference-based or reference-free pairwise judging.
  • Inference-time scaling: Inference-time scaling research studies sampling and reward-model-guided aggregation as alternatives or complements to scaling LLMs during training.Long-horizon chain-of-thought improves solving and judging of difficult verifiable questions.

7 Conclusion and Future Work

The paper introduces SPCT to improve inference-time scalability for generalist reward modeling and reports gains in reward quality and scalability across diverse domains. It also identifies future uses of GRMs in RL pipelines, co-scaling, and offline evaluation.

  • Conclusion: SPCT uses rule-based online RL to enable adaptive principle generation and critique generation in GRMs.The resulting DeepSeek-GRM models improve reward quality and inference-time scalability across diverse domains.
  • Conclusion: DeepSeek-GRM surpasses baseline methods and several strong public reward models, with further inference-time gains from meta-RM guidance.The paper suggests integrating GRMs into online RL, co-scaling them with policy models, and using them as offline evaluators.

Ethics Statement

The ethics statement discusses risks from automated reward generation, including bias amplification, unfaithful principles and critiques, and concerns about transparency and accountability. It emphasizes human oversight, public supervision, and continuous evaluation.

  • Bias and oversight: Automated principle and critique generation may perpetuate or amplify biases when training data is toxic, despite DeepSeek-GRM showing fewer biases empirically.The authors prioritize further meta-RM investigation and bias-mitigation strategies.
  • Bias and oversight: The paper advocates retaining human-in-the-loop frameworks rather than treating SPCT as a replacement for human oversight.SPCT is presented as a proxy method for scaling human oversight more efficiently.
  • Transparency and accountability: Broader deployment across diverse domains raises transparency and accountability concerns because self-bootstrapping may produce unfaithful principles and critiques.The authors point to case studies, stated limitations, and public supervision of the open-source model.
  • Ongoing evaluation: Responsible use requires robust validation and ongoing vigilance across reward-modeling benchmarks and practical scenarios.The statement calls for proactive risk management and continuous bias evaluation.

B Limitations and Future Directions

SPCT improves generative reward modeling but leaves efficiency and task-specific performance limitations. Future work targets more efficient generation, stronger performance on verifiable tasks, and tool-augmented or process-level reward modeling.

  • Limitations: Generative RMs remain less efficient than scalar RMs at the same scale, limiting their use in online RL pipelines.Parallel sampling with a reasonable number of samplings, such as eight, avoids a significant latency increase during inference-time scaling.
  • Limitations: DeepSeek-GRM still lags scalar models on specific verifiable tasks, although reference-based rewards and long-horizon reasoning can mitigate this limitation.Scalar RMs capture hidden features of reasoning queries and responses, whereas GRMs require stronger reasoning to examine responses thoroughly.
  • Future Directions: DeepSeek-GRM’s pointwise design could support process reward modeling in addition to outcome reward modeling, but this direction remains largely unexplored.Performance on the Reasoning subset of Reward Bench, which mainly comprises MATH-prm data, partially supports this potential application.
  • Future Directions: Tool incorporation could improve critique accuracy for tasks requiring strict procedures or extensive knowledge.Proposed tools include code interpreters and search-engine interfaces.

C.1 Model Training

The models use standard GRPO-based rule-driven online RL alongside rejective fine-tuning, with KL regularization and parallel-sampling inference configurations. Training data combine general instruction, rejective-sampled, and RL data, while larger models receive a reduced training recipe.

  • Training objective: Rule-based online RL uses the standard GRPO setting with group-normalized rewards and a KL penalty.The group size is G, and β controls the KL penalty coefficient.
  • Hyperparameters: β = 0.08 is most stable for DeepSeek-GRM-27B, while too-small KL coefficients cause benchmark-subset collapse and domain biases.DeepSeek-GRM-16B uses β = 0.002, and G = 4 balances efficiency and performance.
  • Training data: The training set contains 1256K RFT examples and 237K RL examples spanning general instructions and preferences for single, paired, and multiple responses.Preference data include internal and open-source sources such as MATH, UltraFeedback, OffsetBias, Skywork-Reward-Preference-80K-v0.2, and HelpSteer2-Preference.
  • Meta RM: The meta RM reuses rejective-sampled RFT data and uses three DeepSeek-GRM-27B sampling rounds to reduce potential voting bias.Meta RM training uses learning rate 1 × 10^-5 and batch size 512.
  • Training schedule: Both RFT and RL run for 900 steps, but models larger than 27B omit rule-based RL and train only on 50K rejective-sampled examples.This restriction is attributed to resource constraints.
  • Baselines: Baselines are reimplemented on Gemma-2-27B with compatible training data and settings, while LLM-as-a-Judge can use only pairwise data during RL.CLoud-Gemma-2-27B cannot perform rejective sampling because it has no trained value head for extracting rewards.

D.2 Benchmarks

The evaluation covers reward modeling benchmarks spanning chat, reasoning, safety, correctness, and mistake detection, using benchmark-specific accuracy or ROC-AUC metrics. Results examine scaling, input flexibility, reference use, principle transfer, and training-data generalization.

  • Benchmarks: The benchmark suite includes Reward Bench, PPE, RMB, and ReaLMistake across multiple domains and response-input settings.Reward Bench contains chat, reasoning, and safety preference data; ReaLMistake is evaluated with ROC-AUC.
  • Metrics: Evaluation uses best-response accuracy for Reward Bench, PPE, and RMB, while ReaLMistake uses ROC-AUC.RMB BoN examples are correct only when the best response is identified among multiple responses.
  • Input flexibility: DeepSeek-GRM-27B is barely affected by pair versus list input on RMB BoN, with performance differences below 1% on helpfulness and harmlessness.This supports flexibility to evaluate many responses with different input formats.
  • Mistake detection: DeepSeek-GRM achieves the best same-size performance on ReaLMistake and comparable performance to the strongest public models with inference-time scaling.The benchmark reports ROC-AUC performance for 16B and 27B models.
  • Reference-based rewards: DeepSeek-GRM exceeds 90% accuracy on PPE correctness when supplied with references, mitigating its performance limitation on verifiable tasks.The reference is the ground truth for each query.
  • Principle transfer: Principles generated by DeepSeek-GRM-27B transfer to other models and are slightly better than manually filtered GPT-4o principles.The experiment compares principles generated by different models under the same evaluation setup.
  • Training-data generalization: Removing all MATH training data still leaves math-related preference data beneficial for generalist RM performance, especially on Reward Bench Chat Hard.The result indicates generalization beyond the direct coverage of training data.

E.4 Response Length Analysis for Rule-Based RL

Rule-based online RL changes DeepSeek-GRM response lengths unevenly across Reward Bench subsets. The largest increase occurs for reasoning, where performance also improves substantially relative to the RFT-only model.

  • Response lengths: The largest response-length increase occurs on the Reward Bench Reasoning subset after rule-based online RL.Response lengths are compared before and after RL using each model’s tokenizer.
  • Response lengths: Chat response length barely increases after RL, indicating that additional generated tokens are not uniformly used across benchmark domains.The passage contrasts Chat with larger changes on other subsets.
  • Performance and compute: Reasoning performance improves greatly over DeepSeek-GRM-27B-RFT, suggesting adaptive allocation of more inference compute to reasoning-intensive tasks.The authors contrast this with DeepSeek-R1-0120, which uses more tokens but achieves lower results except on Reasoning.

F.1 Case Study

The case studies compare DeepSeek-GRM-27B with scalar and alternative responses across coding, behavioral neuroscience, and cryptocurrency-related prompts. They illustrate robustness and scalable voting, while also exposing failures in principle-following and real-time-data assessment.

  • Robustness: DeepSeek-GRM-27B generates textual principles and critiques, showing better robustness than DeepSeek-BTRM-27B against specific hacking or bias circumstances.Table 16 describes better robustness in distinguishing slight response differences.
  • Inference-time scaling: In inference-time scaling, DeepSeek-GRM-27B generates accurate rewards after voting on multiple samples, and meta RM scores indicate effective voting guidance.Table 17 reverses response order for results 2 and 3 in this setting.
  • Limitations: A failure case arises when DeepSeek-GRM-27B cannot accurately judge responses under some principles, including real-time data, while principle weights may remain unbalanced.Weight attribution and score collection vary across samplings and test cases.
  • Coding evaluation: The coding case finds both responses mostly correct on Tribonacci calculations but identifies incorrect odd-index formulas in each response.Response 1 adds ‘1 + i + (i + 1) / 2‘, while Response 2 adds ‘1 + (i + 1) / 2‘.
  • Behavioral-neuroscience evaluation: The behavioral-neuroscience case gives both responses an overall score of 8, with instruction adherence and usefulness each rated 8 and 7 respectively.Both responses receive relevance 9 and level-of-detail 8 in the supplied scores.
Loading 2504.02495v3…