PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Reward Models for Multi-Domain Test-Time Scaling

Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song

arXiv:2510.00492v3cs.AI

TL;DR

Evidence on reward-model verifier types beyond math-adjacent settings remains limited, despite their use in test-time scaling. The paper compares four discriminative and generative ORM and PRM variants across 14 domains, finding gORM strongest overall in multi-domain and specialized settings.

  • Problem

    Verifier studies have focused mainly on math-adjacent domains, leaving the relative performance of outcome versus process and discriminative versus generative verifiers underexplored across domains.

    Source: §1 · p. 2

  • Method

    The paper evaluates dORM, dPRM, gORM, and gPRM under a common protocol across 14 diverse domains.

    Source: §1 · p. 2

  • Results

    In multi-domain and specialized settings, dORM performs on par with dPRM, gPRM is not competitive, and gORM delivers consistent gains over the others.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings challenge the assumption that finer-grained process supervision is always better and support generative outcome verification in multi-domain settings.

    Source: §1 · pp. 2–3

    The models use LoRA adapters rather than full-parameter fine-tuning, which may affect performance and scaling behavior.

    Source: §5 · p. 13

Abstract

from arXiv · show

The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic. Prior work has studied both outcome reward models (ORMs), which assess only the final answer, and process reward models (PRMs), which score intermediate reasoning steps. Although PRMs are often viewed as advantageous due to their finer-grained supervision, much of the supporting evidence comes from math-adjacent settings, and their relative benefits across broader domains remain unclear. We present the first unified evaluation of four reward model variants, discriminative ORM and PRM (dORM, dPRM) and generative ORM and PRM (gORM, gPRM), across 14 diverse domains. Contrary to conventional wisdom, we find that (i) dORM performs on par with dPRM, (ii) gPRM is not competitive, and (iii) overall, gORM is the most robust, yielding significant and consistent gains across every tested domain. We attribute the worse performance of gPRM to the stepwise scoring process, which inherits label noise from LLM-based automatic labeling, leading to difficulties in evaluating long reasoning trajectories, including those involving self-correcting reasoning. Both our theoretical analysis and empirical observations indicate that stepwise aggregation compounds errors as reasoning length increases. These findings challenge the common assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. Our \href{https://github.com/db-Lee/Multi-RM}{\underline{code}} is publicly available to facilitate future research in multi-domain settings.

1 Introduction

Because verifier comparisons beyond math-adjacent settings remain limited, the paper evaluates four reward-model variants under a common protocol across 14 diverse domains. It finds gORM strongest overall in multi-domain and specialized settings, while dORM matches dPRM and gPRM is not competitive.

1 Introduction

  • Generative verifiers had previously shown strong results on math-adjacent tasks, outperforming discriminative verifiers.Source: §1 · p. 2
  • The broader comparison addresses a gap: verifier research has focused mainly on math-adjacent domains, and prior multi-domain PRM work covered discriminative models only.The paper highlights legal and medical applications as settings where trustworthy output verification is especially important.Source: §1 · p. 2
  • Across multi-domain and specialized settings, dORM performs on par with dPRM, gPRM is not competitive, and gORM delivers consistent, significant gains.The study compares four verifier variants under a common protocol across 14 diverse domains.Source: §1 · pp. 2–3
  • In math, dPRM outperforms dORM and generative variants outperform discriminative ones, consistent with prior work.Source: §1 · p. 2
  • The paper links gPRM's weaker performance to length-sensitive stepwise aggregation and noisy process labels, including a length-distribution shift after consensus filtering.Stepwise aggregation can miss long reasoning chains that recover from earlier errors; the analysis reports that PRM risk increases with chain length.Source: §1 · pp. 2–3

2 Background and Related Work

This section formalizes test-time scaling with reward-model selection and reviews outcome- and process-based verifier variants. It describes how the variants are trained and how their scores are used to select reasoning chains.

2.1 Problem Formulation

  • In best-of-N test-time scaling, the model samples N reasoning chains and selects the candidate with the highest reward, using a learned verifier as a surrogate for the unknown true reward.Source: §2.1 · p. 3

2.2 Reward Models

  • dORM learns from final-outcome labels, whereas dPRM learns from intermediate process labels and aggregates step scores using the minimum.Source: §2.2 · p. 4
  • gORM generates a verification rationale and binary verdict, retaining training examples only when the parsed verdict agrees with the known outcome label.Source: §2.2 · p. 4
  • Minimum, average, product, and last-step aggregation yield only marginal differences, with minimum slightly outperforming the alternatives overall.Source: §2.2 · p. 4
  • gPRM generates stepwise verdicts and trains only on prefixes whose predicted process labels agree with the ground truth; verification stops at the first predicted incorrect step.Source: §2.2 · p. 5
  • Recent gPRM extensions target math, but the authors follow Khalifa et al. because those approaches do not directly extend to multi-domain data.Source: §2.2 · p. 5

3 Experiments

The experiments compare four reward-model variants under controlled conditions across math, multi-domain, and specialized-domain benchmarks. Across multi-domain test-time scaling, gORM is strongest overall, while the relative performance of discriminative models and PRMs changes from the math setting.

3.1 Experimental Setups

  • The math evaluation trains on human-labeled PRM800K and tests on four ProcessBench splits; GSM8K and Math use 16 Qwen2.5-7B-Instruct CoTs per question for test-time scaling.The setup is intended to reproduce prior findings under controlled conditions.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 Experimental Results

  • In math-domain Best-of-N, dPRM outperforms dORM, generative models outperform discriminative ones, and gORM slightly surpasses gPRM.With 1.5B backbones, dORM underperforms majority voting.Source: Figure 4 · p. 6, §3.2 · p. 8

3.1 Experimental Setups

  • MMLU-Pro covers 14 domains, with 16 training and 128 evaluation CoTs per question; process labels are automatically annotated, and GPQA-Diamond, MedQA, and LEXam broaden specialized-domain evaluation.Additional pLLMs are used to assess generalization across generators.Source: §3.1 · p. 6

3.2 Experimental Results

  • gORM outperforms dORM, dPRM, and gPRM overall on multi-domain Best-of-N, without notable degradation in any domain; dORM performs comparably to dPRM, while gPRM lags.The pattern also holds with a smaller and a non-distilled backbone.Source: Figure 6 · p. 7, §3.2 · p. 8
  • On MMLU-Pro, ORM variants achieve higher outcome-verification F1 scores than PRM variants, reversing the math-domain pattern.The comparison uses R1-Distill-Qwen-14B and a 0.5 decision threshold.Source: Figure 5 · p. 7, §3.2 · p. 8
  • Across GPQA-Diamond, MedQA, and LEXam, generative variants outperform discriminative variants in specialized domains; gORM beats or matches gPRM depending on the pLLM.On GPQA-Diamond, gORM outperforms the other variants; on MedQA and LEXam, generative variants show strong gains over discriminative variants.Source: §3.2 · pp. 8–9

4 Analysis on Why PRMs Fail in the Multi-Domain Settings

The analysis examines why PRMs underperform in multi-domain settings, focusing on chain length and process-label noise. It finds that PRM errors rise with chain length and that consensus filtering is associated with a length-distribution shift that disproportionately affects gPRM.

4.1 Risk of PRMs with CoT Length

  • PRMs perform particularly poorly on “aha” CoTs—those with at least one incorrect step but a correct outcome—which account for 15.3% of ProcessBench cases; replacing intermediate reasoning steps with steps from other CoTs significantly reduces ORM performance.Scaling the backbone from 1.5B to 7B improves ORM performance but degrades PRM performance on “aha” CoTs.Source: §4.1 · p. 9
  • Longer CoTs increase the chance that PRMs misclassify an intermediate step and miss later recovery; majority-voting results also degrade as CoT length increases in multi-domain tasks.“Aha” moments lengthen CoTs, especially in MMLU-Pro.Source: §4.1 · p. 9, Figure 9 · p. 10
  • The “aha” experiments are limited to ProcessBench, whose step labels are human-annotated; on MMLU-Pro, LLM-annotated labels would make the measured “aha” rate depend on annotator reliability.Source: §4.1 · p. 9
  • Under stated assumptions, ORM expected squared log-error remains bounded independently of CoT length T, while dPRM and gPRM lower bounds grow at least linearly with T.The gPRM bound is strictly worse than dPRM’s because generative sampling adds per-step variance.Source: §4.1 · p. 10

4.2 Label Noise and Consensus Filtering of PRMs

  • Across domains, gPRM’s Wasserstein distance has a strong negative correlation with its improvement over majority voting (−0.625), while correlations are weak for dORM, dPRM, and gORM.The corresponding correlations are 0.345/0.048/0.082 for dORM/dPRM/gORM; the results suggest consensus filtering’s length shift disproportionately affects gPRM.Source: Figure 13 · p. 11, §4.2 · pp. 11–12
  • With synthetic process-label noise on PRM800K, dPRM is highly sensitive to label noise, whereas gPRM is more robust.Noise is varied by per-step label-flip probability and by the fraction of examples receiving noise.Source: §4.2 · pp. 10–11, Figure 11 · p. 11
  • In multi-domain training, consensus filtering prunes longer CoTs, shifting gPRM’s training-length distribution away from the test set; overall Wasserstein distances are 0.202/0.532/3.083 for Train/gORM/gPRM.In math, gPRM instead has the smallest distance: 2.760/2.430/1.600 for Train/gORM/gPRM.Source: §4.2 · p. 11, Figure 12 · p. 11
  • Label refinement and relaxed consensus filtering increase CoT survival but neither substantially reduces Wasserstein distance nor improves downstream performance.The tests use Gemini-2.0 Flash for label refinement.Source: §4.2 · p. 12

5 Practical Guidelines, Limitations, and Future Work

The section recommends reward-model variants according to CoT length, label quality, domain stability, and resource constraints. It also identifies limits in training regimes and tool use.

5 Practical Guidelines, Limitations, and Future Work

  • For short CoTs with clean labels and tight latency, use dPRM; for long CoTs or frequent recoveries, prefer gORM if compute permits, otherwise dORM.The PRM recommendation for short, clean CoTs reflects their informative per-step signal; the long-CoT recommendation reflects PRMs’ difficulty rewarding recoveries from earlier errors.Source: §5 · p. 12
  • Generative variants are recommended when training data is limited because they better exploit LLM priors and rationale-style supervision, improving sample efficiency.The authors point to Fig. 7 and Appendix F as supporting evidence.Source: §5 · p. 12
  • For mixed or shifting domains, consensus filtering prunes long CoTs and disproportionately hurts gPRM; distance and per-domain improvement correlate at −0.625.The authors recommend gORM as the default in this setting.Source: §5 · p. 12
  • Under high label noise, ORMs are safer: dPRM is sensitive to per-step label flips, while gPRM inherits a monotonicity assumption that amplifies early errors.The recommendation is based on the PRM800K noise-injection study.Source: §5 · p. 12
  • Discriminative variants are preferred under strict compute or latency budgets; on MMLU-Pro, normalized inference times are 0.0005/0.0005/0.0571/0.0306 seconds per CoT for dORM/dPRM/gORM/gPRM.Generative variants use M rollouts; gPRM is 1.87× faster than gORM but may miss “aha” CoTs.Source: §5 · p. 12
  • The study trains models via supervised fine-tuning, uses LoRA rather than full-parameter fine-tuning, and excludes tool use; these choices may affect performance or scaling behavior.The authors expect qualitative trends to hold and plan to study broader domains, model families, training regimes, and tool-augmented verification.Source: §5 · pp. 12–13

Broader Impact Statement

The work evaluates verification strategies on public benchmarks without human subjects, user studies, or collection of personally identifiable information. Its models and methods are research artifacts, not tools for high-stakes decisions.

Broader Impact Statement

  • The study uses publicly available benchmarks and involves no human subjects, user studies, or collection of personally identifiable information.The authors state that, to their knowledge, the datasets contain no sensitive personal data.Source: Broader Impact Statement · p. 13
  • Legal- and medical-themed experiments do not establish readiness for real-world high-stakes decisions; deployment requires domain-specific validation, safety auditing, and regulatory compliance.The authors say these research artifacts should not substitute for professional judgment.Source: Broader Impact Statement · p. 13
  • The authors describe verification in real systems as a future direction for increasing output trustworthiness and reducing reasoning errors and hallucinations.Source: Broader Impact Statement · p. 13

Reproducibility Statement

The reproducibility statement describes released materials and artifacts, while the appendix overview points readers to supplementary theory, prompts, datasets, implementation details, examples, and results.

Reproducibility Statement

  • The authors report providing prompts, hyperparameters, model backbones, training details, and synthetic-data generation details, and publicly releasing code, datasets, and model checkpoints.Source: Reproducibility Statement · p. 13
  • Released materials include discriminative and generative training/inference code and MMLU-Pro training and test datasets for dORM, dPRM, gORM, and gPRM.The dORM/dPRM code and training data are adapted from VersaPRM (Zeng et al., 2025).Source: Reproducibility Statement · pp. 13–14

Appendix Overview

  • Appendices A–G cover theoretical proofs, prompts, datasets, implementation details, training examples, detailed MMLU-Pro results, and additional analysis.Appendix F includes per-domain MMLU-Pro results using weighted majority voting; Appendix G gives complete §4 results.Source: Appendix Overview · p. 20

A.1 Analysis on Log-Error Bound

The analysis models outcome and stepwise reward errors and derives how their log-error bounds depend on reasoning-chain length. Under its stated assumptions, ORM bounds do not depend on length, whereas PRM bounds grow with it.

A.1 Analysis on Log-Error Bound

  • The analysis defines the true reward as the probability that all reasoning steps are correct and models dPRM aggregation as a product of stepwise conditional predictions.For gPRM, the aggregate prediction averages a product of stepwise correctness probabilities over verification chains.Source: §A.1 · p. 20
  • The lower-bound analysis assumes a variance floor, bounded local error dependence, and a positive slope condition; the local-dependence assumption allows recent self-correction but makes distant steps conditionally independent.Source: §A.1 · p. 21
  • Under the stated assumptions, dPRM and sampled gPRM log-error grows at least linearly with CoT length T, while ORM error bounds are independent of T when conditional noise is bounded.For mean-gPRM, a Jensen gap adds nonnegative bias that scales with the variance of L and hence with T.Source: §A.1 · p. 22

Proof of Theorem A.1

  • Theorem A.1’s proof uses variance decomposition and local dependence to bound accumulated error variance as the chain grows.The proof counts only covariance pairs within the local context window.Source: Proof of Theorem A.1 · p. 22

Proof of Theorem A.3

  • Theorem A.3’s proof expands the variance of sampled gPRM error using the average variance floor and local-dependence assumptions, then takes expectations to obtain the bound.Source: Proof of Theorem A.3 · p. 23

Proof of Theorem A.4; B Prompts

  • Theorem A.4’s proof uses exponential tilting and a Jensen-gap identity; in the calibrated case, the stated simplified bounds follow.The section also presents prompt formats for CoT generation, process-label auto-labeling, and gORM and gPRM training or evaluation.Source: Proof of Theorem A.4 · pp. 23–24, B Prompts · p. 24

C Dataset

The section describes the math and multi-domain datasets used for training and evaluation. The multi-domain benchmark spans 14 subjects, with CoTs generated by several language models.

Math Datasets

  • PRM800K supplies human-annotated process labels for math training, and ProcessBench supplies four math test splits for outcome verification.The splits contain CoTs from GSM8K, Math, Omni-Math, and OlympiadBench; the study also generates 16 CoTs per question for test-time scaling.Source: C Dataset · p. 28

Multi-domain datasets

  • MMLU-Pro covers 14 domains with 5,750 training and 2,063 evaluation questions.The listed domains range from law and psychology to computer science and other.Source: C Dataset · p. 28, Table 1 · p. 28
  • MMLU-Pro training and evaluation CoTs come from Llama-3.1-8B-Instruct, with process labels automatically assigned using Llama-3.1-70B-Instruct.The setup uses 16 CoTs per training question and 128 per evaluation question.Source: C Dataset · p. 28
  • Additional evaluation CoTs from four other model families test generalization across different pLLM outputs.The models are SmolLM3-3B, Qwen2.5-7B-Instruct, gemma-2-9b-it, and Llama-3.1-70B-Instruct.Source: C Dataset · p. 28

D Implementation Details

The implementation uses distinct reward-model backbones for math and multi-domain experiments, shared fine-tuning settings, and filtered verification-CoT training data. Training-example figures illustrate gORM and gPRM examples in law.

D Implementation Details

  • Reward-model backbones differ by setting: distilled Qwen models are used for math, while distilled Llama and Qwen models are used for multi-domain experiments.Qwen3-8B is also used to test whether multi-domain results hold with a non-distilled backbone; dORM and dPRM use the same backbones for fair comparison.Source: D Implementation Details · p. 29
  • The experiments use LoRA fine-tuning, AdamW, and shared hyperparameters without exhaustive tuning.The authors report means over five runs except for R1-Distill-Llama-8B and Qwen3-8B backbone experiments; a learning-rate and LoRA-rank sweep is also reported for dPRM/gPRM.Source: D Implementation Details · p. 29, E Training Examples · p. 29, Table 2 · p. 29
  • gORM and gPRM training uses four QwQ-32B verification CoTs per question-and-CoT pair, then filters examples for parseability, language, length, and label consistency.The resulting multi-domain training sets contain 171,780 gORM and 94,156 gPRM CoTs; math sets contain 34,286 and 35,666, respectively.Source: D Implementation Details · p. 29

E Training Examples

F Detailed Results on MMLU-Pro

The section presents detailed MMLU-Pro results for Best-of-N and weighted majority voting across CoT generators and reward-model backbones. It also includes experiments trained and evaluated within individual domains.

F Detailed Results on MMLU-Pro

G Additional Analysis

The additional analyses examine PRM failure, filtering and label-refinement interventions, sensitivity to training settings, and results across CoT generators and reward-model backbones.

G Additional Analysis

  • The section examines PRM failure and compares gORM and gPRM Wasserstein distances before and after filtering in math and multi-domain settings.The multi-domain analysis also describes label refinement and relaxed consensus filtering as interventions to reduce gPRM CoT-length distribution shift.Source: G Additional Analysis · p. 37, Table 4 · p. 38, Table 5 · p. 38
  • Additional Best-of-N results cover CoTs from SmolLM3-3B, Qwen2.5-7B-Instruct, gemma-2-9b-it, and Llama-3.1-70B-Instruct.A separate figure reports overall results with five pLLMs and a Qwen3-8B reward-model backbone.Source: G Additional Analysis · p. 37
  • A further table reports overall Best-of-N results while varying learning rate and LoRA rank for PRM variants.The caption specifies Llama-3.1-8B-Instruct CoTs and an R1-Distill-Qwen-14B reward-model backbone.Source: G Additional Analysis · p. 37, Table 3 · p. 38
  • Tables compare gORM and gPRM CoT survival proportions and Best-of-N results under label refinement or relaxed filtering on MMLU-Pro.The captions identify the training split for survival proportions and Llama-3.1-8B-Instruct CoTs for the intervention results.Source: G Additional Analysis · p. 37, Table 6 · p. 39, Table 7 · p. 39
Loading 2510.00492v3…