PaperlayerView exact revision on arXiv

Source-linked AI summary

PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models

Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng

arXiv:2501.03124v5cs.CLcs.AIcs.LG

TL;DR

Existing PRMs and benchmarks do not systematically detect the diverse implicit errors that arise during multi-step reasoning. PRMBench addresses this gap with a fine-grained benchmark and evaluation of process-labeling models, finding partial capabilities and substantial weaknesses. The results motivate more detailed error detection and comprehensive PRM evaluation.

  • Problem

    Existing PRMs and process-level benchmarks have limited ability to identify diverse implicit reasoning errors and often focus on single-error or step-correctness evaluation.

  • Method

    PRMBench curates fine-grained reasoning cases with LLM generation and human or structural filtering, covering three major categories and nine sub-categories.

  • Results

    PRMs show partial multi-step process-evaluation capability; Gemini-2-Thinking scores 68.8, below human performance of 83.8.

  • Takeaways & Limitations

    PRMBench supports detailed evaluation of process-labeling models and highlights the need to detect fine-grained error types comprehensively.

  • Takeaways & Limitations

    A larger dataset could provide more robust evaluation and training opportunities, and the current accuracy-based analysis does not examine model neurons or hidden states.

Abstract

from arXiv · show

Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.

1 Introduction

PRMBench addresses the limited ability of existing process reward models to detect diverse reasoning errors by evaluating fine-grained error categories. Experiments show partial multi-step evaluation capability and substantial room for improvement.

  • Motivation: PRMs can miss implicit reasoning errors or produce false positives, as illustrated by failures on redundant, inconsistent, and deceptive steps.Math-Shepherd-7B misses one error, while ReasonEval-34B flags a correct step incorrectly.
  • Benchmark: PRMBench evaluates PRMs across three major categories and nine sub-categories, including simplicity, soundness, and sensitivity.The benchmark is designed to assess diverse error-detection capabilities rather than a single error type.
  • Experiments: 68.8 is Gemini-2-Thinking’s best performance, below human performance of 83.8 on multi-step process evaluation.The study evaluates 25 models, including dedicated PRMs and general-purpose or mathematical LLMs prompted as critics.
  • Benchmark: 6,216 samples and 83,456 step-level labels comprise PRMBench, with data quality supported by professional annotators.The benchmark provides fine-grained process-level evaluation data for reward models.
  • Resources: The authors release PRM-EVAL, an automated evaluation framework and customizable data generation system for future PRM research.They position the toolkit and benchmark as resources for improving step-level reasoning evaluation and PRM reliability.

2 Related Work

Prior work evaluates reasoning processes and develops process-level reward models, but existing approaches do not fully cover implicit errors or fine-grained step-level distinctions. PRMBench extends this line of work with detailed evaluation categories and representative examples.

  • Process-level Reward Models: Process-level reward models improve process-level reasoning accuracy and long-process reasoning abilities over traditional outcome-level reward models.Prior work includes multi-step reasoning datasets, automatic self-supervised labeling, and PRM-based auto-evaluation.
  • PRMBench: PRMBench organizes evaluation into detailed sub-categories and provides representative examples of each category in its overview.The benchmark’s structure and examples are described in Figure 2 and further detailed in the appendix.
  • Reasoning Evaluation: ROSCOE introduced semantic comparison-based evaluation for multi-step reasoning accuracy, while later work used PRMs or critic models for step-level evaluation.These approaches address the challenge that labeled data may not exhaustively cover all solution paths.
  • Benchmark Gap: Existing process-level benchmarks typically evaluate only a single error type, whereas PRMBench assesses diverse error categories systematically.The benchmark focuses on fine-grained detection of reasoning-process errors.

3 PRMBENCH

PRMBENCH evaluates process-level reward models across three domains—simplicity, soundness, and sensitivity—using fine-grained sub-categories for diverse reasoning errors. It constructs and filters test cases from ground-truth solutions, then applies structural checks and human verification to maintain dataset quality.

  • Evaluation Categories: PRMBENCH organizes evaluation into simplicity, soundness, and sensitivity, with nine sub-categories for granular assessment.The sub-categories cover redundancy, circular logic, several consistency and confidence errors, prerequisite sensitivity, deception resistance, and multi-solution consistency.
  • Simplicity: Simplicity tests whether PRMs detect redundant or circular reasoning steps that reduce concision and efficiency without necessarily changing correctness.Its sub-categories are Non-Redundancy and Non-Circular Logic.
  • Soundness: Soundness tests reward accuracy across empirical mistakes, step contradictions, domain inconsistencies, and over-confident incorrect statements.These four sub-categories target distinct forms of reasoning failure, including conflicts between steps and statements invalid in the current domain.
  • Sensitivity: Sensitivity evaluates robustness to missing prerequisites, deceptive alterations, and multiple correct solution paths for the same question.These tests target implicit conditions, subtly misleading statements, and consistency across alternative reasoning processes.

4 Experiments

PRMBENCH evaluates a broad set of PRMs and critic models using normalized process-level metrics, revealing substantial gaps in multi-step error detection and category-specific reliability.

  • 25 models, including dedicated PRMs and prompted general-purpose or mathematical LLMs, are evaluated on PRMBENCH.
  • 4.2 Evaluation Metrics: PRMScore combines F1 and negative F1 through weights designed to maximize differentiation between models.
  • 4.3 Main Results: 68.8 is Gemini-2-Thinking’s best performance, compared with 83.8 for humans, leaving a substantial multi-step evaluation gap.
  • 4.3 Main Results: Some models perform worse than random guessing, while the best open-source PRMs fail to match proprietary general-purpose LLMs.
  • 4.3 Main Results: ReasonEval-34B reaches 73.1 in Sensitivity and 97.2 on Multi-Solutions but falls to 51.5 in Simplicity.
  • PRMBENCH-STEM extends evaluation to STEM-related domains, with its construction and results reported in the appendix.

5 Detailed Analysis

Detailed analyses identify positive-reward bias, position-dependent PRM behavior, limited few-shot effects, and weak alignment between step-level evaluation and outcome selection.

  • 5.1 Inference Bias within PRMs: ReasonEval-7B and RLHFlow-DeepSeek-8B exceed 95% accuracy on positive steps but average only 17% on negative steps.
  • 5.1 Inference Bias within PRMs: Certain models show reward similarity above the normal similarity score of 79.4, indicating difficulty differentiating positive and negative steps.
  • 5.2 Error Position Analysis: Error positions peak at step 5 and decline thereafter; proprietary LLMs remain stable across positions, unlike some PRMs.
  • 5.3 Impact of In-Context Learning: Few-shot prompting produces subtle gains for Gemini models, while GPT-4o shows no significant improvement and sometimes declines with more examples.
  • 5.4 Comparison between BoN Evaluation and PRMBench: The average Somers’ D correlation between PRMBench and BoN is -0.05, showing inconsistent step-level and outcome-level evaluation.
  • 5.4 Comparison between BoN Evaluation and PRMBench: PRMBENCH distinguishes models more sharply than BoN, with standard deviation 6.40 versus 0.81, while exposing false-positive detection weaknesses.

6 Conclusion

The paper introduces PRMBENCH to evaluate whether PRMs detect diverse erroneous reasoning steps and provide reasonable rewards. Its evaluation finds only partial multi-step process-evaluation capability, leaving reward accuracy and reasoning improvement open.

  • PRMBENCH evaluates PRMs using fine-grained categories and challenging requirements.
  • The benchmark contains 6,216 data samples and 83,456 step-level labels curated through LLMs and human filtering.
  • Existing PRMs and critic LLMs exhibit partial capability in multi-step process evaluation.
  • The findings highlight the need to detect detailed error types and conduct comprehensive PRM evaluations.
  • Improving PRM reward accuracy and model reasoning abilities remains an open research challenge.

8 Limitations

The paper identifies dataset scale, evaluation depth, and reasoning modality as limitations. It also documents a benchmark organized around fine-grained categories and sub-categories for textual reasoning evaluation.

  • A larger PRMBENCH dataset could provide more robust evaluation and training opportunities.The current benchmark comprises 6,216 samples and 83,456 step-level labels.
  • The study evaluates error detection primarily through accuracy rather than neuron- and hidden-state analyses.The authors state that deeper analyses could provide more insight into PRM improvements.
  • PRMBENCH currently focuses on textual reasoning and does not yet evaluate multimodal reasoning processes.Future work is intended to examine scenarios integrating text and images.
  • The benchmark organizes evaluation into simplicity, soundness, and sensitivity categories with multiple sub-categories.Examples include non-redundancy, non-circular logic, empirical soundness, step consistency, domain consistency, confidence invariance, prerequisite sensitivity, deception resistance, and multi-solution consistency.

A.3.2 Human Performance Evaluation

Human performance is estimated with annotations from three volunteers on a 450-sample mini-test set. The broader evaluation includes diverse publicly available PRMs and critic models used under their original licenses.

  • Three volunteers annotated a 450-sample mini-test set for human performance evaluation.The set randomly selected 50 instances from each sub-category.
  • Each human annotator was responsible for three subsets of the mini-test set.
  • The model evaluation includes open-source PRMs and multiple types of LLM critic models.Listed PRMs include Skywork-PRM, LlemmaPRMs, MathMinosPRM, MathShepherd-Mistral, ReasonEval, Pure-PRM, and Qwen-PRM variants.

B.3 Evaluation Procedure

The evaluation procedure converts step-level validity and redundancy judgments into binary predictions using model-specific thresholds. The benchmark is also extended to scientific domains and accompanied by detailed metric results.

  • Reward models assign each step a validity score and a redundancy score for annotated question-solution pairs.
  • Model-specific thresholds convert step-level scores into predictions of whether each step is correct or redundant.The resulting task is framed as binary classification.
  • Detailed results across sub-categories and metrics are reported in Tables 15–23 in addition to PRMScore.
  • PRMBENCH-STEM extends the benchmark to biology, chemistry, and physics using a similar data-curation methodology.Representative PRMs are evaluated on the three scientific-domain subsets.

C.1 Error Analysis

The analysis illustrates that models often assign positive rewards to erroneous reasoning steps, motivating fine-grained process evaluation beyond final-answer performance. It also defines modified-process judgments around reasonableness and logical difference from the original solution.

  • Error Analysis: Most models positively reward counterfactual steps that contradict correct computational principles, with GPT-4o the notable exception.The example concerns steps eight through thirteen in a counterfactual reasoning process.
  • Motivation: The broader motivation is that apparently correct multi-step solutions can contain understanding, calculation, or reasoning errors that existing PRMs do not reliably detect.These errors create false-positive situations during process evaluation.
  • Data Construction: The benchmark construction starts from correct multi-step solutions, injects error types with advanced LLMs, and filters the resulting instances using specific features.The detailed error types are described separately in the benchmark methodology.
  • Evaluation Criteria: PRMBench evaluates process-level judgments by asking whether an LLM-modified solution is reasonable and logically different from the original.Reasonableness requires both a possible solution path and an error appropriate to the classification.

D.2.2 Error Types

The benchmark organizes errors into fine-grained reasoning failures, including redundancy, contradictions, counterfactuals, domain misuse, hallucination, and missing conditions. These categories target distinct ways a reasoning process can become invalid while appearing plausible.

  • Simplicity Errors: Redundancy adds removable steps without affecting the overall solution path, while circular logic loops from a step back to itself.Circular logic is represented as S → A → B → S.
  • Soundness Errors: Counterfactual steps contradict ground truth or established theories, including errors caused by outdated theories, omitted constraints, or erroneous assumptions.Domain inconsistency is a special counterfactual involving statements valid elsewhere but not in the current reasoning chain.
  • Soundness Errors: Step contradiction occurs when one reasoning step conflicts with another distinct step in the same reasoning path.The formal condition is Si ⊥ Sj for distinct indices i and j.
  • Soundness Errors: Confident hallucination presents a statement contradicting established ground truth with unwarranted certainty, preserving an appearance of correctness.Deception or traps similarly introduce inaccuracies while maintaining the illusion of correctness.
  • Sensitivity Errors: Missing conditions or prerequisites omit critical premises or assumptions, creating logical gaps and potentially incorrect conclusions.Such cases may require case analysis or further investigation before standard reasoning can proceed.

F Further Discussion

The discussion proposes training and evaluation directions based on PRMBench’s observed weaknesses. It emphasizes fine-grained error-sensitive training and step-level evaluation to address discrepancies between process judgments and outcome-based assessment.

  • Anti-redundancy training: Reducing Neutral samples in PRM training data is proposed as one way to strengthen detection of redundant reasoning steps.Neutral labels often correspond to redundant steps.
  • Contrastive training: Fine-grained error-labeled data can support contrastive learning or preference alignment to improve PRM error sensitivity and detection.The proposed data-curation pipeline is presented as adaptable for creating such training samples.
  • Step-level evaluation for LLMs: Inconsistency between PRMBench and Best-of-8 evaluation reveals false-positive situations and reward-hacking risk in language-model post-training.The discussion uses this discrepancy to motivate broader process-level assessment.
  • Benchmark analysis: The benchmark includes detailed evaluations across categories and subcategories, with representative examples and model-comparison tables covering multiple domains.The structure includes category-level and subcategory-level performance comparisons.
Loading 2501.03124v5…