Source-linked AI summary
ProcessBench: Identifying Process Errors in Mathematical Reasoning
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
TL;DR
Language models still make mistakes on challenging mathematics, motivating automated assessment of their reasoning processes for scalable oversight. ProcessBench benchmarks earliest-error identification using difficult, diverse solutions with expert annotations and evaluates PRMs alongside prompted critics. Existing PRMs struggle to generalize to harder problems, while QwQ-32B-Preview is competitive with GPT-4o but behind o1-mini.
Problem
Language models make mistakes on challenging mathematics, creating a need to assess reasoning errors automatically for scalable oversight.
Method
ProcessBench contains 3,400 difficult math test cases with diverse model-generated solutions and expert error annotations, evaluated using PRMs and prompted critic models.
Results
Existing PRMs underperform critic models and struggle more on challenging mathematics, while QwQ-32B-Preview is competitive with GPT-4o but behind o1-mini.
Takeaways & Limitations
ProcessBench provides a benchmark for future research on automated reasoning-process assessment and scalable oversight.
Takeaways & Limitations
The benchmark may contain inaccurate error-location labels, especially for Olympiad-level problems, and discarded solutions may bias its problem distribution.
Abstract
from arXiv · showhide
As language models regularly make mistakes when solving math problems, automated identification of errors in the reasoning process becomes increasingly significant for their scalable oversight. In this paper, we introduce ProcessBench for measuring the ability to identify erroneous steps in mathematical reasoning. It consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems. Each test case contains a step-by-step solution with error location annotated by human experts. Models are required to identify the earliest step that contains an error, or conclude that all steps are correct. We conduct extensive evaluation on ProcessBench, involving two types of models: process reward models (PRMs) and critic models, where for the latter we prompt general language models to critique each solution step by step. We draw two main observations: (1) Existing PRMs typically fail to generalize to more challenging math problems beyond GSM8K and MATH. They underperform both critic models (i.e., prompted general language models) and our own trained PRM that is straightforwardly fine-tuned on the PRM800K dataset. (2) The best open-source model, QwQ-32B-Preview, has demonstrated the critique capability competitive with the proprietary model GPT-4o, despite that it still lags behind the reasoning-specialized o1-mini. We hope ProcessBench can foster future research in reasoning process assessment, paving the way toward scalable oversight of language models.
1 Introduction
PROCESSBENCH evaluates whether models can identify the earliest erroneous step in challenging mathematical solutions, using diverse model-generated solutions and expert annotations. Evaluations show that existing PRMs struggle on harder problems, while prompted critics demonstrate stronger critique performance.
- Benchmark design: Figure 2 uses label 2 for an earliest error in the second step, indexed from 0, and −1 when no error exists.This label convention illustrates the benchmark’s target prediction.
- Benchmark design: PROCESSBENCH measures whether models can identify the earliest erroneous step in a mathematical solution or determine that all steps are correct.The benchmark is designed for process reward models and critic models.
- Benchmark design: 3,400 test cases cover primarily competition- and Olympiad-level mathematics with diverse solution styles and expert-annotated error locations.Multiple human experts annotate the solutions to support evaluation reliability.
- Evaluation findings: Existing PRMs typically fail to generalize beyond GSM8K and MATH and underperform critic models and a PRM fine-tuned on PRM800K.The comparison raises questions about the generalization and scalability of current PRM data-synthesis methods.
- Evaluation findings: QwQ-32B-Preview is competitive with GPT-4o among evaluated critics but still lags behind the reasoning-specialized o1-mini model.General language models can identify erroneous steps and provide detailed explanations when prompted to critique solutions.
2 Related Work
Prior work includes critic benchmarks, synthetic error-detection data, and PRM datasets, but PROCESSBENCH targets more difficult and diverse mathematical reasoning with large-scale expert annotation. Its design addresses evaluation needs arising from increasingly capable language models.
- Prior benchmarks: CriticBench, MathCheck, and PRM800K assess reasoning-process evaluation through critique, synthetic erroneous solutions, or annotations of solution-step correctness.These resources cover various reasoning tasks, GSM8K, or MATH problems.
- PROCESSBENCH distinctions: PROCESSBENCH primarily uses competition- and Olympiad-level problems, exceeding the difficulty emphasis of prior related benchmarks.The authors position this difficulty as better aligned with modern language-model capabilities.
- PROCESSBENCH distinctions: PROCESSBENCH uses diverse natural model-generated solutions and expert error annotation rather than relying only on synthetic data.The authors associate this design with real-world applicability and label accuracy.
- PROCESSBENCH distinctions: 3,400 test cases make PROCESSBENCH substantially larger and support more comprehensive evaluation than the compared resources.Table 1 compares its scale with related reasoning-process datasets.
3 Benchmark Construction
PROCESSBENCH evaluates whether models can locate the earliest erroneous step in mathematical solutions, using difficult and diverse problems, standardized steps, and expert annotations. Its statistics show that process errors are more common on difficult problems even when final answers are correct.
- Benchmark Task: PROCESSBENCH requires models to identify the earliest erroneous step or conclude that all solution steps are correct.The task represents all-correct solutions with index -1 and errors by the index of the earliest erroneous step.
- Error Definition: The benchmark covers mathematical, logical, conceptual, and completeness errors that affect the solution.Later steps may remain locally valid while following an incorrect path, motivating evaluation of the earliest error.
- Data and Solutions: Problems come from GSM8K, MATH, OlympiadBench, and Omni-MATH, with the latter three containing competition- or Olympiad-level problems.Solutions are generated by twelve Qwen and LLaMA models spanning families, sizes, and downstream performance levels.
- Data and Solutions: Solution reformatting standardizes step granularity by asking Qwen2.5-72B-Instruct to segment paragraphs while preserving their content.Solutions whose final answers changed after reformatting were removed; the reported alteration rate was below 0.5%.
- Expert Annotation: Doctoral-level experts annotate the earliest error, using reference solutions and answers while inspecting generated solutions step by step.Each solution begins with three annotators; unresolved cases receive additional annotation and are discarded if consensus is not reached within five annotators, producing an overall approximately 30% discard rate.
4 Evaluation
The evaluation compares PRMs and prompted critic models on PROCESSBENCH, using F1 to balance performance on erroneous and correct samples. Performance declines as math difficulty increases; existing PRMs generally generalize worse than critic models, while QwQ-32B-Preview is competitive with GPT-4o but behind o1-mini.
- 4.1 Setup: F1 is the primary metric because it combines accuracies on erroneous and correct samples, balancing over-criticism against failure to identify errors.For scalar-output PRMs, thresholds are selected using the GSM8K subset before extracting the earliest erroneous step.
- 4.1 Setup: The evaluation covers process reward models and critic models, including open-source and proprietary language models.Critic models are prompted to critique solutions step by step and return the earliest error index.
- 4.2 Results: As benchmark difficulty rises from GSM8K and MATH to OlympiadBench and Omni-MATH, all models show a consistent performance decline.This pattern indicates a shared generalization challenge for both PRMs and critic models.
- 4.2 Results: Existing PRMs generally underperform top prompted critic models and decline more sharply on OlympiadBench and Omni-MATH.The comparison raises concerns about the generalization and scalability of current PRM data-synthesis methodologies.
- 4.2 Results: Qwen2.5-Math-7B-PRM800K shows significantly stronger performance and generalization than other PRMs after straightforward fine-tuning on the human-annotated PRM800K dataset.The paper attributes weaker process labels in other PRMs partly to errors in model-generated reasoning, including solutions with correct final answers.
- 4.2 Results: QwQ-32B-Preview performs best among open-source models and is highly competitive with GPT-4o, yet still lags behind o1-mini.Its critiques can identify erroneous steps while providing detailed thinking and explanations.
5 Conclusion
ProcessBench is presented as a benchmark for assessing erroneous steps in mathematical reasoning and advancing automated reasoning-process assessment. The benchmark may still contain inaccurate error-location labels, especially for challenging Olympiad-level problems, and discarded solutions may bias its problem distribution.
- 5 Conclusion: ProcessBench measures erroneous-step identification with difficult, diverse problems, large scale, rigorous human annotation, and a simple evaluation protocol.Its intended role is to support progress in automated reasoning-process assessment and scalable oversight.
- 5 Conclusion: ProcessBench may contain inaccurate error-location labels, particularly for challenging Olympiad-level mathematics.This limitation concerns benchmark-label reliability despite the construction and annotation procedures.
- 5 Conclusion: Discarded solutions may bias the benchmark’s problem distribution because they can involve problems exceeding human annotators’ capabilities.The paper identifies this as an additional construction limitation.
A Example of Solution Reformatting
The section presents an example of reformatting a solution to an ellipse problem from the MATH test set, contrasting an original solution with a reformatted version.
- The standard ellipse equation uses (h, k) as the center and a and b as the semi-major and semi-minor axis lengths.
- Figure 6 contrasts an original solution generated by Qwen2-7B-Instruct with a reformatted solution.
- The example problem asks for the focus with the larger x-coordinate and requires the answer as an ordered pair.
B Breakdown Statistics of PROCESSBENCH
PROCESSBENCH’s breakdown statistics include dataset-generation caveats affecting specific model counts and annotation coverage. A code bug produced zero MATH-subset counts for two models, while lower-accuracy models were excluded from later annotation on harder subsets.
- A code bug caused Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct to have zero counts in PROCESSBENCH’s MATH subset.
- The bug occurred when those two models generated solutions for the MATH problems.
- Models with lower accuracies were excluded from subsequent annotation for OlympiadBench and Omni-MATH.
C Training Details of Qwen2.5-Math-7B-PRM800K
Qwen2.5-Math-7B-PRM800K is created by fine-tuning Qwen2.5-Math-7B-Instruct on PRM800K with a binary reward-modeling head and labels derived from the original annotations.
- Qwen2.5-Math-7B-PRM800K is obtained by fine-tuning Qwen2.5-Math-7B-Instruct on the PRM800K training set.
- The original language-modeling head is replaced with a reward-modeling head that outputs binary classification logits.
- The classification loss is computed at second-line-break positions in all occurrences of “\n\n”.
- PRM800K labels 1 and 0 are treated as positive, while -1 is treated as negative.
D Inference Details
Inference settings vary by evaluation stage and model group, with specified sampling parameters for solution generation and majority voting. Evaluation inference was run with vLLM on eight A100 80GB GPUs.
- Solution generation uses p = 0.9 and t = 0.7 for all models.
- For majority voting, Qwen2.5-Math-7/72B-Instruct uses p = 0.8, t = 0.7, and k = 20.
- All other models use only p = 0.9 for majority-voting evaluation.
- All evaluation inference runs use vLLM on eight A100 80GB GPUs.
E Prompt Template for Critic Model Evaluation
The critic-model evaluation prompt separates the math problem and solution from instructions specifying the required output content and format.
- Blue text contains the input math problem and solution, split into paragraphs.
- Red text specifies the required output content and format for the critique.
F Supplementary Evaluation Results
The supplementary tables break down PROCESSBENCH results across four subsets and report critic-model results under greedy decoding, alongside threshold-adjusted PRM upper bounds.
- The GSM8K and MATH breakdown evaluates open-source language models using majority voting over eight samplings.
- Threshold-adjusted Skywork PRM results on GSM8K and MATH represent upper-bound performance and are marginally higher than Table 5 results.
- The OlympiadBench and Omni-MATH breakdown evaluates open-source language models using majority voting over eight samplings.
- Threshold-adjusted Skywork PRM results on OlympiadBench and Omni-MATH represent upper-bound performance and are slightly higher than Table 7 results.
- A separate table reports open-source language-model results when prompted as critic models using greedy decoding.