PaperlayerView exact revision on arXiv

Source-linked AI summary

ProcessBench: Identifying Process Errors in Mathematical Reasoning

Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin

arXiv:2412.06559v4cs.AIcs.CLcs.LG

TL;DR

Automated identification of errors in mathematical reasoning matters for scalable oversight, but existing benchmarks may not meet growing evaluation demand. ProcessBench evaluates earliest-error identification in expert-annotated competition- and Olympiad-level math solutions; its results show PRMs underperform critics, while QwQ-32B-Preview is competitive with GPT-4o but trails o1-mini.

  • Problem

    Existing benchmarks may not meet growing demand for evaluating automated identification of errors in language-model reasoning, a capability relevant to scalable oversight.

    Source: §1 · p. 2

  • Method

    ProcessBench contains 3,400 competition- and Olympiad-level math solution cases with expert-annotated error locations and asks models to identify the earliest erroneous step, if any.

    Source: §1 · p. 2

  • Results

    PRMs underperform critic models and generalize less well to harder math; QwQ-32B-Preview is competitive with GPT-4o but trails o1-mini.

    Source: §1 · p. 3, §4.2 · pp. 8–9, §5 · p. 9

  • Takeaways & Limitations

    The authors hope ProcessBench will foster research in reasoning-process assessment toward scalable oversight.

    Source: §1 · p. 3, §5 · p. 9

    ProcessBench may contain inaccurate error-location labels, especially for Olympiad problems, and discarded annotations may bias the problem distribution toward problems within annotators’ capabilities.

    Source: §5 · p. 9

Abstract

from arXiv · show

As language models regularly make mistakes when solving math problems, automated identification of errors in the reasoning process becomes increasingly significant for their scalable oversight. In this paper, we introduce ProcessBench for measuring the ability to identify erroneous steps in mathematical reasoning. It consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems. Each test case contains a step-by-step solution with error location annotated by human experts. Models are required to identify the earliest step that contains an error, or conclude that all steps are correct. We conduct extensive evaluation on ProcessBench, involving two types of models: process reward models (PRMs) and critic models, where for the latter we prompt general language models to critique each solution step by step. We draw two main observations: (1) Existing PRMs typically fail to generalize to more challenging math problems beyond GSM8K and MATH. They underperform both critic models (i.e., prompted general language models) and our own trained PRM that is straightforwardly fine-tuned on the PRM800K dataset. (2) The best open-source model, QwQ-32B-Preview, has demonstrated the critique capability competitive with the proprietary model GPT-4o, despite that it still lags behind the reasoning-specialized o1-mini. We hope ProcessBench can foster future research in reasoning process assessment, paving the way toward scalable oversight of language models.

1 Introduction

ProcessBench evaluates whether models can locate the earliest error in mathematical reasoning, using challenging problems and expert-annotated solutions. Its evaluation finds that general language models can critique steps effectively, while existing PRMs struggle to generalize beyond GSM8K and MATH.

1 Introduction

  • Existing process-evaluation benchmarks may use problems that recent models find less challenging or label only final-answer correctness, not erroneous steps.The paper motivates automated error identification as a means of scalable oversight for complex tasks.Source: §1 · p. 2
  • ProcessBench contains 3,400 expert-annotated cases, primarily competition- and Olympiad-level problems with solutions generated by diverse open-source language models.Multiple human experts annotate error locations to support data quality and evaluation reliability.Source: §1 · p. 2
  • Models must identify the earliest erroneous step, or conclude that all steps are correct, a simple protocol adaptable to PRMs and critic models.The benchmark labels the earliest error with a zero-indexed step number and uses −1 when no error occurs.Source: §1 · p. 2, Figure 2 · p. 2
  • Existing PRMs underperform prompted critics and a PRM fine-tuned on PRM800K; QwQ-32B-Preview is competitive with GPT-4o but trails o1-mini.General language models can also provide detailed explanations of identified errors.Source: §1 · pp. 2–3

2 Related Work

Prior benchmarks address critique, error judging, and step correctness; PROCESSBENCH targets competition- and Olympiad-level math with diverse natural solutions and expert-labeled errors.

2 Related Work

3 Benchmark Construction

PROCESSBENCH asks models to locate the earliest erroneous step or conclude that a solution is fully correct. Its 3,400 cases use expert-annotated solutions, and process errors are more common on more difficult subsets even when final answers are correct.

3.1 Task Definition

  • Given a problem and its step-by-step solution, models output the index of the earliest error, or −1 if all steps are correct.Errors may be mathematical, logical, conceptual, or completeness-related; PROCESSBENCH focuses on the earliest error because later steps can be ambiguous when they follow an incorrect premise.Source: §3.1 · p. 4

3.2 Data Collection

  • Problems come from GSM8K, MATH, OlympiadBench, and Omni-MATH, while twelve Qwen- and LLaMA-series models generate solutions with diverse styles.GSM8K contains grade-school problems; the other three datasets contain competition- or Olympiad-level problems.Source: §3.2 · p. 4
  • To standardize step granularity, the authors use Qwen2.5-72B-Instruct to resegment solutions into logically complete, progressive steps and discard cases whose final answers change.The method replaces line breaks with spaces before asking the model to insert double line breaks; content changes occurred in fewer than 0.5% of solutions.Source: §3.2 · p. 4
  • Doctoral-level experts inspect solutions step by step, using dataset reference solutions and answers; unresolved cases after up to five annotators are discarded.Each solution starts with three annotators, and the process discards about 30% of cases overall; incorrect-final-answer cases annotated as correct are also removed.Source: §3.2 · pp. 4–5

3.3 Statistics

  • PROCESSBENCH comprises 3,400 test cases across four subsets, with errors concentrated in earlier steps, including steps 0–3 in GSM8K and MATH and steps 1–5 in OlympiadBench and Omni-MATH.The benchmark controls the number of solutions with incorrect and correct final answers at 200 each for GSM8K and 500 each for the other subsets.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Among solutions with correct final answers, 32.2% in OlympiadBench and 18.8% in Omni-MATH contained process errors, and process-error rates were consistently higher on more difficult problems.Source: §3.3 · p. 6

4 Evaluation

The evaluation compares PRMs with prompted critic models using F1 scores. Results show stronger performance from the human-trained PRM and top critics, while all models struggle more as math difficulty increases.

4.1 Setup

  • The evaluation uses PRMs and critic models; general language models are prompted to return the paragraph index of the earliest error.The critic evaluation includes open-source Qwen, Qwen2.5, Qwen2.5-Math, Qwen2.5-Coder, LLaMA-3, and QwQ-32B-Preview models, plus GPT-4o and o1-mini.Source: §4.1 · pp. 6–7
  • PRM performance is derived from the earliest predicted error; scalar-scoring PRMs are thresholded using the GSM8K threshold that maximizes F1.F1 is the harmonic mean of accuracy on erroneous and correct samples, balancing excessive criticism against failure to identify errors.Source: §4.1 · pp. 6–7
  • Open-source models are reported with majority voting over eight samples, with greedy-decoding results also reported; GPT-4o uses greedy decoding and o1-mini single sampling.PRMs are included because they assess intermediate reasoning steps, unlike conventional evaluation based only on best-of-N solution performance.Source: §4.1 · pp. 6–7

4.2 Results

  • Qwen2.5-Math-7B-PRM800K outperforms other PRMs, while QwQ-32B-Preview leads open-source critics, is competitive with GPT-4o, and remains behind o1-mini.The overall evaluation reports F1 scores; the supplied passages do not provide the table’s individual values.Source: Table 3 · p. 7, §4.2 · pp. 8–9
  • All models decline as problems grow harder, and existing PRMs underperform top prompted critics on GSM8K and MATH while declining more sharply on OlympiadBench and Omni-MATH.Math-Shepherd’s on-policy process labels (Wang et al., 2024) may not transfer to other models and can be invalidated by process errors even when final answers are correct.Source: §4.2 · pp. 8–9
  • Critics can reason separately before judging steps, and performance scales with model size within a family; QwQ-32B-Preview also has more balanced accuracies on erroneous and correct samples.Figure 5 illustrates QwQ identifying an erroneous step and providing a detailed explanation.Source: Figure 5 · p. 8, §4.2 · p. 9

5 Conclusion

PROCESSBENCH evaluates identification of erroneous steps in mathematical reasoning, and its evaluations find that existing PRMs typically underperform prompt-driven critics while QwQ-32B-Preview is competitive with GPT-4o but lags behind o1-mini. The benchmark may have inaccurate error-location labels and a problem distribution biased by solutions discarded during human annotation.

5 Conclusion

  • Existing PRMs typically underperform critic models at identifying erroneous reasoning steps and struggle more to generalize to challenging math problems; QwQ-32B-Preview is competitive with GPT-4o but lags behind o1-mini.These findings come from extensive evaluations of process reward models and prompt-driven critic models.Source: §5 · p. 9
  • PROCESSBENCH measures the ability to identify erroneous steps in mathematical reasoning, with high problem difficulty, solution diversity, large scale, rigorous human annotation, and a simple evaluation protocol.The authors envision it as a testbed for advancing automated reasoning-process assessment toward scalable oversight of language models.Source: §5 · p. 9
  • PROCESSBENCH may contain inaccurate error-location labels, particularly for challenging Olympiad-level problems, and discarded solutions may bias its problem distribution if they exceeded annotators’ capabilities.The authors note that discarded solutions may involve particularly challenging problems.Source: §5 · p. 9

A Example of Solution Reformatting

The section illustrates reformatting a Qwen2-7B-Instruct solution to a MATH test-set problem asking for the ellipse focus with the larger x-coordinate.

A Example of Solution Reformatting

B Breakdown Statistics of PROCESSBENCH

The section presents PROCESSBENCH breakdown statistics and notes generation and annotation caveats affecting some model counts and inclusion.

B Breakdown Statistics of PROCESSBENCH

  • A code bug caused Llama-3.1-70B-Instruct and Qwen2.5-72B-Instruct to have zero counts in the MATH subset.†The bug occurred when these models generated solutions for MATH problems.Source: Table 4 · p. 12
  • Models with lower accuracies were excluded from subsequent annotation for the more challenging OlympiadBench and Omni-MATH problems.‡Source: Table 4 · p. 12

C Training Details of Qwen2.5-Math-7B-PRM800K

Qwen2.5-Math-7B-PRM800K is produced by fine-tuning Qwen2.5-Math-7B-Instruct on PRM800K with a binary reward-modeling head.

C Training Details of Qwen2.5-Math-7B-PRM800K

  • The model replaces the language-modeling head with a binary reward head and is fine-tuned on PRM800K after removing training samples whose problems appear in ProcessBench.Classification loss is computed at second-line-break positions; PRM800K labels 1 and 0 are treated as positive and -1 as negative. Training used eight A100 80GB GPUs.Source: C Training Details of… · p. 13

D Inference Details

Inference settings varied by task and model, and evaluation runs used vLLM on eight A100 80GB GPUs.

D Inference Details

  • Solution generation used p = 0.9, t = 0.7; majority voting used p = 0.8, t = 0.7, k = 20 for Qwen2.5-Math-7/72B-Instruct and p = 0.9 for other models.The majority-voting settings for Qwen2.5-Math-7/72B-Instruct were chosen to ensure normal generation.Source: D Inference Details · p. 13
  • Evaluation inference ran with vLLM (Kwon et al., 2023) on eight A100 80GB GPUs.Source: D Inference Details · p. 13

E Prompt Template for Critic Model Evaluation

Figure 7 presents the prompt template used to evaluate critic models, distinguishing the math problem and paragraph-split solution from instructions for the required output.

E Prompt Template for Critic Model Evaluation

  • The template supplies the math problem and paragraph-split solution as input, with separate instructions specifying the required output content and format.Figure 7 marks the input text in blue and the output instructions in red.Source: Figure 7 · p. 13

F Supplementary Evaluation Results

The supplementary results break down PROCESSBENCH evaluations across GSM8K, MATH, OlympiadBench, and Omni-MATH, with additional Skywork PRM threshold analyses and greedy-decoding critic results.

F Supplementary Evaluation Results

  • Tables 5 and 7 break down PROCESSBENCH results across GSM8K, MATH, OlympiadBench, and Omni-MATH, evaluating open-source language models by majority voting over eight samplings.Table 5 covers GSM8K and MATH; Table 7 covers OlympiadBench and Omni-MATH.Source: Table 5 · p. 14, Table 7 · p. 15
  • Tables 6 and 8 report Skywork (2024) PRM results using a separate F1-maximizing threshold for each subset, yielding marginally higher GSM8K/MATH and slightly higher OlympiadBench/Omni-MATH results than Tables 5 and 7.The captions describe these subset-specific thresholds as upper-bound performance; Tables 5 and 7 use the GSM8K-selected threshold.Source: Table 6 · p. 14, Table 8 · p. 15
  • Table 9 breaks down results for open-source language models prompted as critic models using greedy decoding.Source: Table 9 · p. 16

Related papers

Loading 2412.06559v4…