PaperlayerView exact revision on arXiv

Source-linked AI summary

OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verification

Zijian Wu, Lingkai Kong, Wenwei Zhang, Songyang Gao, Yuzhe Gu, Zhongrui Cai, Tianyou Ma, Yuhong Liu, Zhi Wang, Runyuan Ma, Guangyu Wang, Wei Li, Conghui He, Dahua Lin, Kai Chen

arXiv:2512.10756v1cs.CLcs.LG

TL;DR

Outcome-based verifiers miss unreliable intermediate steps in long reasoning chains, while process-based verifiers struggle with complex chains and costly expert annotation. OPV verifies concise summaries of long chains and improves through iterative expert annotation. It achieves state-of-the-art results across multiple benchmarks despite its compact size, with gains also reported for policy-model performance on AIME2025.

  • Problem

    Outcome-based verifiers overlook unreliable intermediate steps in long chains of thought, while process-based verifiers struggle with complex reasoning and costly verification and expert annotation.

    Source: §1 · p. 1

  • Method

    OPV summarizes long chains of thought into concise key-step paths for step-by-step verification, while active learning prioritizes uncertain cases for expert annotation and retraining.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    OPV achieves state-of-the-art results across multiple benchmarks despite its compact size and raises DeepSeek-R1-Distill-Qwen-32B accuracy from 55.2% to 73.3% on AIME2025.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The 2.2k-solution expert-annotated OPV-Bench dataset provides a resource for future research.

    Source: §6 · p. 9

    Using vanilla summaries rather than re-summarizing led OPV to misclassify 2 of 50 checked correct solutions when summaries introduced logical gaps.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) have achieved significant progress in solving complex reasoning tasks by Reinforcement Learning with Verifiable Rewards (RLVR). This advancement is also inseparable from the oversight automated by reliable verifiers. However, current outcome-based verifiers (OVs) are unable to inspect the unreliable intermediate steps in the long reasoning chains of thought (CoTs). Meanwhile, current process-based verifiers (PVs) have difficulties in reliably detecting errors in the complex long CoTs, limited by the scarcity of high-quality annotations due to the prohibitive costs of human annotations. Therefore, we propose the Outcome-based Process Verifier (OPV), which verifies the rationale process of summarized outcomes from long CoTs to achieve both accurate and efficient verification and enable large-scale annotation. To empower the proposed verifier, we adopt an iterative active learning framework with expert annotations to progressively improve the verification capability of OPV with fewer annotation costs. Specifically, in each iteration, the most uncertain cases of the current best OPV are annotated and then subsequently used to train a new OPV through Rejection Fine-Tuning (RFT) and RLVR for the next round. Extensive experiments demonstrate OPV's superior performance and broad applicability. It achieves new state-of-the-art results on our held-out OPV-Bench, outperforming much larger open-source models such as Qwen3-Max-Preview with an F1 score of 83.1 compared to 76.3. Furthermore, OPV effectively detects false positives within synthetic dataset, closely align with expert assessment. When collaborating with policy models, OPV consistently yields performance gains, e.g., raising the accuracy of DeepSeek-R1-Distill-Qwen-32B from 55.2% to 73.3% on AIME2025 as the compute budget scales.

1. Introduction

Long chains of thought make reliable verification important, but outcome-based checks miss intermediate errors while process-based checks struggle with complex reasoning and annotation costs. OPV verifies concise summaries of long reasoning, using active learning to improve the verifier and support fine-grained supervision.

1. Introduction

  • Outcome-based verifiers check only final answers, whereas process-based verifiers inspect entire chains but struggle with complex structures and costly verification and annotation.Earlier coarse-heuristic approaches also failed to provide accurate correctness verdicts or error locations.Source: §1 · p. 1
  • OPV retains key steps in a concise solution path, then verifies that summarized outcome step by step and identifies an error location when it is incorrect.Summarization removes redundant explorations such as trial-and-error attempts, recalculations, and self-overturned assumptions.Source: §1 · pp. 1–2
  • Each active-learning round sends the current OPV’s most uncertain cases for expert annotation, then uses the new data for offline rejection fine-tuning and online reinforcement learning.The process produced 40k annotated solutions across K-12 to undergraduate problem domains, including a 2.2k-sample held-out OPV-Bench.Source: §1 · p. 2
  • Despite its compact size, OPV performs comparably to much larger open-source models across multiple public and internal benchmarks, including held-out OPV-Bench.Source: §1 · p. 2
  • 7.0% of answers in AM-DeepSeek-R1-0528-Distilled were estimated false positives, while OPV raised DeepSeek-R1-Distill-Qwen-32B accuracy from 55.2% to 73.3% on AIME2025.The estimated false-positive rate closely aligned with expert assessments.Source: §1 · p. 3

2. Method

OPV turns long chains of thought into concise solution paths for stepwise verification, then uses iterative expert annotation and model updates to improve verification and build a large labeled dataset.

2.1. Task Formulation

  • OPV summarizes verbose chains of thought into concise paths retaining key steps, then predicts the first erroneous step and an explanation.Fully correct solutions are marked separately; later steps after the first error are treated as built on faulty premises.Source: §2.1 · p. 3

2.2. Active Learning Framework

  • The active-learning loop repeatedly selects uncertain cases for expert labels, then combines offline expert iteration with online reinforcement learning to update OPV.Solutions are pre-summarized before entering the data pool, and newly labeled data are used to train the next verifier.Source: §2.2 · pp. 3–4
  • In each active-learning round, OPV runs multiple independent verifications and uses low prediction consistency to identify uncertain cases for annotation.A dynamically adjustable threshold controls selection, while a small proportion of high-consistency data is also sampled to mitigate overconfidence.Source: §2.2 · p. 4

2.3. OPV Update by Reinforcement Learning

  • Online RL excludes highly ambiguous and trivial cases, while its reward sharply penalizes correctness-status errors and decays with error-location distance.The reward's localization penalty is controlled by λ; the final OPV is trained with DAPO.Source: §2.3 · p. 5

2.4. Final Dataset Statistics

  • The completed framework yields over 40k expert annotations, over 80k high-quality judge trajectories, and a held-out OPV-Bench of 2.2k samples.The annotated set spans multiple difficulty levels and knowledge domains.Source: §2.4 · p. 5

3. Experiment

The experiments assess verifier judgments under exact, approximate, and rough criteria across two benchmarks. OPV-Bench poses more complex verification challenges, while ProcessBench results show performance saturation on rough error detection.

3.1. Experiment Setup

  • Evaluation uses accuracy and F1 under precise, adjacent-step approximate, and rough error-detection criteria, comparing OPV with open-source models and intermediate OPV stages.Table 1 reports results on ProcessBench and OPV-Bench under maj@8 voting.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2. Benchmark Evaluation

  • Most long-reasoning models exceed 90% F1 on ProcessBench's rough criterion, where elementary errors are more readily identifiable.The passage notes that ProcessBench samples answers from models without thinking capabilities and contains explicit, verifiable outcomes, unlike proof-based problems requiring complex multi-step reasoning.Source: §3.2 · p. 7
  • A 32B OPV trained through iterative annotation and training outperforms much larger models on the more challenging OPV-Bench.The passage reports that most open-source reasoning models struggle to identify error positions, with high recall but poor precision.Source: §3.1 · p. 6, §3.2 · p. 7

4. Application

OPV is applied to improve synthetic-data quality and assist policy models at inference time. Across these applications, it detects process errors and supports stronger collaborative reasoning performance.

4.1. Examining AM-DeepSeek-R1-0528-Distilled using OPV

  • OPV flagged 53.7k of 674k checked entries, with human evaluation finding 88% of judgments valid and an estimated 7.0% containing process errors.Each entry was checked eight times; entries were flagged when OPV reported errors at least six times, and human evaluation covered 50 randomly sampled problems.Source: §4.1 · p. 7
  • Two of 50 checked solutions were correct but flagged as incorrect because their vanilla summaries introduced logical gaps.The authors note that re-summarization is important for precise verification.Source: §4.1 · p. 7

4.2. Scaling of collaborative reasoning

  • The collaborative setup samples N complete solutions, checks each M times, and aggregates verdicts; the reported AIME2025 comparison uses N=8 and M=16.Strategies include majority voting, ranking by verification pass rate, and verifier-weighted voting; the evaluation uses original CoT summaries to reduce compute and avoid added noise.Source: §4.2 · pp. 7–8
  • Verifier voting improves AIME2025 accuracy across policies, averaging gains of 6.1% for distilled models and 2.3% for top-tier models over majority voting.It matches Pass@8 for gpt-oss-120b and outperforms Best-of-8 on most policies.Source: Table 3 · p. 8, §4.2 · p. 8
  • At N=64 and M=64, Verifier-Voting reaches 73.3%, improving 6.7 points over Majority-Voting as accuracy rises with larger N across strategies.OPV-enabled Best-of-N and Verifier-Voting consistently outperform Majority-Voting; Best-of-N gradually converges to Majority-Voting as N increases.Source: Figure 5 · p. 8, §4.2 · p. 9

5. Related Work

Prior work contrasts scalable outcome checks with finer-grained process verification, whose annotation costs and long-CoT complexity remain challenges. OPV follows generative verification while using expert supervision and concise rationales.

5. Related Work

  • Generative verification trajectories in a chain-of-thought style have been explored, and OPV adds expert supervision intended to be finer-grained than heuristics.The related-work discussion also distinguishes ProcessBench’s expert annotations from OPV’s focus on more challenging queries and longer frontier-model CoTs.Source: §5 · p. 9
  • Outcome-based verification checks final-answer matches but misses intermediate reasoning reliability, while process-based verification examines reasoning step by step at high annotation and computation cost.Prior process-verifier training has used Monte Carlo credit assignment or stronger teacher models, with simulation bias, label noise, or teacher-capacity constraints noted.Source: §5 · p. 9
  • Summarizing long CoTs into concise rationales is presented as a way to support larger-scale expert supervision for process verification.The paper positions this approach alongside work using stronger models to generate verification trajectories.Source: §5 · p. 9

6. Conclusion

The conclusion presents OPV as a verifier for summarized long-CoT solutions, improved through iterative expert-annotated active learning. It reports broad applicability and frames summarized-rationale verification as a scalable direction for more trustworthy reasoning.

6. Conclusion

  • OPV bridges outcome and process verification by checking summarized solutions from long chains of thought and iteratively improving through expert-annotated active learning.The framework is described as minimizing annotation costs while progressively improving verification capability.Source: §6 · p. 9
  • OPV identifies false positives in outcome-verified synthetic data and yields consistent gains when collaborating with policy models at inference time.The accompanying OPV-Bench contains 2.2k expert-annotated solutions.Source: §6 · p. 9
  • The authors frame efficient, accurate process verification at scale as addressing a bottleneck in reliable reasoning-system development.They describe summarized-rationale verification as a scalable path toward more trustworthy AI-generated reasoning as reasoning chains grow longer and problems more complex.Source: §6 · p. 9

A.1. Data and Annotation Preliminaries

The data and annotation pipeline draws on diverse mathematics sources, generates and summarizes multiple reasoning paths, and uses expert feedback to build OPV training data. Its annotation rules define when labels are accepted and how ambiguous steps are treated.

A.1. Data and Annotation Preliminaries

  • The problem pool combines difficult contest benchmarks, community-curated math problems, and open-source datasets across K–12, high-school competition, and undergraduate levels.The authors retain open-ended calculation and proof problems and apply exact-match and semantic-match decontamination against major public benchmarks.Source: §A.1 · p. 14
  • For each problem, the authors sample 8 to 12 CoTs from R1 and Qwen models, then use Deepseek-V3 to re-summarize reasoning while retaining calculations, enumerations, and case analyses.The resulting unlabeled problem-summary pairs seed active learning.Source: §A.1 · p. 14
  • The annotation budget allocates 80% to low-consistency cases and 20% to random high-consistency cases, allowing feedback on confident errors as well as uncertain ones.The consistency thresholds are 0.25 in the first stage and 0.5 in the second.Source: §A.1 · p. 14
  • Experts identify the first erroneous step, while flawed-but-tolerable steps correctable within 2–3 sentences are not labeled erroneous.Three experts independently evaluate each solution; labels are valid when all agree it is correct or at least two locate an error within a two-step window.Source: §A.1 · p. 14

A.2. Dataset Statistics

OPV-Bench contains 2,202 test cases across three subsets, with errors concentrated early in reasoning overall but peaking later for high-school competition problems.

A.2. Dataset Statistics

  • OPV-Bench has 2,202 test cases across three subsets.Solutions are re-summarized by Deepseek-V3 before evaluation, keeping average step counts consistent across problems within each subset.Source: §A.2 · p. 15
  • Most errors occur in the initial reasoning steps, although high-school competition problems show a later error peak.High-school problems require less formula application but more exploration and analysis.Source: §A.2 · p. 15
  • Incorrect solutions contain slightly more steps than correct solutions.Average step counts remain consistent across problems within each subset after re-summarization.Source: §A.2 · p. 15

B. Verifier Training Details

The verifier uses a specified fine-tuning and on-policy RL setup. The accompanying prompting details describe templates for solution summarization and stepwise process verification.

B. Verifier Training Details

  • Verifier fine-tuning uses 1 epoch, a learning rate of 8e-5, sequence length 32k, and weight decay 10.The base architecture is pre-trained R1-distilled-32B; the on-policy RL stage uses DAPO for 80 steps without warmup.Source: B. Verifier Training Details · p. 15

C. Prompting Details

D. Additional Experiment Results

The additional results section provides detailed evaluations across OPV-Bench and ProcessBench partitions. Its case studies include a correct final answer whose reasoning contains a defect identified by OPV.

D. Additional Experiment Results

  • Detailed evaluations cover the GSM8k, MATH, OlympiadBench, and Omni-MATH partitions of ProcessBench.The supplied table captions identify these partitions but do not provide their evaluation values.Source: Table 5 · p. 18, Table 6 · p. 18, Table 7 · p. 19
  • Detailed evaluations also cover the K-12, High School Competition, and Undergraduate partitions of OPV-Bench.The supplied table captions identify these partitions but do not provide their evaluation values.Source: Table 9 · p. 20, Table 10 · p. 20, Table 11 · p. 21

E. Case Studies

  • The case studies present process errors found by OPV across different datasets.Source: E. Case Studies · p. 21
  • In a case study, OPV identifies defective reasoning despite a correct final answer because the proposed inequality's rigor is unproven.The defective content is marked in red in the generations.Source: Figure 10 · p. 22

Related papers

Loading 2512.10756v1…