PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoPSV: Automated Process-Supervised Verifier

Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Zhijiang Guo

arXiv:2405.16802v4cs.CLcs.LG

TL;DR

Prompting and tuning can be task-specific or resource-intensive, while process supervision requires costly step-level labels. AutoPSV derives process annotations from an outcome-supervised verifier’s confidence changes, and the paper reports improved verification performance across mathematical and commonsense reasoning tasks. Its annotations are estimated, and their quality depends on verifier accuracy.

  • Problem

    Reasoning approaches based on prompting or tuning can be task-specific or resource-intensive, while process supervision requires costly step-level annotations.

    Source: §1 · pp. 1–2

  • Method

    AutoPSV trains an outcome-supervised verifier to score reasoning steps and uses relative confidence changes to generate process annotations without ground-truth step labels.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Experiments report improved verification performance across mathematical and commonsense reasoning tasks, including five datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AutoPSV’s automatically generated process labels are reported to reduce the need for manual intervention and extensive computation.

    Source: §1 · p. 2

    Process labels may misrepresent step correctness, and their quality depends on the verifier’s accuracy in scoring steps.

    Source: A Limitations · p. 16

Abstract

from arXiv · show

In this work, we propose a novel method named \textbf{Auto}mated \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{\textsc{AutoPSV}}) to enhance the reasoning capabilities of large language models (LLMs) by automatically annotating the reasoning steps. \textsc{AutoPSV} begins by training a verification model on the correctness of final answers, enabling it to generate automatic process annotations. This verification model assigns a confidence score to each reasoning step, indicating the probability of arriving at the correct final answer from that point onward. We detect relative changes in the verification's confidence scores across reasoning steps to automatically annotate the reasoning process, enabling error detection even in scenarios where ground truth answers are unavailable. This alleviates the need for numerous manual annotations or the high computational costs associated with model-induced annotation approaches. We experimentally validate that the step-level confidence changes learned by the verification model trained on the final answer correctness can effectively identify errors in the reasoning steps. We demonstrate that the verification model, when trained on process annotations generated by \textsc{AutoPSV}, exhibits improved performance in selecting correct answers from multiple LLM-generated outputs. Notably, we achieve substantial improvements across five datasets in mathematics and commonsense reasoning. The source code of \textsc{AutoPSV} is available at \url{https://github.com/rookie-joe/AutoPSV}.

1 Introduction

Reasoning improvements from prompting, tuning, and verification face task-specificity or annotation and computational costs. AutoPSV uses an outcome-supervised verifier’s confidence changes to generate process labels, with experiments reporting improvements across mathematical and commonsense reasoning.

1 Introduction

  • Process supervision can require costly human or model-generated labels, while outcome supervision is cheaper but less effective; prompting and tuning also have resource or task-specific limits.Prompt design can be labor-intensive, and instruction tuning or distillation typically requires many chain-of-thought annotated examples.Source: §1 · pp. 1–2
  • AutoPSV trains an outcome-supervised verifier to score intermediate reasoning steps and uses relative confidence changes to create process annotations without ground-truth answers.The resulting annotations train an enhanced verifier using process- and outcome-level supervision.Source: §1 · p. 2
  • Experiments across five datasets report improved verification performance in mathematical and commonsense reasoning, with reduced manual intervention and computational demands.The contribution also describes improved scalability.Source: §1 · p. 2
  • After training, AutoPSV is applicable in both labeled and unlabeled dataset settings.The paper presents this flexibility as evidence of potential applicability across LLM applications.Source: §1 · p. 2

2 Related Works

Prior reasoning approaches include prompting, parameter-tuning methods, and verifiers that select among candidate answers. AutoPSV is presented as a verifier that uses confidence changes to produce process annotations without additional sampling or manual labeling.

Improving Reasoning Abilities of LLMs

  • Prompting can be task-specific and labor-intensive, while instruction tuning and distillation require resource-intensive fine-tuning and large chain-of-thought annotated datasets.AutoPSV instead trains an additional verifier to select outputs from the original model.Source: §2 · p. 3

From Outcome to Process Supervision

  • Outcome-supervised verifiers use final-answer correctness, whereas process-supervised verifiers use step-level feedback that can pinpoint errors but costs more to annotate.Prior model-induced process annotation methods use Monte Carlo Tree Search and can demand substantial computation.Source: §2 · p. 3
  • AutoPSV assigns step confidence with an outcome-supervised verifier and computes relative changes, avoiding additional sampling or manual labeling.The related-work discussion contrasts this with MCTS-based annotation methods.Source: §2 · p. 3

3 AUTOPSV

AutoPSV addresses selecting correct answers from multiple LLM-generated solutions. It trains an outcome-supervised verifier, then uses changes in its step-level confidence to create process labels.

3.1 Problem Setting

  • The task is to select the most accurate response from multiple solutions generated by an LLM.The stated objective is to maximize the probability of selecting an accurate candidate.Source: §3.1 · p. 3

3.2 Motivation

  • Prompted selectors from 7B to over 70B parameters remained suboptimal on GSM8K, motivating dedicated verification models.The comparison evaluates selection methods across model sizes for answers generated by Mixtral-Instruct.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 Training Methodology

  • AutoPSV trains an outcome-supervised verifier with MSE, interprets each prefix score as eventual correctness likelihood, then labels steps from relative confidence changes using a threshold.A negative confidence change indicates reduced confidence in reaching a correct answer after the next step.Source: §3.3 · pp. 4–5

4 Preliminary Findings

The preliminary findings test whether outcome-supervised verification can select correct responses and whether changes in step-level confidence can detect calculation errors. These results, together with a theoretical framework, support using the verifier to generate process annotations.

4 Preliminary Findings

  • The section validates OSV response selection and confidence-change error detection as empirical foundations for automatic process annotation, alongside a theoretical framework for estimating future answer correctness from an intermediate step.The framework is described in Theorem 1.Source: §4 · p. 5, §4.2.2 · p. 7

4.1 Experiment on Outcome-Supervised Verifier Performance

  • OSV models trained with Mistral or Phi consistently outperform the self-consistency baseline when selecting correct responses across generator configurations.The evaluation selects from five generated responses and considers generators ranging from 7B to 72B parameters.Source: §4.1 · p. 5, Table 3 · p. 5

4.2 Detecting Calculation Error During Math Reasoning

  • The proposed detection method monitors relative step-level confidence changes to identify inaccuracies during mathematical reasoning.The section evaluates this signal against a self-designed benchmark for calculation errors.Source: §4.2 · p. 6

4.2.1 Math Calculation Error

  • A math calculation error is defined as a mismatch between the numerical values on the two sides of an equals sign.The authors use this definition to establish a preliminary benchmark for math calculation error detection.Source: §4.2.1 · p. 6

4.2.2 Quantitative Results

  • Across thresholds from -0.5 to -0.9, stricter error labeling raises precision and lowers recall, while F1 remains relatively stable.The authors describe this as balancing detection of real errors against incorrect flagging of valid calculations.Source: §4.2.2 · pp. 6–7, Table 5 · p. 7

5 Experiment

The experiments evaluate process-supervision-enhanced verification for selecting among LLM-generated answers across mathematical and commonsense benchmarks. Results generally favor the enhanced verifier, with some generator- and benchmark-specific exceptions.

5.1 Experimental Setup

  • The evaluation uses three instruction-tuned generators, five benchmarks, and selects among five candidate responses; PSV scores are aggregated by multiplying step-level scores.The generators range from 7B to over 70B parameters; the benchmarks cover mathematical and commonsense reasoning.Source: §5.1 · p. 7

5.2 Enhanced LLMs Reasoning via Process Supervision

  • OSV + PSV outperforms OSV and Self-Consistency on GSM8K for all tested generators, and on MATH for Mistral-Instruct and Mixtral-Instruct, but not Qwen.The MATH exception is that Self-Consistency is slightly more effective with Qwen.Source: §5.2 · p. 8, Table 6 · p. 8
  • On HellaSwag, OSV + PSV leads for every generator; results on Winogrande and ANLI vary by generator and method.OSV + PSV leads on Winogrande with Mistral-Instruct and on ANLI with Mistral-Instruct and Mixtral-Instruct; OSV has a marginal Winogrande advantage with Mixtral-Instruct and Qwen.Source: §5.2 · p. 8, Table 7 · p. 8

6 Analysis

Section 6 compares AutoPSV with MCTS in labeled and unlabeled settings, then assesses process-supervised verification trained on AutoPSV labels against outcome supervision and self-consistency.

6.1 Advantages of AutoPSV in Labeled and Unlabeled Settings

  • AutoPSV labeling performs competitively with MCTS on GSM8K and MATH, and sometimes outperforms it, especially on MATH.Table 8 compares Pass@5, Self-Consistency, and response selection across response generators.Source: Table 8 · p. 9, §6.1 · p. 9
  • AutoPSV requires significantly fewer processed tokens than MCTS-based methods for process labeling on GSM8K and MATH.It avoids generating multiple samples for each reasoning step.Source: Table 9 · p. 9, §6.1 · p. 9
  • Adding 7,000 unlabeled WizardLM math problems to GSM8K annotations improved results across all evaluated response generators.MCTS and OSV-only training cannot leverage these unlabeled questions.Source: §6.1 · pp. 9–10
  • 63.11 (OSV+PSV with GSM8K+WizardLM) versus 61.18 (OSV) for Mistral-Instruct illustrates the gains from adding unlabeled data.The added WizardLM questions have no gold solutions; MCTS and OSV-only training cannot use them.Source: §6.1 · pp. 9–10, Table 10 · p. 9

6.2 Outcome-Supervised Verification vs. Process-Supervised Verification

  • Across three response generators, AutoPSV-labeled process supervision beats self-consistency, while retaining a small performance gap from OSV.The authors interpret the small gap as evidence that relabeled process supervision inherits information from OSV without requiring ground-truth annotations.Source: §6.2 · p. 10

7 Conclusion

The conclusion presents AutoPSV as a way to label reasoning steps from changes in verifier confidence and reports improved verifier performance across reasoning tasks. It also identifies generator supervision with step-wise optimization as future work.

7 Conclusion

  • AutoPSV automatically labels reasoning steps by detecting relative changes in model confidence.The conclusion frames this as automatic process labeling for LLMs.Source: §7 · p. 10
  • Experiments report improved verifier precision and scalability across mathematical and commonsense reasoning tasks.The authors state that the approach reduces the need for intensive computation and manual intervention.Source: §7 · p. 10
  • Future work will use automatically constructed process supervision for step-wise proximal policy optimization of generators, aiming to improve greedy-decoding accuracy without reranking.This is presented as a future research direction, not a demonstrated result.Source: §7 · p. 10

A Limitations

The appendix notes limitations tied to verifier accuracy and task scope, discusses potential societal benefits and risks, and reviews external and self-generated feedback approaches.

A Limitations

  • AutoPSV annotations may be inaccurate because confidence changes only estimate step correctness, and their quality depends on verifier accuracy.The authors also say applicability may be limited to complex multi-step reasoning, with generalization to other domains unclear.Source: A Limitations · p. 16

B Broader Impact

  • The authors suggest that automatic annotation could reduce manual effort and improve verifier scalability, potentially supporting more capable LLMs.They also identify possible benefits for information reliability and decision-making in sectors including healthcare, education, and finance.Source: B Broader Impact · p. 17
  • Potential negative impacts include annotator job displacement, misuse to spread misinformation, reduced critical thinking, and privacy or security issues.The passage presents these as potential risks.Source: B Broader Impact · p. 17

C More Related Work

  • External feedback can offer perspectives unavailable to an LLM alone, while self-generated feedback may help when external feedback is scarce [61].A cited study suggests LLMs struggle to identify and correct errors independently [63].Source: C More Related Work · p. 17

D Vanilla Evaluation Methods Description

The section describes evaluation prompts that select, compare, or score candidate answers, sometimes requiring written analysis alongside the judgment.

D Vanilla Evaluation Methods Description

E Math Calculation Error Benchmark

The benchmark uses automated expression evaluation to identify computational errors and illustrates how step-level confidence decreases mark calculation errors. It also defines the evaluation scope and excludes expressions that cannot be evaluated.

  • Regular expressions extract computational steps, and Python’s eval checks expressions left of “=” against their right-hand-side results.Source: E Math Calculation Error Benchmark · p. 19
  • LLaMA performance is evaluated on GSM8K training data using few-shot chain-of-thought prompting, as scoped by Table 12.Source: Table 12 · p. 19
  • Unsolvable or incorrectly formatted expressions are excluded, keeping ground truth focused on computational errors and offering a scalable alternative to human labeling.Source: E Math Calculation Error Benchmark · p. 19
  • Significant decreases in model confidence detect calculation errors; Figure 2 marks the ground-truth error in red and shows step-level confidence changes in brackets.Source: E Math Calculation Error Benchmark · p. 19

F.1 Training Hyperparameters

The appendix specifies training and generation settings, compares score aggregation functions, and describes process-labeling implementations. Computational constraints reduce the implementation’s sample count relative to the main experiments.

F.1 Training Hyperparameters

  • The verifier is trained for 1 epoch with batch size 512, learning rate 2 × 10−6, and 3% warmup; experiments use eight 40GB A100 GPUs and AdamW.Source: §F.1 · p. 19, Table 13 · p. 19
  • Before verifier training, an SFT model generates responses and supports outcome labels by checking final-answer correctness; its settings are listed in Table 14.Source: §F.1 · p. 19, Table 14 · p. 20

F.2 Generation Settings

  • Greedy decoding uses temperature 0.0, while pass@k uses temperature 0.7; pass@k generates 20 samples per problem, evaluates unit tests, and repeats five times for 95% confidence intervals.Source: §F.2 · p. 20
  • The product of step-level scores is the primary aggregation; on GSM8K it performs comparably or slightly better than the minimum across response generators.Source: §F.2 · p. 20, Table 15 · p. 21

F.3 Process-Supervision Labelling Strategy

  • Process Ours trains an output-supervised verifier on 150,000 generated samples, relabels them using relative confidence changes, then retrains; MCTS labels paths using eight complete variations per incomplete path.Source: §F.3 · p. 21
  • The implementation uses 10 samples per problem rather than the main experiments’ 50 because of computational constraints; full MCTS labeling at that scale would take about 10–12 days.Source: §F.3 · p. 21
Loading 2405.16802v4…