Source-linked AI summary
Making Large Language Models Better Reasoners with Step-Aware Verifier
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen
TL;DR
Large language models still have limited multi-step reasoning abilities, motivating methods that better elicit and assess reasoning paths. DIVERSE combines diverse prompts, verifier-guided voting, and step-level verification, achieving new state-of-the-art results on six of eight reasoning benchmarks. The approach improves benchmark accuracy while providing step-level analysis, but its reasoning paths are not guaranteed to be fully faithful and inference requires large language models.
Problem
Large pretrained language models remain limited on multi-step reasoning tasks, despite their scale and latent reasoning capabilities that require carefully designed prompts.
Method
DIVERSE varies prompts to generate diverse reasoning paths, uses a verifier for weighted voting, and verifies individual reasoning steps.
Results
DIVERSE achieves new state-of-the-art results on six of eight reasoning benchmarks, including GSM8K 74.4% →83.2%.
Takeaways & Limitations
Step-aware verification improves final-answer performance while providing detailed analysis of correctness at each reasoning step.
Takeaways & Limitations
DIVERSE cannot guarantee that generated reasoning paths are 100 percent faithful, and its inference requires more time and budget than fine-tuning models such as RoBERTa.
Abstract
from arXiv · showhide
Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from 17.9% to 58.1% in problem-solving rate. In this paper, we present DIVERSE (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DIVERSE has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DIVERSE on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GSM8K 74.4% to 83.2%).
1 Introduction
Large pretrained language models still struggle with multi-step reasoning, while prompting methods can activate latent reasoning abilities. DIVERSE addresses this challenge with diverse prompts, verifier-guided voting, and step-level verification, improving results across reasoning benchmarks.
- Large pretrained language models remain limited on tasks requiring multiple reasoning steps, especially arithmetic, commonsense, and inductive reasoning.
- Chain-of-thought prompting inserts multi-step reasoning paths before final answers and substantially improves GSM8K performance.
- DIVERSE increases reasoning-path diversity by varying prompts, uses a verifier to guide voting, and assigns fine-grained labels to individual reasoning steps.
- Step-aware verification is designed to identify correct steps followed by errors and support diagnosis and improvement of the reasoning process.
- DIVERSE achieves new state-of-the-art results on six benchmarks, including GSM8K 74.4% →83.2% and CLUTRR 67.0% → 95.9%.The reported benchmarks also include AsDiv, MultiArith, SVAMP, and SingleEq.
2 Diverse Verifier on Reasoning Step
DIVERSE combines diverse prompting, verifier-guided voting, and step-aware supervision to evaluate and aggregate multiple reasoning paths. Its step-level labeling compares intermediate reasoning across positive and negative paths to locate where errors begin.
- Diverse reasoning paths: DIVERSE samples reasoning paths from multiple prompts rather than relying on a fixed exemplar set, obtaining M = M1 × M2 paths per question.The main experiments use M1 = 5 and M2 = 20.
- Voting Verifier: The verifier takes a question and candidate reasoning path as input and outputs the probability that the path leads to the correct answer.It uses deberta-v3-large with a scalar prediction head.
- Voting Verifier: Voting verifier uses verifier scores to aggregate multiple reasoning paths, addressing cases where majority voting may favor misleading paths.
- Step-aware Voting Verifier: Step-aware voting adds an auxiliary loss that predicts whether each reasoning step is correct, alongside the original path-level loss.The step-level probability is represented by f′(inputi, j), and α balances the original and auxiliary losses.
- Step-aware Voting Verifier: Step-level labels for wrong-answer paths are extracted by comparing intermediate steps with positive paths; unmatched steps and subsequent steps are labeled negative.An off-the-shelf natural-language-inference model checks semantic equivalence across reasoning tasks.
3 Experimental Setup
The experiments evaluate DIVERSE across arithmetic, commonsense, and inductive reasoning benchmarks using three OpenAI language models. The setup compares DIVERSE with established decoding and prompting baselines, while step-level scores are examined for interpretability.
- Datasets: The benchmark suite covers five arithmetic tasks, two commonsense tasks, and one inductive reasoning task requiring kinship inference.The tasks are AsDiv, SingleEq, MultiArith, SVAMP, GSM8K, CommonsenseQA, StrategyQA, and CLUTRR.
- Language Models: Experiments use davinci, text-davinci-002, and code-davinci-002, with temperature 0.5 for sampling.
- Exemplars: Each DIVERSE question uses five prompts and 20 sampled reasoning paths per prompt, while pseudo-exemplars are constructed for commonsense datasets lacking enough annotated exemplars.
- Verifier Training: The verifier is trained on 1,000 sampled question-answer pairs per task using deberta-v3-large, with α selected from 0.0/0.1/0.2/0.3.
- Comparisons: Table 1 compares DIVERSE with Greedy Decode, Self-Consistency, and previous state-of-the-art results based on fine-tuning or PaLM.
- Step-level Analysis: Table 2 presents a GSM8K example in which step-level verifier scores indicate where reasoning paths begin to be incorrect and support understanding of model behavior.
4 Main Results
DIVERSE is compared with greedy decoding, self-consistency, fine-tuning results, and PaLM-based self-consistency across reasoning benchmarks. It achieves strong gains on arithmetic and inductive reasoning, while performing slightly below PaLM-based self-consistency on commonsense tasks.
- Comparisons: DIVERSE is evaluated against greedy decoding and self-consistency using overall experimental results.The comparisons also include previous fine-tuning-based and PaLM-based state-of-the-art results.
- Overall effectiveness: DIVERSE consistently improves over recent strong baselines across models and reasoning skills.On GSM8K, improvements over Greedy Decoding and Self-Consistency are reported for davinci, text-davinci-002, and code-davinci-002; average improvements across the three reasoning skills are 5.6%/5.1%/54.3%.
- Commonsense reasoning: −1.9% separates DIVERSE from PaLM-based self-consistency on the two commonsense reasoning tasks.The paper notes these tasks are multiple-choice rather than open-ended generation tasks.
- Inductive reasoning: 95.9% is DIVERSE's performance on CLUTRR, outperforming the previous fine-tuning state-of-the-art result by 28.9%.CLUTRR represents inductive reasoning in the reported benchmark set.
5 Case Study
The case study examines step-level scores from the step-aware verifier to show how it distinguishes correct and incorrect reasoning paths. The scores identify where an erroneous path begins to diverge and provide interpretability beyond final performance.
- Step-level verification: Correct reasoning paths receive relatively high step-level scores from the step-aware verifier.Wrong paths reveal the step at which reasoning starts to become incorrect.
- Interpretability: Step-level scoring provides interpretability by exposing the correctness of individual reasoning steps.The paper also presents additional majority-voting examples in Table 10.
6 Analysis
The analysis examines how diverse prompts, voting verification, and step-aware verification affect reasoning performance and quality. It also studies output counts, training-data size, exemplar counts, and error patterns.
- 6.1 The Effectiveness of Diverse Prompts: DIVERSE combines diverse prompts and sampling, using ⟨M1 = 5, M2 = 20⟩ versus ⟨M1 = 1, M2 = 100⟩ for sampling decoding.Neither diverse prompts alone nor sampling alone is optimal; the combination achieves the best performance.
- 6.2 The Effectiveness of Voting Verifier: DIVERSE’s voting verifier significantly and consistently outperforms majority voting across tasks and language models.Verifier-only selection often beats majority voting, while combining verification with voting improves performance further.
- 6.3 The Effectiveness of Step-aware Verifier: Step-aware verification improves performance in most GSM8K and CommonsenseQA experiments, except for code-davinci-002 on GSM8K.The authors hypothesize that code-davinci-002’s stronger reasoning paths reduce the benefit of step-level information.
- 6.3 The Effectiveness of Step-aware Verifier: 20% of correct reasoning paths contain redundant steps after step-aware verification, down from 31% with the verifier and 40% with random paths.Human evaluation judged reasoning quality using correctness, fluency, and coherence.
- 6.3 The Effectiveness of Step-aware Verifier: 95% of incorrect reasoning-step errors arise from incorrect formulations, including wrong intermediate results, operators, or invalid formulas.The verifier and step-aware verifier can assign low scores to paths containing division-calculation errors.
- 6.4 How Many Diverse Outputs Do We Need?: DIVERSE accuracy increases with more reasoning paths but improves only marginally at M ≥50, while consistently exceeding self-consistency across M values.Here, M is the number of paths sampled from 100 generated paths per question.
- 6.5 The Impact of Training Dataset Size: Reducing verifier-training data by 75%, from 1,000 to 250 reasoning paths, lowers performance by about 2%.With identical reasoning paths, voting verifier performs better than majority voting, whereas verifier-only selection causes significant drops.
- 6.6 The Impact of the Number of Exemplars: 83.2% GSM8K accuracy is achieved when each prompt contains 8 exemplars.The experiments compare prompts containing k = 3/5/8 exemplars.
7 Related Work
Related work spans reasoning benchmarks, symbolic systems, language-model reasoning, and prompting strategies for gigantic language models. These approaches vary in generality, data requirements, and prompting mechanisms.
- Reasoning Skills: Reasoning benchmarks evaluate commonsense, numerical, multi-hop, arithmetic, and logical reasoning skills.
- Reasoning with Symbolic Systems: Symbolic systems use knowledge graphs or question taxonomies to improve reasoning, but often require domain-specific designs and human effort.These requirements can limit generalizability.
- Reasoning via Language Models: Language-model approaches enhance reasoning through reasoning-aware pretraining, logic-guided augmentation, verifier training, or generated training examples.
- Reasoning via Prompting Gigantic Language Models: Prompting methods for gigantic language models include chain-of-thought, self-consistency, question decomposition, zero-shot stepwise reasoning, and joint exemplar-explanation selection.The paper continues this line of work by focusing on diverse verification of reasoning steps.
8 Conclusion and Future Work
DIVERSE enhances language-model reasoning with diverse prompts, a voting verifier, and a stepwise verifier. Experiments report state-of-the-art performance on most reasoning tasks, while future work targets false positives, prompting, and broader task coverage.
- 8 Conclusion: DIVERSE introduces diverse prompts, a voting verifier, and a stepwise verifier that separately evaluates reasoning steps.The paper describes the stepwise verifier as especially novel and effective for analyzing model behavior.
- 8 Conclusion: Using code-davinci-002, DIVERSE achieves state-of-the-art performance on most reasoning tasks and surpasses PaLM with previous prompting techniques.
- 8 Future Work: Future work will address false-positive pseudo exemplars, improve diverse prompts beyond simple sampling, and extend DIVERSE to other tasks.
9 Limitations
The paper identifies limits involving computational cost, faithfulness, training-data requirements, and the construction of diverse prompts and reasoning exemplars. It also describes prompting and self-teaching procedures used to support reasoning-path generation.
- Inference with DIVERSE requires large language models such as GPT-3 or PaLM, costing more time and budget than fine-tuning models like RoBERTa.
- DIVERSE improves final-answer accuracy but cannot guarantee that generated reasoning paths are fully faithful.
- DIVERSE needs more labeled reasoning paths for constructing diverse prompts and a training dataset for supervising the verifier.
- Chain-of-thought prompting inserts a reasoning path before answer generation, with each reasoning step describing how an answer is derived.
- DIVERSE constructs prompts by sampling exemplars, varying prompts across questions, and generating multiple sampled reasoning paths per prompt.
- When exemplar data are insufficient, self-teaching generates pseudo reasoning paths and predicted answers to expand the exemplar base.
B.2 Noises in Multiple Choice Tasks
Self-teaching can introduce invalid reasoning paths whose answers happen to match the ground truth, creating more serious noise in two-choice StrategyQA than in four-choice CommonsenseQA. This noise is associated with a 3.0% performance decline to PaLM on StrategyQA, while CommonsenseQA remains comparable at −0.8%.
- Invalid self-taught reasoning paths can yield answers that coincide with the ground truth, creating misleading exemplars.
- Two-choice StrategyQA is more vulnerable to this noise than four-choice CommonsenseQA because guessing a correct answer is easier while producing an invalid reasoning path.
- 3.0% performance decline to PaLM occurs on StrategyQA, compared with comparable performance at −0.8% on CommonsenseQA.
C Data Statistics
The experiments use established reasoning benchmarks, with specific handling for StrategyQA and CLUTRR. StrategyQA uses reciprocal train/test splits, while CLUTRR is modified to include candidate answers and templated reasoning paths.
- The benchmark suite includes GSM8K, AsDiv, MultiArith, SVAMP, SingleEq, CommonsenseQA, StrategyQA, and CLUTRR.
- StrategyQA contains 2,290 test cases, which are split into equal parts and alternated between exemplar construction and testing.
- CLUTRR experiments use its clean version rather than the supporting, irrelevant, or disconnected perturbed variants.
- CLUTRR inputs append candidate answers, while outputs use reasoning paths constructed from knowledge-graph rules and text templates.
- Without candidate answers, the language models cannot exceed 50% accuracy in preliminary CLUTRR experiments.
- DIVERSE selects correct-but-not-most frequent answers in some GSM8K examples, improving over majority voting.