Making Large Language Models Better Reasoners with Step-Aware Verifier

• Large pretrained language models remain limited on tasks requiring multiple reasoning steps, especially arithmetic, commonsense, and inductive reasoning.Source: §1 · p. 1
• Chain-of-thought prompting inserts multi-step reasoning paths before final answers and substantially improves GSM8K performance.Source: §1 · p. 1
• DIVERSE increases reasoning-path diversity by varying prompts, uses a verifier to guide voting, and assigns fine-grained labels to individual reasoning steps.Source: §1 · pp. 1–2
• Step-aware verification is designed to identify correct steps followed by errors and support diagnosis and improvement of the reasoning process.Source: §1 · p. 2
• DIVERSE achieves new state-of-the-art results on six benchmarks, including GSM8K 74.4% →83.2% and CLUTRR 67.0% → 95.9%.The reported benchmarks also include AsDiv, MultiArith, SVAMP, and SingleEq.Source: §1 · p. 2

Diverse reasoning paths

• DIVERSE samples reasoning paths from multiple prompts rather than relying on a fixed exemplar set, obtaining M = M1 × M2 paths per question.The main experiments use M1 = 5 and M2 = 20.Source: §2.1 · p. 2

Voting Verifier

• The verifier takes a question and candidate reasoning path as input and outputs the probability that the path leads to the correct answer.It uses deberta-v3-large with a scalar prediction head.Source: §2.2 · p. 2
• Voting verifier uses verifier scores to aggregate multiple reasoning paths, addressing cases where majority voting may favor misleading paths.Source: §2.2 · p. 2

Step-aware Voting Verifier

• Step-aware voting adds an auxiliary loss that predicts whether each reasoning step is correct, alongside the original path-level loss.The step-level probability is represented by f′(inputi, j), and α balances the original and auxiliary losses.Source: §2.3 · p. 3
• Step-level labels for wrong-answer paths are extracted by comparing intermediate steps with positive paths; unmatched steps and subsequent steps are labeled negative.An off-the-shelf natural-language-inference model checks semantic equivalence across reasoning tasks.Source: §2.3 · p. 3

Datasets

• The benchmark suite covers five arithmetic tasks, two commonsense tasks, and one inductive reasoning task requiring kinship inference.The tasks are AsDiv, SingleEq, MultiArith, SVAMP, GSM8K, CommonsenseQA, StrategyQA, and CLUTRR.Source: §3.1 · p. 3

Language Models

• Experiments use davinci, text-davinci-002, and code-davinci-002, with temperature 0.5 for sampling.Source: §3.2 · p. 3

Exemplars

• Each DIVERSE question uses five prompts and 20 sampled reasoning paths per prompt, while pseudo-exemplars are constructed for commonsense datasets lacking enough annotated exemplars.Source: §3.2 · p. 3

Verifier Training

• The verifier is trained on 1,000 sampled question-answer pairs per task using deberta-v3-large, with α selected from 0.0/0.1/0.2/0.3.Source: §3.2 · p. 3

Comparisons

• Table 1 compares DIVERSE with Greedy Decode, Self-Consistency, and previous state-of-the-art results based on fine-tuning or PaLM.Source: Table 1 · p. 4

Step-level Analysis

• Table 2 presents a GSM8K example in which step-level verifier scores indicate where reasoning paths begin to be incorrect and support understanding of model behavior.Source: Table 2 · p. 4

Comparisons

• DIVERSE is evaluated against greedy decoding and self-consistency using overall experimental results.The comparisons also include previous fine-tuning-based and PaLM-based state-of-the-art results.Source: §4 · p. 4, §4.2 · p. 4

Overall effectiveness

• DIVERSE consistently improves over recent strong baselines across models and reasoning skills.On GSM8K, improvements over Greedy Decoding and Self-Consistency are reported for davinci, text-davinci-002, and code-davinci-002; average improvements across the three reasoning skills are 5.6%/5.1%/54.3%.Source: §4.1 · p. 4

Commonsense reasoning

• −1.9% separates DIVERSE from PaLM-based self-consistency on the two commonsense reasoning tasks.The paper notes these tasks are multiple-choice rather than open-ended generation tasks.Source: §4.2 · p. 4

Inductive reasoning

• 95.9% is DIVERSE's performance on CLUTRR, outperforming the previous fine-tuning state-of-the-art result by 28.9%.CLUTRR represents inductive reasoning in the reported benchmark set.Source: §4.2 · p. 5

Step-level verification

• Correct reasoning paths receive relatively high step-level scores from the step-aware verifier.Wrong paths reveal the step at which reasoning starts to become incorrect.Source: §5 · p. 5

Interpretability

• Step-level scoring provides interpretability by exposing the correctness of individual reasoning steps.The paper also presents additional majority-voting examples in Table 10.Source: §5 · p. 5

6.1 The Effectiveness of Diverse Prompts

• DIVERSE combines diverse prompts and sampling, using ⟨M1 = 5, M2 = 20⟩ versus ⟨M1 = 1, M2 = 100⟩ for sampling decoding.Neither diverse prompts alone nor sampling alone is optimal; the combination achieves the best performance.Source: §6.1 · p. 5

6.2 The Effectiveness of Voting Verifier

• DIVERSE’s voting verifier significantly and consistently outperforms majority voting across tasks and language models.Verifier-only selection often beats majority voting, while combining verification with voting improves performance further.Source: §6.2 · p. 5

6.3 The Effectiveness of Step-aware Verifier

• Step-aware verification improves performance in most GSM8K and CommonsenseQA experiments, except for code-davinci-002 on GSM8K.The authors hypothesize that code-davinci-002’s stronger reasoning paths reduce the benefit of step-level information.Source: §6.3 · p. 6
• 20% of correct reasoning paths contain redundant steps after step-aware verification, down from 31% with the verifier and 40% with random paths.Human evaluation judged reasoning quality using correctness, fluency, and coherence.Source: §6.3 · p. 6
• 95% of incorrect reasoning-step errors arise from incorrect formulations, including wrong intermediate results, operators, or invalid formulas.The verifier and step-aware verifier can assign low scores to paths containing division-calculation errors.Source: §6.3 · p. 7

6.4 How Many Diverse Outputs Do We Need?

• DIVERSE accuracy increases with more reasoning paths but improves only marginally at M ≥50, while consistently exceeding self-consistency across M values.Here, M is the number of paths sampled from 100 generated paths per question.Source: §6.4 · p. 7, Figure 7 · p. 7

6.5 The Impact of Training Dataset Size

• Reducing verifier-training data by 75%, from 1,000 to 250 reasoning paths, lowers performance by about 2%.With identical reasoning paths, voting verifier performs better than majority voting, whereas verifier-only selection causes significant drops.Source: §6.5 · p. 7

6.6 The Impact of the Number of Exemplars

• 83.2% GSM8K accuracy is achieved when each prompt contains 8 exemplars.The experiments compare prompts containing k = 3/5/8 exemplars.Source: §6.6 · p. 7

Reasoning Skills

• Reasoning benchmarks evaluate commonsense, numerical, multi-hop, arithmetic, and logical reasoning skills.Source: §7 · p. 7

Reasoning with Symbolic Systems

• Symbolic systems use knowledge graphs or question taxonomies to improve reasoning, but often require domain-specific designs and human effort.These requirements can limit generalizability.Source: §7 · p. 7

Reasoning via Language Models

• Language-model approaches enhance reasoning through reasoning-aware pretraining, logic-guided augmentation, verifier training, or generated training examples.Source: §7 · pp. 7–8

Reasoning via Prompting Gigantic Language Models

• Prompting methods for gigantic language models include chain-of-thought, self-consistency, question decomposition, zero-shot stepwise reasoning, and joint exemplar-explanation selection.The paper continues this line of work by focusing on diverse verification of reasoning steps.Source: §7 · p. 8

8 Conclusion

• DIVERSE introduces diverse prompts, a voting verifier, and a stepwise verifier that separately evaluates reasoning steps.The paper describes the stepwise verifier as especially novel and effective for analyzing model behavior.Source: §8 · p. 8
• Using code-davinci-002, DIVERSE achieves state-of-the-art performance on most reasoning tasks and surpasses PaLM with previous prompting techniques.Source: §8 · p. 8

8 Future Work

• Future work will address false-positive pseudo exemplars, improve diverse prompts beyond simple sampling, and extend DIVERSE to other tasks.Source: §8 · pp. 8–9

• Inference with DIVERSE requires large language models such as GPT-3 or PaLM, costing more time and budget than fine-tuning models like RoBERTa.Source: §9 · p. 9
• DIVERSE improves final-answer accuracy but cannot guarantee that generated reasoning paths are fully faithful.Source: §9 · p. 9
• DIVERSE needs more labeled reasoning paths for constructing diverse prompts and a training dataset for supervising the verifier.Source: §9 · p. 9
• Chain-of-thought prompting inserts a reasoning path before answer generation, with each reasoning step describing how an answer is derived.Source: A Preliminaries · p. 14
• DIVERSE constructs prompts by sampling exemplars, varying prompts across questions, and generating multiple sampled reasoning paths per prompt.Source: §B.1 · p. 14
• When exemplar data are insufficient, self-teaching generates pseudo reasoning paths and predicted answers to expand the exemplar base.Source: §B.1 · pp. 14–15

• Invalid self-taught reasoning paths can yield answers that coincide with the ground truth, creating misleading exemplars.Source: §B.2 · p. 15
• Two-choice StrategyQA is more vulnerable to this noise than four-choice CommonsenseQA because guessing a correct answer is easier while producing an invalid reasoning path.Source: §B.2 · p. 15
• 3.0% performance decline to PaLM occurs on StrategyQA, compared with comparable performance at −0.8% on CommonsenseQA.Source: §B.2 · p. 15

• The benchmark suite includes GSM8K, AsDiv, MultiArith, SVAMP, SingleEq, CommonsenseQA, StrategyQA, and CLUTRR.Source: C Data Statistics · p. 15
• StrategyQA contains 2,290 test cases, which are split into equal parts and alternated between exemplar construction and testing.Source: C Data Statistics · pp. 15–16
• CLUTRR experiments use its clean version rather than the supporting, irrelevant, or disconnected perturbed variants.Source: C Data Statistics · p. 16
• CLUTRR inputs append candidate answers, while outputs use reasoning paths constructed from knowledge-graph rules and text templates.Source: D Our Changes to CLUTRR · p. 16, Table 9 · p. 16
• Without candidate answers, the language models cannot exceed 50% accuracy in preliminary CLUTRR experiments.Source: D Our Changes to CLUTRR · p. 16
• DIVERSE selects correct-but-not-most frequent answers in some GSM8K examples, improving over majority voting.Source: Table 10 · p. 17