PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou

arXiv:2203.11171v4cs.CLcs.AI

TL;DR

Chain-of-thought prompting commonly relies on a single greedily decoded reasoning path, limiting exploration of alternative solutions. The paper introduces self-consistency, which samples diverse paths and selects the most consistent answer, significantly improving accuracy across arithmetic and commonsense reasoning benchmarks.

  • Problem

    Chain-of-thought prompting uses greedy decoding, leaving open whether diverse reasoning paths can improve language-model reasoning performance.

  • Method

    Self-consistency samples diverse reasoning paths from one language model and marginalizes them to select the most consistent final answer.

  • Results

    Self-consistency significantly improves arithmetic reasoning across four language models and achieves new state-of-the-art results on almost all tasks, including +12%-18% absolute accuracy gains on AQuA and GSM8K for larger models.

  • Takeaways & Limitations

    Self-consistency also supports rationale collection, uncertainty estimation, and improved calibration of language-model outputs.

Abstract

from arXiv · show

Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).

1 INTRODUCTION

The paper introduces self-consistency, a decoding strategy that replaces greedy chain-of-thought decoding by sampling diverse reasoning paths and selecting the most consistent answer. It is unsupervised, requires no additional training or annotations, and improves chain-of-thought reasoning across arithmetic and commonsense tasks and multiple language models.

  • Method: Self-consistency replaces greedy chain-of-thought decoding with a sample-and-marginalize procedure over diverse reasoning paths.The method samples multiple paths, then marginalizes them and chooses the most consistent final answer.
  • Method: The method relies on the intuition that complex reasoning tasks can have multiple reasoning paths reaching a correct answer.Different sampled paths may produce different final answers, which are aggregated to select the most consistent one.
  • Advantages: Self-consistency is unsupervised, works with pretrained language models, and requires no human annotations, additional training, auxiliary models, or fine-tuning.This makes it simpler than approaches that train verifiers or rerankers to improve generation quality.
  • Evaluation: Across four language models of varying scales, self-consistency improves chain-of-thought prompting on arithmetic and commonsense reasoning tasks.The evaluation includes UL2-20B, GPT-3-175B, LaMDA-137B, and PaLM-540B.

2 SELF-CONSISTENCY OVER DIVERSE REASONING PATHS

Self-consistency replaces greedy reasoning with diverse sampled reasoning paths and selects answers by aggregating or marginalizing across those paths. It benefits fixed-answer reasoning tasks while remaining limited by the requirement for a defined answer set or consistency metric.

  • Method: Self-consistency samples diverse candidate reasoning paths after chain-of-thought prompting, then aggregates their final answers by marginalizing over the paths.The method is compatible with temperature, top-k, and nucleus sampling.
  • Method: Each sampled output couples a latent reasoning sequence r_i with an answer a_i, where the reasoning path is used to reach the final answer.Answers are drawn from a fixed set A across m sampled outputs.
  • Answer aggregation: Answer aggregation can use majority vote or probability-weighted sums based on P(r_i, a_i | prompt, question).Weights may use unnormalized sequence probability or length-normalized conditional probability.
  • Answer aggregation: The unweighted sum produces very similar accuracy to the normalized weighted sum because output probabilities do not strongly distinguish correct from incorrect solutions.The probability-based comparison and calibration explanation are reported across the aggregation discussion.
  • Motivation: Introducing diversity in reasoning can improve tasks with fixed answers, despite their conventional reliance on greedy decoding.Self-consistency occupies an intermediate space between open-ended and fixed-answer generation.
  • Limitations: Self-consistency applies directly only when final answers come from a fixed set, but could extend to open-text generation if answer consistency can be measured.Possible consistency metrics include whether generated answers agree or contradict each other.

3 EXPERIMENTS

Experiments evaluate self-consistency across reasoning benchmarks and language models, comparing it with greedy chain-of-thought decoding and alternative decoding or ensembling methods. The results show robust accuracy gains, stronger improvements at larger scales, and benefits from diverse reasoning paths.

  • Experimental setup: Experiments span arithmetic, commonsense, and symbolic reasoning benchmarks across four transformer-based language models with varying scales.The models include UL2-20B, GPT-3 175B, LaMDA-137B, and PaLM-540B.
  • Experimental setup: Self-consistency is evaluated over 10 runs with 40 independently sampled decoder outputs per run, against chain-of-thought prompting with greedy decoding.Sampling uses temperature-based decoding, with top-k truncation for UL2-20B, LaMDA-137B, and PaLM-540B.
  • Arithmetic reasoning: +3%-6% absolute accuracy improvement occurs over UL2-20B, compared with +9%-23% for LaMDA-137B and GPT-3 on arithmetic reasoning.Self-consistency significantly improves arithmetic reasoning over chain-of-thought prompting across all four language models, with larger gains as model scale increases.
  • Commonsense and symbolic reasoning: Self-consistency yields large gains across commonsense and symbolic reasoning tasks and obtains state-of-the-art results on 5 out of 6 tasks.The symbolic reasoning evaluation includes a more challenging out-of-distribution setting.
  • Reasoning-path sampling: 40 sampled reasoning paths consistently outperform smaller sample counts, showing that greater path diversity improves performance.Accuracy and standard deviation are evaluated over 10 runs for 1, 5, 10, 20, and 40 sampled paths.
  • Comparisons: Self-consistency outperforms beam search significantly on both tested tasks and can boost performance when chain-of-thought hurts standard prompting.On PaLM-540B, it also outperforms standard prompting for tasks including ANLI-R1, e-SNLI, and RTE.

4 RELATED WORK

Prior work addresses language-model reasoning through specialized methods, diverse decoding, re-ranking, reasoning-path extraction, and consistency improvements. Self-consistency differs by applying broadly without additional supervision or fine-tuning.

  • Reasoning in language models: Language models struggle with arithmetic, logical, and commonsense reasoning, motivating specialized approaches for improving reasoning.
  • Sampling and re-ranking in language models: Prior decoding research includes temperature, top-k, nucleus, minimum Bayes risk, and typical sampling, alongside methods promoting diversity.
  • Sampling and re-ranking in language models: Re-ranking methods improve generation quality using human-annotation-trained filters, verifiers for mathematical solutions, and consistency-oriented approaches.
  • Extract reasoning paths: Earlier work identifies reasoning paths through semantic graphs, Wikipedia retrieval, human annotations, or heuristic-based pseudo reasoning paths.
  • Consistency in language models: Language-model inconsistency has been studied in conversation, explanation generation, factual knowledge extraction, recurrent sequences, and logical reasoning.

5 CONCLUSION AND DISCUSSION

Self-consistency is a simple method that improves accuracy across arithmetic and commonsense reasoning tasks and multiple language models, while also supporting rationale collection, uncertainty estimation, and calibration. Its main limitation is increased computation cost, although a small number of sampled paths often captures most gains, motivating future use for supervised-data generation and fine-tuning.

  • Self-consistency significantly improves accuracy across arithmetic and commonsense reasoning tasks and four large language models with varying scales.
  • Beyond accuracy, self-consistency supports collecting rationales, estimating uncertainty, and improving calibration of language-model outputs.
  • Self-consistency incurs higher computation cost, but trying 5 or 10 paths often captures most gains because performance typically saturates quickly.The passage identifies 5 or 10 paths as practical starting points.
  • Future work could use self-consistency to generate better supervised data for fine-tuning models toward more accurate single-inference predictions.

REPRODUCIBILITY STATEMENT

The experiments used four language models at varying scales, including public UL2 and GPT-3 resources to support reproducibility.

  • REPRODUCIBILITY STATEMENT: Four language models with varying scales were evaluated, including open-source UL2 checkpoints and GPT-3’s public API with engines code-davinci-001 and code-davinci-002.UL2 checkpoints are available through Google Research, while GPT-3’s public engines further aid reproducibility.

ETHICS STATEMENT

The authors caution that language models may produce nonsensical or non-factual reasoning paths, so their outputs require extra care. Generated rationales are primarily for inspecting answers and may help investigate mistakes or biases.

  • Language models can generate nonsensical or non-factual reasoning paths, requiring cautious use of their outputs.
  • Generated rationales are used mainly to inspect how a model reaches its answer.
  • Generated rationales could help examine why models make mistakes or whether they contain biases on specific tasks.

A APPENDIX · A.1 ADDITIONAL EXPERIMENT RESULTS · A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS

The appendix evaluates self-consistency across sampling strategies, sampling parameters, model scales, and sampled reasoning paths. It finds robustness across sampling choices, improvements over greedy decoding, and benefits from diverse paths during answer aggregation.

  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Self-consistency remains robust when varying temperature-sampling parameter T and Top-k sampling parameter k on LaMDA-137B.The evaluation appears in Figure 6 and covers different sampling strategies and parameters.
  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Figure 6 evaluates GSM8K accuracy over LaMDA-137B under various sampling strategies and sampling parameters.The figure specifically reports GSM8K accuracy.
  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Self-consistency improves substantially over greedy decoding on both LaMDA-137B and PaLM-540B.These gains build on the high accuracy already achieved by scaling model sizes.
  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Across arithmetic and commonsense reasoning tasks, increasing the number of diverse reasoning paths consistently improves accuracy over LaMDA-137B.Figure 7 presents self-consistency results across these tasks.
  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Additional sampled reasoning paths are reported for LaMDA-137B in Table 12 and PaLM-540B in Table 13.The appendix provides examples from both models.
  • A.1.1 ROBUSTNESS TO SAMPLING STRATEGIES AND PARAMETERS: Diversity among additionally sampled reasoning paths helps the model reach a more correct final answer after aggregation.This conclusion is drawn from the sampled paths shown for both models.

A.1.2 ROBUSTNESS TO DIFFERENT SETS OF PROMPTS … A.3 FULL SETS OF PROMPTS

The appendix shows that self-consistency remains effective across manually written prompt sets and compares it with model ensembling and combined ensemble strategies. It also documents inference resources, example error repairs, and the exact prompts used across the evaluated datasets.

  • A.1.2 ROBUSTNESS TO DIFFERENT SETS OF PROMPTS: Self-consistency yields consistent gains over the original CoT approach across three manually written sets of chain-of-thought prompts.Table 9 reports GSM8K accuracy over PaLM-540B and evaluates robustness to different input prompts.
  • A.1.3 COMPARED TO MODEL ENSEMBLES: 74.4% accuracy is achieved by self-consistency over PaLM-540B, outperforming greedy majority voting across three language models.The multiple-model ensemble averages predictions over 10 runs, while lower-capacity models can drag down the higher-capacity model’s performance.
  • A.1.4 COMBINING SELF-CONSISTENCY WITH OTHER ENSEMBLING STRATEGIES: Self-consistency is compatible with other ensemble strategies, but its gains are significantly higher and can override their performance improvements.The comparison uses 40 prompt sets or permutations versus 40 self-consistency paths, with all experiments based on PaLM-540B.
  • A.2 DETAILS ON RESOURCES AND INFERENCE: Inference uses prompting-based evaluation across UL2, GPT-3, LaMDA-137B, and PaLM-540B, with most jobs taking 1 to 4 hours or 2 to 12 hours depending on model.The appendix specifies TPU configurations for UL2, LaMDA-137B, and PaLM-540B, and public-API evaluation for GPT-3 models.
  • A.2 DETAILS ON RESOURCES AND INFERENCE: Generated outputs are parsed until the start of the next “Q:”, while GPT-3 experiments use 128 max tokens without frequency or presence penalties.These settings are applied consistently with the prompting format.
  • A.2 DETAILS ON RESOURCES AND INFERENCE: Additional examples show self-consistency repairing greedy-decoding errors on LaMDA-137B and PaLM-540B using sampled reasoning paths consistent with ground truth.Tables 12 and 13 each present two such reasoning paths.
  • A.3 FULL SETS OF PROMPTS: The appendix provides exact few-shot prompts for AQUA-RAT, ARC, HotpotQA, arithmetic tasks, NLI datasets, and closed-book question-answering tasks.Prompt sources include manually composed exemplars, prompts from Wei et al. (2022), and crowd-sourced rationales for e-SNLI.
Loading 2203.11171v4…