PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Zero-Shot Reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa

arXiv:2205.11916v4cs.CLcs.AIcs.LG

TL;DR

Large language models have struggled with multi-step reasoning, while strong results from chain-of-thought prompting have largely been attributed to few-shot learning. This paper introduces Zero-shot-CoT, a single task-agnostic prompt, and shows substantial gains over standard zero-shot prompting across diverse reasoning tasks, including MultiArith accuracy rising from 17.7% to 78.7%.

  • Problem

    Large language models struggled with multi-step system-2 reasoning, leaving limited evidence about their zero-shot reasoning abilities beyond few-shot prompting.

  • Method

    Zero-shot-CoT adds “Let’s think step by step” to a single task-agnostic prompt to elicit chain-of-thought reasoning without few-shot examples.

  • Results

    Zero-shot-CoT substantially outperformed standard zero-shot prompting across arithmetic, symbolic, and logical reasoning tasks, raising MultiArith accuracy from 17.7% to 78.7%.

  • Takeaways & Limitations

    A simple single prompt can elicit broad, multi-task reasoning capabilities from large language models without hand-crafted few-shot examples.

  • Takeaways & Limitations

    The study lacks public information about the training datasets used for the evaluated language models.

Abstract

from arXiv · show

Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding "Let's think step by step" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.

1 Introduction

The paper argues that large language models can perform multi-step reasoning without task-specific examples by adding the prompt “Let’s think step by step.” Zero-shot-CoT substantially improves over standard zero-shot prompting, though it remains below carefully crafted Few-shot-CoT.

  • Motivation: 100B-plus-parameter language models struggled with system-2 tasks requiring slow, multi-step reasoning despite strong system-1 performance.This motivates prompting methods designed to elicit intermediate reasoning.
  • Method: Adding “Let’s think step by step” before answering makes LLMs decent zero-shot reasoners without task-specific examples.Zero-shot-CoT uses the same simple prompt to facilitate step-by-step thinking across questions.
  • Results: 17.7% to 78.7% on MultiArith with text-davinci-002 shows Zero-shot-CoT’s gain over the zero-shot baseline.The comparison is reported for the large-scale InstructGPT model.
  • Results: 10.4% to 40.7% on GSM8K with text-davinci-002 shows another substantial improvement over zero-shot prompting.Zero-shot-CoT underperforms Few-shot-CoT but achieves enormous gains over the zero-shot baseline.
  • Results: 540B-parameter PaLM exhibits similar magnitudes of improvement on MultiArith and GSM8K, while zero-shot scaling becomes significantly better.The improved scaling curve is comparable to that of the few-shot CoT baseline.

2 Background

This section introduces large language models and prompting, then reviews chain-of-thought prompting for multi-step reasoning. Scaling larger models and datasets has improved LLM performance across downstream NLP tasks, while CoT prompting boosts performance on challenging arithmetic and logical benchmarks.

  • Overview: The background covers two preliminary concepts: LLMs and prompting, and CoT prompting for multi-step reasoning.These concepts form the basis of the work.
  • Large language models and prompting: LLMs estimate probability distributions over text and have become adept at downstream NLP tasks through scaling model size and training data.The reviewed scale ranges from millions to hundreds of billions of parameters, alongside larger datasets such as webtext corpora.
  • Chain of thought prompting: CoT prompting modifies few-shot answers into step-by-step solutions to address difficult multi-step arithmetic and logical reasoning benchmarks.The passage describes CoT as a few-shot prompting method that achieves significant performance boosts, particularly with very large language models.

3 Zero-shot Chain of Thought

Zero-shot-CoT is a task-agnostic, template-based prompting method that elicits multi-hop reasoning without few-shot examples. It uses a two-stage pipeline: first extracting a reasoning path, then extracting the final answer in the correct format.

  • Method overview: Zero-shot-CoT elicits multi-hop reasoning across diverse tasks with one task-agnostic template, without step-by-step few-shot examples.This reduces prompt engineering compared with Few-shot-CoT, which requires carefully designed task-specific examples.
  • Pipeline comparison: Unlike few-shot prompting, Zero-shot-CoT prompts the model twice because it separately extracts reasoning and answers in the correct format.The zero-shot baseline also uses answer-extraction prompting, whereas few-shot examples can directly demonstrate the required answer format.
  • 1st prompt: reasoning extraction: The first prompt transforms question x into x′ using “Q: [X]. A: [T]”, where [T] is a hand-crafted trigger sentence.With “Let’s think step by step” as the trigger, the prompt becomes “Q: [X]. A: Let’s think step by step.”
  • 1st prompt: reasoning extraction: The reasoning prompt feeds x′ to a language model to generate a full reasoning path for the input question.The generated reasoning text is denoted z and is used by the subsequent answer-extraction stage.
  • 2nd prompt: answer extraction: The second prompt concatenates x′, generated reasoning z, and an answer trigger [A] to extract the final answer.Because z is generated by the same language model, this stage is self-augmented.

4 Experiment

Experiments across 12 reasoning datasets and 17 models show that Zero-shot-CoT substantially improves performance over standard zero-shot prompting on arithmetic, symbolic, and other logical reasoning tasks. Gains are strongest when tasks require multi-step reasoning, while commonsense performance does not improve.

  • Tasks and datasets: 12 datasets span arithmetic, commonsense, symbolic, and other logical reasoning tasks.The evaluation covers six arithmetic datasets, two commonsense datasets, two symbolic datasets, and two logical reasoning datasets.
  • Models: 17 models include Instruct-GPT3, original GPT3, PaLM, and additional models for scaling analysis.The main experiments use multiple GPT-3 and PaLM sizes, while GPT-2, GPT-Neo, GPT-J, T0, and OPT support model-scaling studies.
  • Zero-shot-CoT vs. Zero-shot: Zero-shot-CoT substantially outperforms standard zero-shot prompting on four arithmetic tasks, all symbolic tasks, and all other logical reasoning tasks.The arithmetic improvements occur on MultiArith, GSM8K, AQUA, and SVAMP, while SingleEq and AddSub remain on par.
  • Zero-shot-CoT vs. Zero-shot: 17.7% to 78.7% is the MultiArith score gain, while 10.4% to 40.7% is the GSM8K score gain.These gains are reported for the large InstructGPT model text-davinci-002.
  • Zero-shot-CoT vs. Zero-shot: Zero-shot-CoT does not improve commonsense reasoning, although generated chains of thought are often logically correct or contain human-understandable mistakes.The reported commonsense results concern CommonsenseQA and StrategyQA, with larger-model effects noted for StrategyQA.
  • Does model size matter for zero-shot reasoning?: Chain-of-thought performance rises sharply with model size, whereas standard zero-shot performance remains flat or increases slowly.This pattern is observed on MultiArith and GSM8K across Original/Instruct GPT-3 and PaLM models.

5 Discussion and Related Work

The discussion situates Zero-shot-CoT among approaches that elicit step-by-step reasoning through fine-tuning or few-shot prompting, while distinguishing it from instruction tuning and task-specific prompt engineering. It also acknowledges limitations concerning opaque training data and biases inherited from web-scale corpora.

  • Reasoning Ability of LLMs: Pretrained models’ reasoning ability can increase substantially when they produce step-by-step reasoning through fine-tuning or few-shot prompting.The paper summarizes prior work spanning arithmetic and commonsense reasoning tasks.
  • Zero-shot Abilities of LLMs: Zero-shot-CoT is orthogonal to instruction tuning and improves zero-shot performance for InstructGPT, vanilla GPT3, and PaLM.The paper contrasts this approach with explicitly fine-tuning models to follow instructions.
  • From Narrow (task-specific) to Broad (multi-task) Prompting: Most prompts are task-specific, whereas the paper positions its approach as moving from narrow task-specific prompting toward broad multi-task prompting.The discussion notes that both few-shot prompts and many zero-shot prompts commonly rely on task-specific examples or template engineering.
  • Training Dataset Details: A limitation is the lack of public information about the training datasets underlying different GPT and PaLM model variants.Despite this limitation, the passage reports large Zero-shot-to-Zero-shot-CoT gains across recent large models and arithmetic and nonarithmetic tasks.
  • Limitation and Social Impact: Because LLMs capture and amplify biases in web training data, prompting inherits those shortcomings while exploiting patterns learned by the models.The paper identifies this as a social-impact limitation of its prompting-based approach.

6 Conclusion

The paper proposes Zero-shot-CoT, a single zero-shot prompt that elicits chain-of-thought reasoning across diverse tasks without hand-crafted examples for each task. It presents the method as a minimalist and strong zero-shot baseline for difficult multi-step system-2 reasoning tasks.

  • Contribution: Zero-shot-CoT uses a single zero-shot prompt to elicit chain of thought across a variety of reasoning tasks.This contrasts with prior few-shot in-context approaches requiring hand-crafted examples per task.
  • Contribution: The method is presented as a minimalist and strongest zero-shot baseline for difficult multi-step system-2 reasoning tasks.These tasks had long evaded the scaling laws of large language models.
  • Contribution: The work encourages further discovery of capabilities associated with zero-shot reasoning in large language models.

Checklist … A.2 Datasets

The checklist reports that the paper accurately states its claims and scope, discusses limitations and potential negative societal impacts, and documents experimental reproducibility and asset-use practices. Theoretical-results and human-subjects requirements were marked not applicable, while error bars were omitted because experiments used deterministic greedy decoding.

  • Checklist: The authors affirmed that the abstract and introduction accurately reflect the paper’s contributions and scope, and that limitations are described.
  • Checklist: Potential negative societal impacts were discussed, and the authors confirmed conformity with ethics review guidelines.
  • Checklist: Theoretical-results checklist items on assumptions and complete proofs were marked N/A.
  • Checklist: The paper provides code, data, instructions, and training details needed to reproduce the main experimental results.
  • Checklist: No error bars were reported because the experiments mainly used the GPT-3 API with greedy decoding and had no randomness.
  • Checklist: The authors reported the total compute and resource types used for the experiments.
  • Checklist: Existing assets were cited and licensed, while new assets were included in supplemental material or provided through a URL.
  • Checklist: The paper addressed consent, personally identifiable information, and offensive content in used or curated data; crowdsourcing and human-subjects items were marked N/A.

A.2.1 Dataset Description … A.6 Answer Cleansing

The experiments use publicly available datasets alongside two newly created reasoning datasets, multiple language models, standardized decoding and extraction procedures, and dataset-specific answer cleansing. The appendices specify how these datasets, models, prompts, and cleansing approaches were constructed and applied.

  • A.2.1 Dataset Description: Table 7 summarizes the description of each dataset used in the experiments.The experiments used publicly available datasets except for Last Letters and Coin Flip, which were created by the authors.
  • A.2.2 Dataset creation: Last Letter Concatenation was created by inserting four randomly selected human names into a template requiring their final letters to be concatenated.Names were sampled from the names-dataset library, following Wei et al. [2022] with minor question-template rephrasing.
  • A.2.2 Dataset creation: Coin Flip was created by inserting four randomly selected human names and randomly assigning “flips” or “does not flip” to each action in a coin-state question.The template begins with a heads-up coin and asks whether it remains heads up after the specified actions, where “flip” means “reverse.”
  • A.4 Implementation details: OpenAI API was used for Original GPT-3 and Instruct-GPT3, while Hugging Face Transformer Library was used for OPT, T0, GPT-J, GPT-Neo, and GPT-2.These implementation choices identify the software interfaces used for the evaluated model families.
  • A.5 Prompts For Answer Extraction: Tables 9 and 10 list answer-extraction prompts, with Zero-shot and Zero-shot-CoT used as the default prompts across experiments.The tables distinguish extraction prompts for the Zero-shot and Zero-shot-CoT conditions, including task abbreviations such as C.S.QA, D.U., and S.O.
  • A.6 Answer Cleansing: Table 11 summarizes answer-cleansing approaches used across all experiments and links answer formats to datasets through Table 7.The cleansing procedures are described in relation to the corresponding dataset answer formats.

B Additional Experiment Results · C Sample Study · C.1 CommonsenseQA

The additional results provide Zero-shot-CoT examples across datasets, templates, models, and prompting conditions, while the sample study examines the correctness of generated reasoning on CommonsenseQA. The analysis finds that correct predictions can contain flawed chains of thought, whereas incorrect predictions often reflect commonsense mistakes despite logically coherent reasoning.

  • B Additional Experiment Results: Zero-shot-CoT examples span each evaluated dataset, reasoning-extraction template, and language model.The appendix also includes examples for Few-shot, Few-shot-CoT, cross-task Few-shot-CoT, Zero-Plus-Few-Shot-CoT, and PaLM outcome comparisons.
  • B Additional Experiment Results: $330,000 is correctly derived for the plane’s first-year purchase and maintenance cost using step-by-step arithmetic.The response sums the $150,000 purchase, $60,000 hanger cost, and $120,000 fuel cost.
  • B Additional Experiment Results: 120,000 litres is correctly obtained by calculating the tank’s 120m^3 volume and converting cubic metres to litres.The response selects answer choice D.
  • B Additional Experiment Results: PaLM outcome examples include both successful and unsuccessful Zero-shot-CoT and Zero-shot cases, alongside an instability in which prompting can produce continuous newline tokens.The cited examples include incomplete or incorrect generated solutions and a footnote attributing some errors to PaLM instability.
  • C Sample Study: The sample study validates chain-of-thought correctness by analyzing Zero-shot-CoT texts from CommonsenseQA and MultiArith with Instruct-GPT3 (text-davinci-002).For CommonsenseQA, researchers randomly selected 50 correct and 50 incorrect samples and categorized them by CoT type.
  • C.1 CommonsenseQA: Correct CommonsenseQA predictions sometimes contain incorrect chains of thought because the model gives multiple answers while its first answer is correct.The main tendency is failure to narrow the prediction to one answer choice.
  • C.1 CommonsenseQA: Incorrect CommonsenseQA predictions most frequently involve commonsense mistakes, even when the generated chain of thought is flexible, reasonable, and logically correct.The reasoning can therefore lack common sense despite appearing coherent.

C.2 MultiArith

On MultiArith, the study compared Zero-shot-CoT and Few-shot-CoT by categorizing sampled correct and incorrect chains of thought. Both methods usually produced logically consistent reasoning on correct samples, but their errors followed different patterns.

  • Method: The comparison randomly examined 50 correct and 50 incorrect samples generated by each method, except that Few-shot-CoT produced only 42 incorrect samples.The samples were categorized by type to contrast the chains of thought produced by Zero-shot-CoT and Few-shot-CoT.
  • Correct samples: Almost all chains of thought in correct samples were logical and consistent with the correct prediction, although Zero-shot-CoT made slightly more reasoning mistakes than Few-shot-CoT.This result concerns the sampled correct predictions from both methods.
  • Incorrect samples: Zero-shot-CoT sometimes added unnecessary reasoning after reaching the correct prediction, changing it to an incorrect answer.This was identified as one of the main error patterns among incorrect samples.
  • Incorrect samples: Zero-shot-CoT also sometimes failed to begin reasoning and merely rephrased the input question.This contrasts with errors caused by reasoning steps that altered an initially correct prediction.
  • Incorrect samples: Few-shot-CoT tended to fail when its chain of thought included ternary operations such as (3+2)∗4.The passage contrasts this failure mode with Zero-shot-CoT’s unnecessary-step and rephrasing errors.

D Further Zero-shot Experiments with PaLM 540B

The study further evaluates Zero-shot-CoT with PaLM 540B, both without and with self-consistency. The reported evaluation metric is accuracy.

  • PaLM 540B evaluation: Zero-shot-CoT was additionally evaluated on PaLM 540B.The evaluation included settings without and with self-consistency.
  • Self-consistency: Self-consistency generates reasoning paths by random sampling N times.It then determines the final prediction through majority voting.
  • Evaluation metric: The PaLM 540B experiment reports Accuracy as its evaluation metric.

E Detailed experiment results of model scale study

This section presents a model-scale study using accuracy on MultiArith and GSM8K, with results summarized through Tables 26 and 27 and a corresponding Figure 3 curve. The study reports that chain-of-thought effectiveness increases with model size and establishes an ascending performance order from zero-shot to few-shot-CoT prompting.

  • Experimental setup: The model-scale study evaluates accuracy on MultiArith and GSM8K, with Figure 3 curves derived from Tables 26 and 27.Table 26 covers MultiArith, while Table 27 covers GSM8K with PaLM.
  • Model scale findings: CoT is effective when models are larger, including Instruct GPT-3 and Original GPT-3.The cited models include text-davinci-001, text-davinci-002, and Original GPT-3 at 175B parameters.
  • Model scale findings: Performance increases in the order Zero-shot, Few-shot (8samples), Zero-shot-CoT, and Few-shot-CoT (8samples).This ascending order is reported for the model-scale experiment.
Loading 2205.11916v4…