PaperlayerView exact revision on arXiv

Source-linked AI summary

Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim

arXiv:2305.04091v3cs.CL

TL;DR

Zero-shot-CoT reduces manual prompting effort but still exhibits calculation, missing-step, and semantic-understanding errors. The paper introduces PS and PS+ prompting, which plan subtasks and add detailed calculation guidance; across ten datasets, PS+ outperforms zero-shot baselines and performs similarly to few-shot CoT on arithmetic reasoning.

  • Problem

    Zero-shot-CoT still suffers from calculation, missing-step, and semantic-understanding errors in multi-step reasoning tasks.

  • Method

    Plan-and-Solve prompting asks LLMs to devise a plan, execute its subtasks, and, in PS+, follow additional instructions for variables and calculations.

  • Results

    Across ten datasets and three reasoning problems, Zero-shot-PS+ consistently outperforms Zero-shot-CoT, is comparable to or exceeds Zero-shot-PoT, and resembles 8-shot CoT performance in arithmetic reasoning.

  • Takeaways & Limitations

    Zero-shot PS+ can generate higher-quality reasoning than Zero-shot-CoT and has potential to outperform manual few-shot CoT prompting.

  • Takeaways & Limitations

    Prompt design requires effort because GPT-3 is sensitive to prompt expressions, and semantic-understanding errors remain unresolved.

Abstract

from arXiv · show

Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. To tackle multi-step reasoning tasks, few-shot chain-of-thought (CoT) prompting includes a few manually crafted step-by-step reasoning demonstrations which enable LLMs to explicitly generate reasoning steps and improve their reasoning task accuracy. To eliminate the manual effort, Zero-shot-CoT concatenates the target problem statement with "Let's think step by step" as an input prompt to LLMs. Despite the success of Zero-shot-CoT, it still suffers from three pitfalls: calculation errors, missing-step errors, and semantic misunderstanding errors. To address the missing-step errors, we propose Plan-and-Solve (PS) Prompting. It consists of two components: first, devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan. To address the calculation errors and improve the quality of generated reasoning steps, we extend PS prompting with more detailed instructions and derive PS+ prompting. We evaluate our proposed prompting strategy on ten datasets across three reasoning problems. The experimental results over GPT-3 show that our proposed zero-shot prompting consistently outperforms Zero-shot-CoT across all datasets by a large margin, is comparable to or exceeds Zero-shot-Program-of-Thought Prompting, and has comparable performance with 8-shot CoT prompting on the math reasoning problem. The code can be found at https://github.com/AGI-Edgerunners/Plan-and-Solve-Prompting.

1 Introduction

Zero-shot-CoT removes the need for manually crafted demonstrations but still exhibits calculation, missing-step, and semantic-understanding errors. Plan-and-Solve prompting addresses these problems by structuring reasoning into planned subtasks and adding detailed instructions, achieving broad gains across reasoning datasets.

  • Zero-shot-CoT avoids manual demonstrations by appending “Let’s think step by step” to the target problem.
  • 7% of errors were calculation errors, 12% were missing-step errors, and 27% were semantic-understanding errors.These figures come from the reported error analysis of arithmetic test examples.
  • Plan-and-Solve prompting first devises a plan dividing the task into subtasks, then carries out those subtasks step by step.
  • PS+ adds instructions to extract relevant variables, attend to calculation and commonsense, and calculate intermediate results.The added instructions target calculation errors and the quality of generated reasoning steps.
  • Across ten datasets and three reasoning problems, Zero-shot-PS+ consistently outperformed Zero-shot-CoT and was comparable to or better than Zero-shot-PoT.The evaluation used GPT-3 and included arithmetic, commonsense, and symbolic reasoning datasets.
  • PS+ achieved performance similar to 8-shot CoT prompting in arithmetic reasoning without requiring manual demonstration examples.

2 Plan-and-Solve Prompting

Plan-and-Solve prompting separates reasoning generation into planning and execution, while PS+ adds explicit guidance for variables, calculations, and intermediate results. A second prompt extracts the final answer from the generated reasoning.

  • 2 Plan-and-Solve Prompting: PS prompting generates a reasoning process that explicitly devises a plan and completes its subtasks.
  • 2.1 Step 1: Prompting for Reasoning Generation: The method aims to elicit subtasks and accomplish them while improving attention to calculations and intermediate results.
  • 2.1 Step 1: Prompting for Reasoning Generation: PS replaces Zero-shot-CoT’s trigger with instructions to devise a plan and carry it out step by step.
  • 2.1 Step 1: Prompting for Reasoning Generation: PS+ extends the plan-based trigger with instructions to pay attention to calculation and extract relevant variables with their numerals.
  • 2.1 Step 1: Prompting for Reasoning Generation: PS+ also instructs the model to calculate intermediate results and attend to commonsense during reasoning.
  • 2.2 Step 2: Prompting for Answer Extraction: A second prompt appends an answer-extraction instruction to the first prompt and the generated reasoning text.
  • 2.2 Step 2: Prompting for Answer Extraction: In the illustrated extraction example, the reasoning computes 125 + 498 = 623 and the model returns 623.

3 Experimental Setup

The study evaluates PS and PS+ across ten benchmarks spanning arithmetic, commonsense, and symbolic reasoning, comparing them with zero-shot and few-shot prompting baselines. GPT-3 is used as the backbone, and accuracy is reported across datasets.

  • 3.1 Benchmarks: The evaluation covers ten benchmark datasets across arithmetic, commonsense, and symbolic reasoning problems.
  • 3.1 Benchmarks: The arithmetic benchmarks include GSM8K, SVAMP, MultiArith, AddSub, AQuA, and SingleEq.
  • 3.2 Baselines: The study compares zero-shot PS and PS+ with Zero-shot-CoT, Zero-shot-PoT, Manual-CoT, and Auto-CoT baselines.
  • 3.2 Baselines: Zero-shot-CoT uses a trigger sentence without demonstrations, whereas Zero-shot-PoT generates Python programs whose executed outputs provide answers.
  • 3.2 Baselines: Manual-CoT uses eight hand-crafted demonstrations, while Auto-CoT automatically selects diverse examples and generates reasoning chains.
  • 3.3 Implementations: The experiments use public 175B-parameter GPT-3, specifically text-davinci-003 for the main results.
  • 3.3 Implementations: All experiments use temperature 0 for greedy decoding, and accuracy is reported across datasets.

4 Experimental Results

Across arithmetic, commonsense, and symbolic reasoning, PS and PS+ generally improve over Zero-shot-CoT, with PS+ often matching or exceeding competitive baselines. Analyses associate detailed planning and variable-focused instructions with fewer reasoning errors and stronger results.

  • Arithmetic Reasoning: PS+ improves arithmetic accuracy over Zero-shot-CoT on every dataset, by at least 5% except for a 2.9% improvement on GSM8K.PS prompting also achieves 2.5% higher average accuracy than Zero-shot-CoT.
  • Arithmetic Reasoning: PS+ outperforms PoT on five of six arithmetic datasets, while PS outperforms PoT on three.The results suggest that more detailed prompt instructions elicit higher-quality reasoning steps.
  • Arithmetic Reasoning: PS+ reaches 76.7% average accuracy across arithmetic datasets, below Manual-CoT at 77.6% but above Auto-CoT at 75.9%.The comparison indicates that zero-shot prompting can approach manually demonstrated chain-of-thought performance with less manual effort.
  • Symbolic Reasoning: On symbolic reasoning, PS+ scores 75.2% versus 70.6% for Manual-CoT and 65.2% for Zero-shot-CoT on Last Letters, while scoring 99.6% on Coin Flip.On Coin Flip, PS+ is slightly below Manual-CoT at 100.0% but above Zero-shot-CoT at 96.8%.
  • Prompt Analysis: With self-consistency, PS+ reaches 73.7% on GSM8K and 84.4% on SVAMP, versus 58.7% and 75.7% without it.With self-consistency, PS+ also exceeds Zero-shot-CoT, which reaches 70.7% and 81.7% on the same datasets.
  • Prompt Analysis: Among six trigger prompts, variable extraction alone performs worse than Zero-shot-CoT because it lacks instructions for devising and completing a plan.The finding supports combining detailed instructions with explicit planning rather than adding variable extraction in isolation.

5 Related Work

Related work develops prompting methods that elicit explicit reasoning from LLMs, progressing from few-shot demonstrations toward zero-shot and automatically constructed approaches. These methods target mathematical, commonsense, and logical reasoning tasks through different prompt formats and reasoning representations.

  • Reasoning Tasks: Complex reasoning benchmarks span mathematical, commonsense, and logical reasoning, requiring capabilities including calculation, multi-step reasoning, commonsense judgment, and logical reasoning.These task families motivate the prompting methods compared in the paper.
  • Chain-of-Thought Prompting: Chain-of-Thought prompting adds multiple reasoning steps before the answer and improves LLM performance on complex reasoning problems through few-shot demonstrations.Subsequent work explores prompt format and other ways to improve CoT prompting.
  • Zero-Shot and Automated Prompting: Zero-shot-CoT elicits reasoning without exemplars, while Auto-CoT automatically obtains examples by clustering a dataset before generating rationales.PoT instead uses code-oriented prompting to separate computation from reasoning.

6 Conclusion

The paper introduces PS and PS+ as zero-shot prompting strategies that organize reasoning into planned subtasks and improve over prior zero-shot baselines. Across ten datasets and three reasoning problem types, PS+ performs on par with few-shot CoT on multiple arithmetic datasets and may sometimes outperform manual few-shot CoT.

  • PS and PS+ guide LLMs to divide tasks into smaller subtasks, create a plan, and execute the subtasks according to that plan.
  • Evaluation on ten datasets across three reasoning problem types shows that PS+ outperforms previous zero-shot baselines.
  • PS+ performs on par with few-shot CoT on multiple arithmetic reasoning datasets without requiring manual demonstration examples.
  • The results suggest that PS+ can generate higher-quality reasoning than Zero-shot-CoT and may outperform manual few-shot CoT in some instances.
  • The authors identify refining the plan and applying PS(+) to non-reasoning tasks as directions for future work.

7 Limitations

The paper has two limitations: prompt design requires effort because GPT-3 is sensitive to prompt wording, and semantic misunderstanding errors remain unresolved by plan-and-solve prompting.

  • Prompt design requires careful effort because GPT-3 models are sensitive to the expressions used in prompts.
  • Plan-and-solve prompting addresses calculation and missing-reasoning-step errors, but semantic misunderstanding errors remain.

8 Ethics

The ethics section identifies the evaluated datasets and states that the prompts avoid personal information and discriminatory language. The listed dataset licenses vary, with some unspecified.

  • The experiments use six math, two commonsense, and two symbolic reasoning datasets, including AQuA, GSM8K, MultiArith, AddSub, SingleEq, SVAMP, CommonsenseQA, StrategyQA, Last Letter, and Coin Flip.
  • GSM8K and SVAMP use MIT License code, AQuA and StrategyQA use Apache-2.0 code, and the remaining datasets have unspecified licensing in the passage.
  • The prompts do not collect or use personal information about other individuals.
  • The prompts do not contain words that discriminate against any individual or group.

A Appendix

The appendix reports results for all tried prompts and provides example texts generated by Zero-shot-PS+. Unless otherwise stated, experiments use GPT3 text-davinci-003.

  • The appendix contains results for all prompts tried and example texts generated by Zero-shot-PS+.
  • Unless otherwise mentioned, the experiments use the GPT3 text-davinci-003 model.

A.1 Results of All Trigger Sentences

Tables 7 to 16 provide the results for all tried prompts across the evaluated datasets, documenting the performance comparisons for this section.

  • Tables 7 to 16 list results for all prompts tried on each dataset.
  • The tables collectively document prompt-performance comparisons across the datasets.
  • This section consolidates the complete set of tried-prompt results rather than a single dataset or prompt.

A.2 Example Outputs by Zero-shot-PS+

This section presents prompt comparisons for Step 1 of Zero-shot-PS+ and example outputs generated by Zero-shot-PS+ across the evaluated datasets.

  • Performance comparisons: The Step 1 prompt comparisons cover AQuA, GSM8K, MultiArith, SVAMP, AddSub, SingleEq, CSQA, StrategyQA, Last Letters, and Coin Flip.
Loading 2305.04091v3…