PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data

KaShun Shum, Shizhe Diao, Tong Zhang

arXiv:2302.12822v3cs.CL

TL;DR

CoT prompting depends on manually designed rationale chains, limiting easy adaptation when labeled data lacks explanations. Automate-CoT automatically generates, prunes, and selects rationale chains, achieving superior results across reasoning and non-reasoning tasks. Its linguistic-style definition remains non-rigorous.

  • Problem

    Most CoT methods rely on human-written rationale chains, requiring substantial effort to select questions and design reasoning steps for new datasets.

  • Method

    Automate-CoT augments a labeled dataset with model-generated rationale chains, prunes chains inconsistent with ground-truth answers, and selects exemplars using variance-reduced policy gradients.

  • Results

    Automate-CoT achieves superior results across arithmetic, commonsense, symbolic, and non-reasoning tasks, outperforming Manual-CoT and self-consistency by 2.6% and 3.7% on average with text-davinci-002.

  • Takeaways & Limitations

    Automate-CoT provides an automatic way to construct and optimize CoT prompts across multiple task types without relying on human-authored rationale chains.

  • Takeaways & Limitations

    The paper does not rigorously define what constitutes good or bad linguistic style and cannot determine what perfect style entails.

Abstract

from arXiv · show

Chain-of-thought (CoT) advances the reasoning abilities of large language models (LLMs) and achieves superior performance in complex reasoning tasks. However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains. This paper proposes a new strategy, Automate-CoT (Automatic Prompt Augmentation and Selection with Chain-of-Thought), that can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and then pruning low-quality chains to construct a candidate pool of machine-generated rationale chains based on the labels. Finally, it selects the optimal combination of several rationale chains from the pool for CoT prompting by employing a variance-reduced policy gradient strategy to estimate the significance of each example. Automate-CoT enables a quick adaptation of the CoT technique to different tasks. Experimental results demonstrate the effectiveness of our method, where competitive results are achieved on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and non-reasoning tasks (+2.5%). The code is available at https://github.com/SHUMKASHUN/Automate-CoT.

1 Introduction

CoT prompting improves LLM reasoning by using rationale chains as exemplars, but manually designing and selecting those chains is costly and difficult to adapt. Automate-CoT addresses this by automatically generating, pruning, and selecting rationale chains.

  • CoT prompting adds rationale chains as exemplars to show intermediate reasoning steps.
  • Manual CoT development requires selecting questions and carefully designing reasoning steps for each exemplar.
  • Automate-CoT generates model-produced rationale chains, prunes low-quality chains using answer consistency, and selects useful chains automatically.
  • Automate-CoT is designed to find optimal and diverse CoT prompts while reducing dependence on manually written exemplars.

2 Motivation

CoT performance is sensitive to exemplar order, reasoning complexity, diversity, and linguistic style. These sensitivities make manually constructing effective prompts labor-intensive and motivate automated prompt optimization.

  • Order Sensitivity: GPT-3 CoT performance varies with exemplar order, with GSM8K accuracy reaching 59.8%, 3.3% below the reported 63.1% average.
  • Complexity: Manual-CoT uses simple exemplars of at most three hops, while Complex-CoT improves complex questions but performs poorly on simple ones.
  • Diversity: Combining exemplars with different complexity levels outperforms using only complex exemplars, indicating a complexity-diversity trade-off.
  • Style Sensitivity: Different annotator styles can produce up to 28.2% accuracy differences in symbolic reasoning, while identifying good styles requires validation-set trial and error.

3 Approach

Automate-CoT constructs a pool of labeled, machine-generated rationale chains, removes inconsistent chains, and optimizes exemplar selection with variance-reduced policy gradients under prompt constraints.

  • Augment and Prune: The method starts from labeled questions and answers and organizes its architecture around augment-and-prune followed by selection.
  • Augment and Prune: For each question, the language model generates rationale chains that augment a candidate pool, with k = 1 used to balance cost and efficiency.
  • Augment and Prune: Generated chains are pruned using consistency between predicted and ground-truth answers to obtain a high-quality exemplar pool.
  • Select: Selection is necessary because context length, fair comparison, and sensitivity constrain how many exemplars can be included.
  • Select: Because prompt gradients are inaccessible, VR-PGE estimates gradients through forward propagation and updates the exemplar distribution with projected stochastic gradient descent.

4 Experimental Settings

The evaluation covers eleven reasoning and non-reasoning datasets using exact-match accuracy, compares Automate-CoT with established prompting baselines, and specifies API-based training and inference settings.

  • Datasets and Evaluation Metrics: Experiments evaluate eight reasoning tasks and three non-reasoning tasks, including arithmetic, commonsense, symbolic, question-answering, inference, and sentiment datasets.
  • Datasets and Evaluation Metrics: All tasks use exact-match accuracy after prediction preprocessing removes special symbols before comparison with ground truth.
  • Baselines: Automate-CoT is compared with Manual-CoT, self-consistency, and Auto-CoT using text-davinci-002 and code-davinci-002 APIs.
  • Implementation: The main experiments construct pools of 100 high-quality exemplars while retaining four to eight exemplars for prompting.
  • Implementation: Inference queries the model once by default, or 40 times under self-consistency before choosing the most consistent answer.

5 Experimental Results

Automate-CoT outperforms the compared prompting baselines across reasoning and non-reasoning tasks. Its gains hold across language models and include improvements on arithmetic, commonsense, symbolic, and general language tasks.

  • 2.6% and 3.7% average gains over Manual-CoT and SC, respectively, are achieved with text-davinci-002.Automate-CoT achieves superior results on all tasks under this model.
  • 2.7% and 2.2% average gains over Manual-CoT and SC, respectively, are achieved with code-davinci-002.
  • Arithmetic Reasoning: 2.7% improvement over Manual-CoT is achieved across five arithmetic reasoning tasks with text-davinci-002.Automate-CoT also improves SC by an average of 3.3% in the self-consistency setting.
  • Commonsense and Symbolic Reasoning: 2.5% and 3.4% improvements are achieved on commonsense and symbolic reasoning with text-davinci-002 and code-davinci-002, respectively.The results are reported as effective across different task types.
  • Non-Reasoning Tasks: 2.8%, 3.4%, and 1.3% improvements are achieved on OpenBookQA, e-SNLI, and SST-2, respectively.These results show generalization to non-reasoning tasks.

6 Additional Experiments and Analysis

Additional experiments evaluate selection, pool size, training-set robustness, chain complexity, and whether Automate-CoT can bypass manually written rationales. Across these settings, Automate-CoT generally outperforms comparison methods, while performance depends on pool size and exemplar composition.

  • Selection: Random exemplar selection has high variance and lower accuracy than Manual-CoT and Automate-CoT, although it can outperform Manual-CoT on GSM8K and CSQA.Automate-CoT consistently outperforms both random selection and Manual-CoT.
  • Pool Size: Automate-CoT matches or exceeds Manual-CoT once the exemplar pool reaches 20 items, whereas a pool of 10 performs worse or comparably.Performance trends upward as the pool grows.
  • Chain Complexity: Automate-CoT balances simple and complex questions by selecting exemplars with varied rationale complexity, outperforming Manual-CoT and Complex-CoT on GSM8K.Manual-CoT has 2–3-hop exemplars, Complex-CoT uses 9-hop exemplars, and Automate-CoT averages 4 hops across 2–6 hops.
  • Training-Set Robustness: Automate-CoT remains robust across different randomly chosen training sets, with only a small impact on results.The experiment uses three randomly chosen training sets across three datasets.
  • Zero-Shot-CoT: In a zero-shot setting, prompting with “Let’s think step by step.” still lets Automate-CoT achieve comparable or slightly better results than Manual-CoT and Auto-CoT.This setting is evaluated on GSM8K, SVAMP, and Letter (4) with text-davinci-002.

7 Ablation Study

The ablation study examines order, complexity, diversity, and style as factors affecting rationale selection. Automate-CoT reduces the need for manual ordering and constructs exemplar combinations that trade off complexity and preserve effective style.

  • Order Sensitivity: Automate-CoT’s selected exemplar order outperforms five random permutations, showing that it can choose a good order automatically.Order sensitivity remains, but the selected ordering performs better than every tested permutation.
  • Complexity: Automate-CoT automatically augments and selects exemplars with different complexity, improving the accuracy trade-off between simple and complex questions.Manually written chains are typically less than or equal to 3 hops.
  • Diversity: Automate-CoT selects an optimal combination of exemplar complexities for a dataset, unlike Manual-CoT or Complex-CoT, whose complexity ranges are more limited.On GSM8K, the selected exemplars average 5.4 hops and range from 3 to 8 hops.
  • Style: When original and manually degraded styles are paired while order, complexity, and diversity remain fixed, Automate-CoT selects the original exemplars, which outperform the alternatives by 2%.The experiment uses eight original and eight edited exemplars for GSM8K.

8 Related Work

The related work situates Automate-CoT within prompt-based learning, chain-of-thought prompting, and black-box optimization. These areas motivate automated prompt construction for models whose parameters and gradients are inaccessible.

  • Prompt-Based Learning: Prompt-based learning uses large language models to elicit useful knowledge for downstream tasks and includes discrete and continuous prompt types.The passage introduces prompting as a broad family of downstream-task methods.
  • Chain-of-Thought Prompting: Chain-of-thought prompting adds rationale steps to in-context exemplars and improves performance on arithmetic, commonsense, and symbolic reasoning tasks.Several subsequent methods modify or extend this prompting strategy.
  • Black-Box Optimization: Cloud-deployed language-model APIs create a black-box optimization setting because their parameters and gradients are inaccessible.This motivates optimization methods that do not directly use model gradients.

9 Conclusion

The paper presents Automate-CoT as a three-step chain-of-thought optimization method and reports effectiveness across reasoning and non-reasoning tasks.

  • Conclusion: Automate-CoT augments rationale chains, prunes incorrect chains using answer consistency, and selects better CoTs with variance-reduced policy gradients.The method is designed for black-box language-model prompting.
  • Conclusion: Experimental results demonstrate Automate-CoT’s effectiveness on arithmetic, commonsense, symbolic, and non-reasoning tasks.The conclusion reports effectiveness across all four task categories.

10 Limitations

The paper identifies limitations in its comparison scope and in its treatment of prompt style, while describing variance-reduced gradient estimation as part of the optimization procedure.

  • Fine-tuning large language models is not evaluated because the study focuses on prompting and has limited resources.
  • Automate-CoT does not provide a rigorous definition of good versus bad linguistic style.The authors identify defining prompt style as an important direction for further exploration.
  • The gradient estimator samples rationale-related quantities independently and updates prompt-token distributions with projected stochastic gradient descent.The update uses a learning rate, sample size, and projection calculation.

B.1 Datasets and Evaluation Metrics

The evaluation covers reasoning and non-reasoning tasks, compares Automate-CoT with several prompting and retrieval baselines, and uses GPT-3 engines for the experiments. The method constructs a pool of model-generated rationale exemplars from sampled questions and labels.

  • Datasets: The evaluation spans five math word-problem datasets, two commonsense datasets, and one symbolic reasoning task.The datasets are GSM8K, ASDiv, SVAMP, AQuA, SingleOp, CommonsenseQA, StrategyQA, and Last Letter Concatenation.
  • Datasets: Additional evaluation covers OpenBookQA, e-SNLI, and sentiment analysis on SST.These extend the method beyond reasoning tasks to question answering, natural language inference, and sentiment analysis.
  • Baselines: Manual-CoT, self-consistency, and Auto-CoT serve as the main baselines.Manual-CoT uses human-written intermediate steps; self-consistency samples diverse reasoning paths; Auto-CoT clusters questions before generating chains.
  • Models: Experiments use the text-davinci-002 and code-davinci-002 GPT-3 engines, with additional experiments mainly using code-davinci-002.The paper identifies code-davinci-002 as the most capable available model at experiment time and notes its lower cost relative to text-davinci-002.
  • Evaluation procedure: Automate-CoT constructs a pool of 100 model-generated rationale exemplars for all datasets in the main experiments.Questions are randomly sampled and fed to language models to generate answers with rationale chains; the paper keeps 4–8 exemplars in prompts.

C.1 Experiments under ChatGPT

Under gpt-3.5-turbo, Automate-CoT improves performance across reasoning categories and overall, outperforming retrieval-based PromptPG and generally exceeding clustering-based selection. It also shows low variance and uses fewer training examples and lower estimated cost than fine-tuning in the reported comparison.

  • Experiments under ChatGPT: 2.8% improvement on arithmetic reasoning, 3.9% on commonsense reasoning, 3.2% on symbolic reasoning, and 2.8% overall are reported for Automate-CoT.These results are obtained in additional experiments with gpt-3.5-turbo.
  • Retrieval-based comparison: BM25 retrieval selects semantically similar examples but overlooks diversity, and its average performance is reported as limited in the comparison.The paper compares BM25 with Automate-CoT across all datasets.
  • Retrieval-based comparison: Automate-CoT outperforms PromptPG under matched settings using 100 training examples, a 100-example candidate pool, and gpt-3.5-turbo.The prompt format is kept consistent with the original chain-of-thought and Automate-CoT settings.
  • Clustering-based comparison: Clustering-based selection generally outperforms Manual-CoT, but the paper states that complexity and diversity are overlooked.The additional comparison uses K-Means to create as many clusters as the specified number of exemplars and selects representative examples.
  • Variance analysis: Automate-CoT achieves quite low variance across three runs, especially compared with the large variance of Manual-CoT.The variance comparison is reported for results averaged over three runs, including self-consistency settings in the accompanying table.
  • Fine-tuning comparison: Automate-CoT requires 100 training examples, whereas fine-tuning gpt-3 is reported to require thousands, such as 8000, to be effective.The comparison also reports higher training and inference costs for fine-tuning.
  • Fine-tuning comparison: $57.6 for fine-tuning 8000 examples for three epochs is reported as 12x more costly than Automate-CoT.The estimate uses a gpt-3.5-turbo training price of $0.008 per 1K tokens and 300-token average training examples.
  • Additional analysis: Adding prompting tricks raises GSM8K accuracy to 69.8% (+2.2%) under normal prompting and 83.0% (+0.6%) under self-consistency.The tricks include adding “Let’s think step by step.” before rationale chains and replacing “Q:” with “Question:”.
Loading 2302.12822v3…