PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data

KaShun Shum, Shizhe Diao, Tong Zhang

arXiv:2302.12822v3cs.CL

TL;DR

CoT prompting often depends on fixed human-written exemplars that take substantial effort to create and adapt. Automate-CoT generates, filters, and selects rationale chains, and with text-davinci-002 outperforms Manual-CoT by 2.6% and SC by 3.7% on average.

  • Problem

    Many CoT methods rely on fixed human-written exemplars that require substantial effort to create and adapt to new datasets.

    Source: §1 · p. 1

  • Method

    Automate-CoT generates rationale chains, prunes chains whose answers disagree with ground truth, and uses variance-reduced policy gradients to select helpful chains.

    Source: §1 · p. 2, §9 · p. 9

  • Results

    2.6% over Manual-CoT and 3.7% over SC on average, Automate-CoT achieves superior results across tasks with text-davinci-002.

    Source: §5 · p. 5

  • Takeaways & Limitations

    Automate-CoT offers a way to adapt CoT prompting to different tasks without human effort.

    Source: §1 · p. 2

    The paper does not provide a rigorous definition of good versus bad linguistic style.

    Source: §10 · p. 9

Abstract

from arXiv · show

Chain-of-thought (CoT) advances the reasoning abilities of large language models (LLMs) and achieves superior performance in complex reasoning tasks. However, most CoT studies rely on carefully designed human-annotated rational chains to prompt LLMs, posing challenges for real-world applications where labeled data is available without rational chains. This paper proposes a new strategy, Automate-CoT (Automatic Prompt Augmentation and Selection with Chain-of-Thought), that can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and then pruning low-quality chains to construct a candidate pool of machine-generated rationale chains based on the labels. Finally, it selects the optimal combination of several rationale chains from the pool for CoT prompting by employing a variance-reduced policy gradient strategy to estimate the significance of each example. Automate-CoT enables a quick adaptation of the CoT technique to different tasks. Experimental results demonstrate the effectiveness of our method, where competitive results are achieved on arithmetic reasoning (+2.7%), commonsense reasoning (+3.4%), symbolic reasoning (+3.2%), and non-reasoning tasks (+2.5%). The code is available at https://github.com/SHUMKASHUN/Automate-CoT.

1 Introduction

Automate-CoT addresses the effort and sensitivity of manually designing chain-of-thought exemplars by automatically generating, pruning, and selecting rationale chains. The introduction reports gains across reasoning and non-reasoning tasks.

1 Introduction

  • Manually written CoT exemplars require substantial effort because performance is sensitive to exemplar order, reasoning complexity, diversity, and writing style.These sensitivities motivate an automatic, task-agnostic way to adapt exemplars to downstream tasks.Source: §1 · p. 1
  • Automate-CoT generates multiple pseudo-chains, prunes them using agreement between generated and ground-truth answers, then selects helpful chains for prompting.Pruning rests on the assumption that correct reasoning is necessary for a correct answer, so answer consistency can reduce noisy pseudo-chains.Source: §1 · pp. 1–2
  • +2.7% on arithmetic reasoning, +3.4% on commonsense reasoning, +3.2% on symbolic reasoning, and +2.5% on non-reasoning tasks are reported for Automate-CoT.The paper presents these results as evidence of the method’s effectiveness.Source: §1 · p. 2
  • A variance-reduced policy gradient strategy estimates gradients to optimize chain selection for each task.The selection step operates on data annotated with rationale paths.Source: §1 · p. 2
  • Unlike Auto-CoT (Zhang et al., 2023), which clusters diverse questions to generate rationale chains, Automate-CoT targets exemplar sensitivities and reports larger performance boosts.The introduction describes Automate-CoT as a fully automatic pipeline for finding prompts adaptable to different tasks.Source: §1 · p. 2

2 Motivation

The paper motivates Automate-CoT by identifying order, complexity, diversity, and style sensitivities in chain-of-thought prompting, then proposes augmenting and adaptively selecting rationale chains to address them without human trial and error.

Complexity

  • Human-written CoT exemplars tend to use ≤3 hops and struggle with complex questions, while 9-hop Complex-CoT helps complex questions but performs poorly on simple ones.The authors argue exemplar complexity should match the reasoning steps required by the question.Source: Figure 1 · p. 2, §2 · p. 2

Diversity

  • A combination of exemplars with different complexity levels outperforms using only complex exemplars, indicating a complexity-diversity trade-off.The authors note that testing combinations to find an effective mix would require substantial human effort.Source: Figure 1 · p. 2, §2 · p. 2

Order Sensitivity

  • On GSM8K, the lowest shuffled Manual-CoT accuracy was 59.8%, 3.3% below the reported 63.1% average, showing order sensitivity persists.The authors randomly shuffled exemplar order 10 times.Source: §2 · p. 2

Style Sensitivity

  • Different annotators produced up to a 28.2% accuracy difference on a symbolic reasoning task, and identifying helpful prompting styles requires validation-set trial and error.The authors therefore aim to augment rationale chains and select helpful ones adaptively, reducing order and style sensitivities while improving the complexity-diversity trade-off.Source: §2 · p. 3

3 Approach

Automate-CoT builds a pool of label-checked, machine-generated rationale chains and selects a subset for few-shot prompting with policy-gradient optimization.

3 Approach

  • Automate-CoT takes a training dataset of n questions and their corresponding n answers as input.The paper presents the overall architecture in Figure 2.Source: §3 · p. 3

3.1 Augment and Prune

  • For each question, the LLM generates k rationale chains (k=1 in experiments, including zero-shot m=0) and retains only chains whose final answers match the ground truth.The retained chains form a pool of K high-quality exemplars.Source: §3.1 · pp. 3–4

3.2 Select

  • To select suitable 4–8 exemplars under context-length, fairness, sensitivity, and task-adaptation constraints, Automate-CoT models each exemplar index as a categorical latent variable.It optimizes the exemplar distributions with a variance-reduced policy-gradient estimator using forward propagation, followed by projected stochastic gradient descent.Source: §3.2 · p. 4

4 Experimental Settings

The experiments evaluate Automate-CoT across reasoning and non-reasoning tasks, compare it with established prompting baselines, and specify the data and optimization settings used.

4.1 Datasets and Evaluation Metrics

  • The evaluation spans eight reasoning and three non-reasoning datasets, using exact-match accuracy after special-symbol removal from predictions.Reasoning tasks include five math datasets, two commonsense datasets, and Letter (4); non-reasoning tasks include OpenBookQA, e-SNLI, and SST-2.Source: §4.1 · p. 4

4.2 Baselines

4.3 Implementation

  • Automate-CoT augments and prunes a 100-exemplar pool per dataset, using 4–8 exemplars, then selects combinations with 100-example training and validation sets.Selection optimizes latent variables with AdamW for five epochs (learning rate 1 × 10−3, batch size 10) and chooses the combination with highest validation accuracy. Inference uses one query by default, or 40 self-consistency queries; max_tokens is 256, logprobs is 5 for training, and evaluation temperature is 0.7 for self-consistency and 0 otherwise.Source: §4.3 · pp. 4–5

5 Experimental Results

Automate-CoT outperforms Manual-CoT and self-consistency across the reported task categories, with improvements spanning arithmetic, commonsense and symbolic reasoning, and non-reasoning tasks.

Overall

  • With text-davinci-002, Automate-CoT beats Manual-CoT and SC by 2.6% and 3.7% on average; with code-davinci-002, it beats them by 2.7% and 2.2%, respectively.Automate-CoT results are averaged over three runs.Source: §5 · p. 5

Arithmetic Reasoning

  • On arithmetic reasoning, Automate-CoT improves over Manual-CoT by 2.7% with text-davinci-002 and by 2.4% across five tasks with code-davinci-002.It also improves over SC by an average of 3.3% with text-davinci-002, outperforms Auto-CoT on GSM8K, SVAMP, and AQuA with text-davinci-002, and exceeds Auto-CoT on code-davinci-002 GSM8K by 4.8%.Source: §5 · p. 5

Commonsense and Symbolic Reasoning

  • On commonsense and symbolic tasks, Automate-CoT improves by 2.5% with text-davinci-002 and 3.4% with code-davinci-002, with a significant improvement on out-of-distribution Letter (4).Source: §5 · pp. 5–6

Non-Reasoning Tasks

  • On non-reasoning tasks, Automate-CoT improves by 2.8% on OpenBookQA, 3.4% on e-SNLI, and 1.3% on SST-2.The paper reports these results as evidence that the method generalizes beyond reasoning tasks.Source: §5 · p. 6

6 Additional Experiments and Analysis

Additional experiments find that Automate-CoT benefits from deliberate exemplar selection and larger pools, balances rationale complexity, and is robust to training-example choice. The method can also operate without manually written initial chains.

6.1 Effects of Selection Algorithm

  • Automate-CoT consistently outperformed random selection and Manual-CoT, while random selection had high variance and lower accuracy despite exceeding Manual-CoT on GSM8K and CSQA on average.Randomly selected exemplars were combined regardless of order; the average advantage over Manual-CoT appeared only on some datasets.Source: §6.1 · p. 6

6.2 Effects of Pool Size

  • With a pool of 10 exemplars, Automate-CoT underperformed or matched Manual-CoT, but outperformed it at 20 or more, with performance trending upward as pools grew.Experiments were limited to a maximum pool size of 150 because additional GPT-3 queries were time-consuming and expensive.Source: §6.2 · p. 6, Figure 4 · p. 6

6.3 Effects of Chain Complexity

  • On GSM8K, Manual-CoT scored 62% overall and 9-hop Complex-CoT scored 60%, while Automate-CoT with 2–6-hop chains averaging 4 hops outperformed both.Manual-CoT did well on simple questions but struggled on 7- and 8-hop questions; Complex-CoT improved on complex questions but performed poorly on simple ones. Automate-CoT combined exemplars of different complexity levels.Source: §6.3 · p. 7

6.4 Effects of Training Example Selection

  • Across three randomly chosen training sets, Automate-CoT was robust, with training-example selection having little impact on results.Table 2 reports performance over three datasets.Source: §6.4 · p. 7, Table 2 · p. 7

6.5 Bypass Manual Effort by Zero-shot-CoT

  • Replacing 4–8 manually written exemplars with “Let’s think step by step.” yielded zero-shot Automate-CoT results comparable to or slightly better than Manual-CoT and Auto-CoT on GSM8K, SVAMP, and Letter (4).The method could select suitable exemplar combinations even from a pool that might contain low-quality chains.Source: §6.5 · p. 7, Table 3 · p. 7

7 Ablation Study

The ablations report that Automate-CoT can choose exemplar orders without human tuning, accommodate varied rationale complexity and diversity, and select examples with better linguistic style. These findings support its ability to construct prompts that address four factors affecting prompt quality.

Advantage over Order Factor

  • Automate-CoT’s selected exemplar order outperformed all five random permutation runs, showing it can choose a good order without human effort.The comparison tests random permutations of the selected exemplars against Automate-CoT’s selected order.Source: §7 · pp. 7–8, Table 4 · p. 8

Advantage over Complexity Factor

  • Automate-CoT can select exemplars with different complexity to balance accuracy on simple and complex questions, unlike manually written chains limited to three hops.The cited complexity comparison is reported in Appendix Table 9.Source: §7 · p. 8

Advantage over Diversity Factor

  • On GSM8K, Automate-CoT’s selected exemplars averaged 5.4 hops and ranged from 3 to 8, combining simple and complex examples.The paper contrasts this variety with Complexity-CoT exemplars sharing one complexity and Manual-CoT exemplars ranging from 1–3 hops.Source: §7 · p. 8

Advantage over Style Factor

  • Automate-CoT selected the original, better-style exemplars over manually degraded alternatives, whose performance was 2% lower.For GSM8K, the experiment held order, complexity, and diversity constant and constrained selection to one version of each exemplar while preserving order.Source: §7 · p. 8

8 Related Work

The related work reviews prompt-based learning, chain-of-thought prompting, and black-box optimization. It frames Automate-CoT against methods that avoid fine-tuning or gradients but face limits in reasoning or task scope.

8.1 Prompt-based Learning

  • Prompting methods use discrete tokens or continuous vectors to elicit LLM knowledge while avoiding model fine-tuning and optimizing only a small parameter set.Discrete prompts optimize token sequences, whereas continuous prompts optimize vector sequences.Source: §8.1 · p. 8

8.2 Chain-of-thought Prompting

8.3 Black-box Optimization

  • Black-box prompt learning avoids gradient access, but prior methods had limited reasoning ability and were restricted to zero-shot classification (Diao et al., 2022; Sun et al., 2022; Prasad et al., 2022).Cloud APIs such as OpenAI’s GPT-3 (Brown et al., 2020) conceal model parameters and gradients, complicating optimization; earlier black-box optimization largely studied score-based adversarial attacks.Source: §8.3 · p. 9

9 Conclusion

Automate-CoT optimizes chain-of-thought prompts through augmentation, pruning, and selection, and the paper reports effectiveness across reasoning and non-reasoning tasks.

9 Conclusion

  • Automate-CoT demonstrates effectiveness on arithmetic, commonsense, and symbolic reasoning tasks, as well as non-reasoning tasks.Source: §9 · p. 9
  • The method generates rationale chains, prunes incorrect ones by comparing predicted and ground-truth answers, and selects better chains.Source: §9 · p. 9
  • A variance-reduced policy gradient strategy estimates gradients to optimize latent variables for selecting better chain-of-thought prompts.Source: §9 · p. 9

10 Limitations · A Algorithm Details · B Detailed Experimental Setting

The limitations concern an omitted comparison with fine-tuning and the lack of a rigorous definition of good linguistic style. The appendices detail the variance-reduced policy-gradient procedure and dataset evaluation settings.

10 Limitations

  • Automate-CoT is reported to outperform previous chain-of-thought prompting methods, but training large language models on labeled data was outside this study’s prompt-based scope and resource limits.Source: §10 · p. 9
  • The paper does not rigorously define good versus bad linguistic style, although it reports that Automate-CoT mitigates style sensitivity in Manual-CoT.The authors identify defining linguistic style as an area for future exploration.Source: §10 · p. 9

A Algorithm Details

  • The appendix derives the prompt-distribution gradient by estimating the gradient of pi and explicitly solving components of ∇pi log P(ti).Source: A Algorithm Details · p. 15
  • To mitigate PGE’s high variance, the method uses a variance-reduced policy-gradient estimator and independently samples T^(k) from P(T).The appendix then updates the prompt-token distribution pi with projected stochastic gradient descent, using learning rate η, sample size I, and projection projC.Source: A Algorithm Details · pp. 15–16

B Detailed Experimental Setting

  • Table 5 summarizes dataset statistics, including few-shot exemplar counts, evaluation counts and splits, and whether exemplars were transferred across datasets.Its notes specify validation-set evaluation for CSQA, StrategyQA, and SST-2, and evaluation on the first 1,000 data points for a fair comparison.Source: Table 5 · p. 16

B.1 Datasets and Evaluation Metrics … C More Experiment Results

The experiments evaluate Automate-CoT across reasoning and non-reasoning tasks against established prompting baselines, using labeled data to build and select rationale exemplars. The supplied passages describe dataset scope and experimental procedures but provide no results from section C.

B.1 Datasets and Evaluation Metrics

  • Evaluation covers eight reasoning datasets and three non-reasoning datasets, with exact-match accuracy after special-symbol normalization.Reasoning tasks include five math word-problem datasets, CSQA, StrategyQA, and Letter (4); the non-reasoning tasks are OpenBookQA, e-SNLI, and SST-2.Source: §B.1 · p. 16
  • The evaluation uses test splits by default, validation splits when test labels are unavailable, and only the out-of-distribution Letter (4) setting.Letter (4) prompts use 2-letter inputs and test on 4-letter examples; the in-distribution setting is omitted because accuracy is nearly 100%.Source: §B.1 · p. 16

B.2 Baselines

  • The main baselines are Manual-CoT with human-written reasoning, self-consistency with sampled reasoning paths, and Auto-CoT with clustering-based exemplar construction.The study follows prior work in matching exemplar counts and evaluation-split settings.Source: §B.2 · pp. 16–17, §B.1 · p. 16
  • Main experiments use text-davinci-002 and code-davinci-002, while additional experiments mainly use code-davinci-002.The paper describes code-davinci-002 as the most capable available model at the time and free during the limited beta period, unlike costly text-davinci-002.Source: §B.2 · p. 17

B.3 Implementation

  • A pool of 100 model-generated rationale exemplars is constructed and label-pruned to retain correct examples.Pool questions are sampled from task-specific sources, with GSM8K pool results transferred to datasets that have only a test split.Source: §B.3 · p. 17
  • Exemplar selection optimizes latent variables using answer-token cross-entropy, then evaluates the combination with the highest validation accuracy.Training and validation samples each have size 100; Letter (4) instead uses samples from the 3-letter out-of-distribution set.Source: §B.3 · p. 17
  • Few-shot runs use max_tokens = 256, logprobs = 5 for training, temperature = 0.7 for self-consistency, and temperature = 0 otherwise.Inference queries the model once by default and 40 times under self-consistency; selection uses AdamW for 5 epochs at learning rate 1 × 10^-3 and batch size 10.Source: §B.3 · p. 17

C.1 Experiments under ChatGPT … G Full Exemplars generated by Automate-CoT

Under gpt-3.5-turbo, Automate-CoT improves performance across reasoning tasks; the merged sections also compare selection methods, report variance and costs, and present supplemental analyses and exemplar chains.

C.1 Experiments under ChatGPT

  • +2.8% overall, Automate-CoT improves gpt-3.5-turbo results by 2.8% on arithmetic, 3.9% on commonsense, and 3.2% on symbolic reasoning.The overall comparison with retrieval- and clustering-based exemplar selection methods is shown in Table 6.Source: §C.1 · pp. 17–18, Table 6 · p. 18

C.2 Comparison with Retrieval Methods

  • BM25 (Robertson, 2009) averaged 1% below Manual-CoT and 3.8% below Automate-CoT, which also outperformed PromptPG (Lu et al., 2023).PromptPG was evaluated with 100 training examples, a candidate pool of 100, and gpt-3.5-turbo.Source: §C.2 · p. 18, Table 6 · p. 18

C.3 Comparison with Clustering Methods

  • Clustering-based exemplars generally outperform Manual-CoT but trail Automate-CoT by 2.6%, with GSM8K exemplars often missing more complex questions.The selected GSM8K exemplars were mostly around 3–4 hops, overlooking complex and moderately difficult questions as well as complexity and diversity.Source: §C.3 · p. 18, Table 6 · p. 18

C.4 Variance Report

  • Automate-CoT has low result variance across three runs, especially compared with the large variance reported for Manual-CoT.Table 7 reports variance for Table 1 results, including the self-consistency setting.Source: §C.4 · p. 18, Table 7 · p. 18

D Additional Comparison with Fine-tuning

E Additional Analysis; E.1 Effects of Several Tricks

  • In additional GSM8K analysis, adding “Let’s think step by step” and replacing “Q:” with “Question:” raised accuracy to 69.8% (+2.2%) and 83.0% (+0.6%).The two figures correspond to normal and self-consistency settings, respectively.Source: E Additional Analysis · p. 19, §E.1 · p. 19

F Exact Match Number over Each Hop

G Full Exemplars generated by Automate-CoT

  • The appendix provides selected model-generated exemplar sets with rationale chains for arithmetic, CommonsenseQA, StrategyQA, Letter transfer, e-SNLI, and SST-2.These are example sets trained and selected on the named datasets, except the Letter set, which is trained on Letter (3) and selected on Letter (2).Source: Table 10 · p. 21, Table 12 · p. 23, Table 13 · p. 24

Related papers

Loading 2302.12822v3…