PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Prompting with Chain-of-Thought for Large Language Models

Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, Tong Zhang

arXiv:2302.12246v5cs.CL

TL;DR

Fixed human-selected CoT exemplars may not suit different reasoning tasks. Active-Prompt selects questions with high model uncertainty for human annotation and uses them as task-specific exemplars, improving results across eight reasoning datasets. The paper also finds that disagreement, entropy, and variance work similarly well, while self-confidence does not.

  • Problem

    Fixed human-annotated CoT exemplars are not necessarily effective across tasks with differing reasoning difficulty, scope, and domain.

  • Method

    Active-Prompt queries the model repeatedly, measures answer uncertainty, selects the most uncertain questions for human CoT annotation, and prepends the resulting exemplars during inference.

  • Results

    Active-Prompt achieves promising performance across eight arithmetic, commonsense, and symbolic reasoning datasets and surpasses competitive baselines by a large margin.

  • Takeaways & Limitations

    Disagreement, entropy, and variance provide similarly effective uncertainty estimates, while self-confidence is ineffective because LLMs are prone to over-confidence.

  • Takeaways & Limitations

    Experiments were limited by the cost of newer models, including GPT-4 and self-consistency with gpt-3.5-turbo.

Abstract

from arXiv · show

The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning. It is known that the effective design of task-specific prompts is critical for LLMs' ability to produce high-quality answers. In particular, an effective approach for complex question-and-answer tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs. However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks. This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning). For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful ones to annotate from a pool of task-specific queries. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation. Experimental results demonstrate the superiority of our proposed method, achieving state-of-the-art on eight complex reasoning tasks. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationship demonstrate the effectiveness of our method. Our code will be available at https://github.com/shizhediao/active-prompt.

1 Introduction

Active-Prompt addresses the task-specific weakness of fixed human-selected CoT exemplars by selecting uncertain questions for annotation. It uses uncertainty-based selection to construct exemplars and improves reasoning performance across complex tasks.

  • Motivation: CoT prompting improves complex reasoning but depends on humans selecting and annotating informative exemplars.Fixed exemplars may not be effective across tasks with different difficulty, scope, and domain.
  • Approach: Active-Prompt estimates uncertainty from multiple model answers and selects the most uncertain questions for human CoT annotation.The selected annotations become task-specific exemplars for inference.
  • Uncertainty metrics: Disagreement, entropy, and variance perform similarly well, whereas self-confidence fails because LLMs tend to be over-confident.
  • Results: Across eight reasoning datasets, Active-Prompt surpasses competitive baselines by a large margin.The experiments cover arithmetic, commonsense, and symbolic reasoning.

2 Active-Prompt

Active-Prompt constructs few-shot exemplars by estimating uncertainty over repeated answers, annotating the most uncertain questions, and using the resulting rationales during inference.

  • 2 Active-Prompt: Active-Prompt selects n questions from unlabeled training data, annotates their reasoning steps and answers, and uses them as exemplars for test prediction.
  • 2.1 Uncertainty Estimation: The method queries the LLM k times per question and measures uncertainty from the resulting answers.It considers disagreement, entropy, variance, and self-confidence.
  • 2.1 Uncertainty Estimation: Disagreement is the fraction of unique answers among k predictions, computed as u = h/k.
  • 2.1 Uncertainty Estimation: Higher entropy indicates greater uncertainty, so questions with larger entropy are selected as candidates.
  • 2.1 Uncertainty Estimation: Variance is considered as an uncertainty metric that may be especially suitable for arithmetic answers.Predictions are normalized by numbers mentioned in the question to reduce domination by large values.
  • 2.1 Uncertainty Estimation: Disagreement, entropy, and variance perform competitively well and significantly outperform self-confidence; the first two are used mainly because they are simple.
  • 2.2 Selection and Annotation: The method ranks questions by uncertainty, selects the top n, and has humans annotate rationale chains and answers to form new exemplars.If more than n questions tie at the largest uncertainty, n are randomly chosen among them.
  • 2.3 Inference: During inference, the new exemplars prompt each question, while self-consistency may generate m answers at temperature T and choose the most consistent one.

3 Experimental Settings

The experiments evaluate Active-Prompt on eight reasoning datasets with standard accuracy-based evaluation and several prompting baselines. Implementation choices include transferred exemplars for datasets without training splits and uncertainty estimation over candidate pools.

  • Datasets and Metrics: Experiments cover arithmetic, commonsense, and symbolic reasoning datasets, including an out-of-distribution four-letter setting for last letter concatenation.
  • Datasets and Metrics: Exact match accuracy is reported as the evaluation metric.
  • Baselines: Baselines include CoT, self-consistency, Auto-CoT, and Random-CoT, with Random-CoT differing only in random question sampling.
  • Baselines: The experiments primarily use code-davinci-002 and additionally test text-davinci-002, text-davinci-003, and gpt-3.5-turbo.
  • Implementation: The number of exemplars varies by dataset, and GSM8K annotations are transferred to ASDiv, SVAMP, and SingleEq because they have only test splits.
  • Implementation: Uncertainty estimation uses up to 1,000 candidate instances and k = 10 model answers per dataset in the main experiments.The authors report robust performance with a pool size of 1,000.
  • Implementation: Human annotation writes reasoning steps and answers for selected questions, with the annotator instructed to minimize trial-and-error engineering.

4 Experimental Results

Active-Prompt outperforms baseline methods across the evaluated reasoning tasks, with especially clear gains over self-consistency. Arithmetic results are stronger when task-specific training data supports direct selection, while some datasets require cross-task CoT transfer.

  • Overall Results: Active-Prompt outperforms all baseline models by a large margin overall.Table 1 reports the overall performance comparison.
  • Overall Results: Active-Prompt (D) improves over self-consistency by an average of 7.0% with text-davinci-002 and 1.8% with code-davinci-002.
  • Arithmetic Reasoning: In arithmetic reasoning, Active-Prompt (D) exceeds self-consistency by averages of 2.1% with code-davinci-002 and 7.2% with text-davinci-002.
  • Arithmetic Reasoning: With code-davinci-002, the largest arithmetic gains occur on GSM8K at 4.2% and AQuA at 3.1%.
  • Arithmetic Reasoning: ASDiv, SVAMP, and SingleEq lack training data, requiring their annotated CoT to be transferred from GSM8K.The paper identifies improved cross-task prompt transfer as future research.
  • Commonsense and Symbolic Reasoning: Active-Prompt consistently improves commonsense and symbolic reasoning and remains best in the more challenging out-of-distribution Letter (4) setting.

5 Analysis

The analysis evaluates Active-Prompt through ablations of prompting, selection, annotation, uncertainty estimation, pool size, and model transfer. Results show gains from active selection, robustness to exemplar and annotator choices, and usefulness across model settings.

  • Effects of Few-Shot Prompts: Zero-Shot-Active-Prompt performs competitively with Active-Prompt, demonstrating that the method is not necessarily dependent on few-shot exemplars.
  • Effects of Active Selection: Active-Prompt outperforms Random-CoT, showing that uncertainty-based question selection—not merely annotation—drives performance gains.On GSM8K, Active-Prompt reaches 82.2 versus 78.6 for Random-CoT and 78.0 for self-consistency.
  • Effects of Annotators: Annotators A and B both outperform baseline models, while existing GSM8K rationales outperform the paper’s annotations, indicating robust selection but variable annotation quality.
  • Effects of Pool Size: With text-davinci-003, performance increases as the prediction pool grows from 1 to 15 answers and converges at k = 10.
  • Uncertainty Analysis: Uncertainty is highly negatively correlated with accuracy: as uncertainty decreases, accuracy increases on GSM8K, ASDiv, and SingleEq.
  • Model Transfer: Active-Prompt outperforms CoT by a large margin on instruction-tuned Llama2-70b-chat, extending its usefulness beyond GPT-series models.

6 Related Work

Prior work shows that chain-of-thought prompting improves reasoning by adding explicit reasoning steps to few-shot examples. Related research extends prompting through consistency, decomposition, bootstrapping, verification, selection, and metaheuristic methods, while active learning motivates efficient annotation.

  • Chain-of-Thought Prompting: Chain-of-thought prompting enriches few-shot examples with reasoning steps to improve performance on complex tasks.
  • Chain-of-Thought Prompting: Subsequent studies improve standard chain-of-thought prompting through consistency, decomposition, bootstrapping, self-training, verification, augmentation, selection, and metaheuristics.
  • Active Learning: Active learning seeks helpful unlabeled data to annotate under reasonable budgets and has been applied to fine-tuning large language models for classification.

7 Conclusion

Active-Prompt elicits LLM reasoning by selecting task-specific questions for human annotation through uncertainty-based active selection. Across eight reasoning datasets, it achieves promising performance, with further analyses supporting the effectiveness of its components.

  • Active-Prompt selects the most important and helpful task-specific questions for human annotation instead of choosing exemplars arbitrarily.
  • The method estimates question uncertainty using disagreement, entropy, variance, and self-confidence, primarily applying disagreement and entropy.
  • Active-Prompt achieves promising performance across eight datasets covering arithmetic, commonsense, and symbolic reasoning.
  • Analyses of uncertainty metrics, annotators, pool sizes, zero-shot learning, and accuracy-uncertainty relationships demonstrate the method’s effectiveness.

Limitations

The paper identifies limited model coverage and reproducibility constraints as limitations of its experiments. Most experiments use code-davinci-002, whose discontinued access makes replication difficult.

  • Experiments provide complete results for code-davinci-002 and text-davinci-002, but only one of these costly models could be studied extensively.The authors identify experiments with more models, including GPT-4 and self-consistency with gpt-3.5-turbo, as future work.
  • Most experiments use code-davinci-002 because it was free during its initial limited beta period.
  • Access to code-davinci-002 was shut off, making the experiments difficult for researchers to reproduce.The paper notes that researcher access may still provide a route to the model.
  • The experiments cover arithmetic, commonsense, and symbolic reasoning datasets, including an out-of-distribution last-letter concatenation setting.
  • The main baselines are Chain-of-thought, Self-consistency, Auto-CoT, and Random-CoT, with Random-CoT differing from Active-Prompt only in random question selection.

A.3 Implementation

The implementation fixes task-specific exemplar counts and uses manually annotated exemplars to stabilize uncertainty estimation. The evaluation draws on dataset statistics, annotation procedures, and Figure 3’s accuracy-uncertainty analysis.

  • Evaluation uses task-specific exemplar counts: 8 for four arithmetic datasets, 7 for CSQA, 6 for StrategyQA, and 4 for AQuA and Letter.
  • Uncertainty estimation begins with manually annotated exemplars taken from Wei et al. (2022b) to stabilize predictions.The method can also use exemplar-free zero-shot prompting.
  • Table 6 reports dataset training and test sizes, exemplar counts, transferred exemplars, and the use of 1,000 sampled training examples to reduce computational cost.
  • The annotation procedure has one co-author write reasoning steps and answers for selected questions, following previous annotation practices.
  • Figure 3 presents the relation between accuracy and uncertainty.

C Variance Analysis

The uncertainty-estimation experiment addresses computational cost by sampling training instances and repeating that sampling across seeds. The reported results indicate robustness to this sampling randomness.

  • For large datasets such as GSM8K, the method randomly samples 1,000 training instances to estimate uncertainty and conserve resources.The full GSM8K training set contains 7,473 instances.
  • Three repetitions of random sampling examine variance arising from the sampling process.
  • Sampling 1,000 instances produces robust results across random samples and is sufficient for stable, satisfactory performance.
  • Table 7 reports experimental results on GSM8K with three seeds.
  • Self-confidence estimates uncertainty by asking the model to classify confidence in its predicted answer among four categorical choices.

E Logits-based Uncertainty Estimation

Logit-based uncertainty estimation can support Active-Prompt, which outperforms traditional CoT and is slightly better than disagreement-based selection. However, Llama-2-70b logits exhibit overconfidence that harms uncertainty estimation.

  • The logits-based uncertainty comparison is presented in Table 9.
  • Active-Prompt using gpt-3.5-turbo-0301 logits outperforms traditional Chain of Thought and is slightly better than the Disagreement-based method.The comparison is reported in Table 9.
  • Llama-2-70b logits show overconfidence, leading to poorer results when used as an uncertainty measure.The authors identify calibration methods for logits as future work.

F Comparison with Diversity-based Methods

Active-Prompt is compared with diversity- and complexity-based exemplar selection, while analyses examine efficiency, annotation cost, and whether selection or longer reasoning drives gains. The method outperforms Complex-CoT, and longer CoT alone does not improve performance.

  • F Comparison with Diversity-based Methods: Auto-CoT uses diversity-based question selection, whereas Active-Prompt uses uncertainty-based selection.The comparison is conducted on GSM8K, MultiArith, and AddSub under Auto-CoT’s reported setting.
  • F Comparison with Diversity-based Methods: Uncertainty and diversity are both useful for selecting informative questions and are described as complementary.Combining them is identified as an important future direction.
  • G Comparison with Complexity-based Methods: Active-Prompt outperforms Complex-CoT, while combining uncertainty and complexity may achieve better performance.The combination is left for future work.
  • Efficiency and Annotation Cost: Selecting questions algorithmically is more efficient than repeated human trial and error, while a 1,000-instance candidate pool balances cost and performance.Performance converges with pools larger than 1,000 instances, and ten uncertainty-estimation passes are described as acceptable.
  • Efficiency and Annotation Cost: Human annotation remains costly, motivating future investigation of zero-shot-CoT and other low-cost annotation methods.
  • I Ablation Study of Longer CoT Annotations: Merely extending average CoT length to 155 words does not improve performance and sometimes reduces it, whereas Active-Prompt remains superior.The corresponding Active-Prompt annotations average 160 words, supporting selection rather than length as the key factor in this ablation.
Loading 2302.12246v5…