PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Thought Reasoning Without Prompting

Xuezhi Wang, Denny Zhou

arXiv:2402.10200v2cs.CL

TL;DR

Prior work typically elicits LLM reasoning through manually intensive prompting, leaving intrinsic reasoning difficult to assess. This paper instead explores alternative top-k decoding paths and uses confidence associated with CoT paths to select outputs. Across reasoning benchmarks, CoT-decoding reveals reasoning paths and improves over greedy decoding, while adding computational cost and facing reduced precision for some open-ended answers.

  • Problem

    Prompting can encode task-specific human priors, making it difficult to assess LLMs’ intrinsic reasoning abilities without human prompt refinement.

  • Method

    CoT-decoding explores alternative top-k tokens during decoding and selects CoT paths using the model’s final-answer confidence.

  • Results

    CoT-decoding elicits CoT reasoning and demonstrates significant improvements over greedy decoding across various reasoning benchmarks.

  • Takeaways & Limitations

    Changing decoding rather than prompting reveals reasoning paths that standard greedy decoding can obscure and supports a more direct assessment of intrinsic reasoning.

  • Takeaways & Limitations

    Alternative decoding paths incur additional computational costs, and top-two-token probability differences may be less precise for more open-ended answers.

Abstract

from arXiv · show

In enhancing the reasoning capabilities of large language models (LLMs), prior research primarily focuses on specific prompting techniques such as few-shot or zero-shot chain-of-thought (CoT) prompting. These methods, while effective, often involve manually intensive prompt engineering. Our study takes a novel approach by asking: Can LLMs reason effectively without prompting? Our findings reveal that, intriguingly, CoT reasoning paths can be elicited from pre-trained LLMs by simply altering the \textit{decoding} process. Rather than conventional greedy decoding, we investigate the top-$k$ alternative tokens, uncovering that CoT paths are frequently inherent in these sequences. This approach not only bypasses the confounders of prompting but also allows us to assess the LLMs' \textit{intrinsic} reasoning abilities. Moreover, we observe that the presence of a CoT in the decoding path correlates with a higher confidence in the model's decoded answer. This confidence metric effectively differentiates between CoT and non-CoT paths. Extensive empirical studies on various reasoning benchmarks show that the proposed CoT-decoding effectively elicits reasoning capabilities from language models, which were previously obscured by standard greedy decoding.

1. Introduction

The paper asks whether LLMs can reason without prompting and finds that changing decoding can reveal CoT paths and support more reliable answers. CoT-decoding explores alternative top-k paths and uses higher answer confidence associated with CoT paths to select outputs.

  • Motivation: Prompting can encode task-specific human priors, complicating assessment of LLMs’ intrinsic reasoning abilities.The paper therefore asks whether LLMs can reason effectively without prompting.
  • CoT-decoding: Alternative top-k decoding paths can naturally contain CoT reasoning, whereas greedy paths may miss it.The approach considers alternative tokens instead of relying solely on the top-1 greedy path.
  • CoT-decoding: CoT-decoding sifts through top-k decoding paths to isolate more reliable paths for model output.The method is based on the observation that CoT paths can emerge within decoding trajectories.
  • Contribution: The method eliminates human-provided prompts, enabling assessment of intrinsic reasoning without prompting confounders.The paper contrasts decoding changes with prior work centered on refining prompts.
  • Confidence signal: Higher final-answer confidence is associated with the presence of a CoT reasoning path.The paper illustrates this contrast with paths yielding the correct answer “8” and incorrect answer “5”.

2. Chain-of-Thought (CoT) Decoding

The paper elicits chain-of-thought reasoning from pre-trained language models by exploring alternative top-k decoding paths instead of relying only on greedy decoding. It selects promising CoT paths using answer-confidence differences, improving reasoning-path extraction and performance across tasks.

  • Pre-trained Language Models Can Reason without Prompting: The method compares greedy decoding at k=0 with paths beginning from the k-th token, where k>0 denotes an alternative first-step choice.The comparison is illustrated on math and commonsense reasoning tasks using PaLM-2 Large.
  • Pre-trained Language Models Can Reason without Prompting: Alternative top-k paths can naturally contain CoT reasoning, whereas greedy paths often produce direct answers without intermediate reasoning.Examples include a valid GSM8K CoT at k=9 and year-parity CoTs at k=3 and k=7.
  • CoT-Decoding for Extracting CoT Paths: Confidence-based extraction is preferred because CoT paths do not consistently have the highest probability, and majority voting can select an unreliable answer.The GSM8K example shows that the prevalent answer “60” can coincide with the greedy but unreliable path.
  • CoT-Decoding for Extracting CoT Paths: CoT paths typically yield larger answer-confidence margins than non-CoT paths, enabling CoT-decoding to identify more reliable reasoning trajectories.The confidence measure averages the probability difference between the top two tokens across answer tokens.
  • CoT-Decoding for Extracting CoT Paths: Early branching generally produces more diverse paths, although the best branching location can depend on the task.Later branching is constrained by earlier tokens, while mid-path branching can work effectively for year parity.
  • CoT-Decoding for Extracting CoT Paths: Weighted aggregation over decoded paths improves stability by selecting the answer with the largest summed confidence margin across paths producing that answer.This aggregation is intended to reduce sensitivity to small differences in model logits.

3. Experiments

Experiments evaluate CoT-decoding across tasks, model families, scales, and decoding settings. The method consistently improves reasoning, while its benefits vary with task difficulty and reveal specific reasoning vulnerabilities.

  • CoT-Decoding Effectively Elicits Reasoning from Language Models: CoT-decoding is the only tested decoding strategy reported to significantly enhance language-model reasoning, whereas some alternatives underperform greedy decoding.The comparison includes temperature sampling, top-k sampling, nucleus sampling, and beam search.
  • CoT-Decoding Effectively Elicits Reasoning from Language Models: CoT-decoding yields consistent accuracy gains across PaLM-2, Mistral, and Gemma on three math and commonsense reasoning tasks, sometimes doubling or tripling greedy-decoding performance.The cross-family result is reported in Figure 3.
  • CoT-Decoding Effectively Elicits Reasoning from Language Models: On GSM8K, CoT-decoding produces +10-30% absolute accuracy gains across PaLM-2 scales, while year-parity performance reaches almost perfect accuracy at larger scales.Greedy decoding remains flat on year parity as model size increases, but CoT-decoding recovers reasoning paths and substantially boosts performance.
  • CoT-Decoding Effectively Elicits Reasoning from Language Models: CoT-decoding partially closes the pre-trained versus instruction-tuned gap, with pre-trained PaLM-2 Large reaching 63.2% accuracy versus 67.8% for its instruction-tuned counterpart.The comparison is reported without supervised data for the pre-trained model.
  • CoT-Decoding Effectively Elicits Reasoning from Language Models: Higher k generally improves performance, but gains vary by task difficulty; CoT-decoding can also further improve instruction-tuned models and exposes state-tracking and arithmetic-order vulnerabilities.The paper reports stronger correct-path availability on simpler tasks and errors on Coin-Flip, Web-of-Lies, and Multi-step Arithmetic.
  • CoT-Decoding Effectively Elicits Reasoning from Language Models: CoT-decoding can be combined with CoT prompting to obtain larger reasoning gains, while maintaining strong performance relative to self-consistency at comparable multi-path cost.The paper also evaluates a path-aggregation algorithm using k = 10 for methods requiring multiple decoding paths.

4. Related Work

Related work primarily elicits chain-of-thought through prompting, tuning, or decoding methods designed for other objectives. This paper differs by searching token-level decoding paths without CoT prompts, fine-tuning, or additional models.

  • Chain-of-thought reasoning in large language models: Prior reasoning research largely improves language models through task-specific few-shot or zero-shot prompting that elicits chain-of-thought paths.Such prompts can encode task decomposition, induce additional tokens, or teach procedures through demonstrations.
  • Chain-of-thought reasoning in large language models: Step-control and verification methods still require CoT prompting, whereas this work searches token-level decoding paths and uses answer confidence.The contrast is with step-by-step verification, process feedback, self-evaluation-guided beam search, and PathFinder.
  • Instruction-tuning to elicit CoTs in language models: Instruction-tuning and distillation elicit reasoning without explicit prompting but typically require resource-intensive fine-tuning and many CoT-annotated examples.The paper contrasts these supervised approaches with its entirely unsupervised method, which uses no additional models.
  • Decoding algorithms for language models: Standard language-model decoding research emphasizes fluency, coherence, repetition reduction, and diversity, with relatively little work targeting reasoning specifically.Diverse beam search emphasizes generation diversity rather than accuracy, while other methods include sampling, beam search, and refined decoding algorithms.
  • Decoding algorithms for efficiency: Efficiency-focused decoding, such as speculative decoding, is orthogonal because it primarily targets decoding speed rather than reasoning performance.The paper notes that such techniques could potentially improve CoT-decoding efficiency.

5. Conclusion and Discussion

The paper finds that alternative top-k decoding paths can reveal CoT reasoning without specialized prompting, and introduces confidence-based CoT-decoding to extract more reliable paths. This approach improves reasoning performance, while incurring extra computation and leaving broader search and open-ended-answer limitations.

  • Top-k decoding reveals CoT reasoning paths that are naturally present in language models without specialized prompting.The approach investigates intrinsic reasoning capabilities without external prompt constraints.
  • CoT-decoding uses increased answer confidence associated with CoT paths to extract more reliable decoding paths.The method is based on the observed correlation between CoT presence and model confidence.
  • CoT-decoding enhances overall reasoning performance by selecting more reliable paths from the decoding space.
  • Exploring alternative decoding paths incurs additional computational costs, and searching at any token would increase those costs substantially.
  • For more open-ended answers, using probability differences between the top two tokens to indicate answer preference may be less precise.

A. Qualitative analysis on CoT paths elicited by CoT-decoding and existing approaches

Qualitative examples show that CoT-decoding generates more free-form reasoning than prompting approaches and can expose strategies less shaped by prompt designers. Path aggregation uses answer-confidence scores across paths to identify answers more reliably.

  • CoT-decoding produces more free-form CoTs than alternative CoT prompting methods.The authors attribute this to initial decoding diversity and the absence of explicit constraints.
  • CoT-decoding can reveal intrinsic problem-solving strategies without external prompts that may reflect prompt designers’ biases.
  • Few-shot CoT prompts consistently impose a standard analytical approach, whereas CoT-decoding reveals paths that deviate from that conventional approach.
  • For the correct answer 18, four paths have Δ values 0.994, 0.911, 0.584, and 0.999, yielding an overall sum of 3.5.
  • The path aggregation example shows why combining answer-confidence scores can identify the correct answer more reliably than other options.

B. Examples of CoT-decoding Paths on Additional Tasks

Additional examples show CoT paths emerging when alternative first-step tokens are explored, including for pretrained and instruction-tuned Mistral-7B models. CoT paths can remain lower-ranked, while CoT-decoding may make reasoning behavior more consistent.

  • Alternative first-step tokens again reveal CoT reasoning in the Mistral-7B pretrained model, despite CoT paths ranking lower during decoding.
  • Examples compare greedy decoding with CoT-decoding across various reasoning tasks.
  • CoT-decoding improves some instruction-tuned Mistral-7B cases by eliciting a CoT before the final answer more consistently.In one example, a CoT appears after an initial answer, so the generated reasoning can be affected by that answer.
  • Top-k paths for math and year-parity tasks show that CoT paths exist but are generally ranked lower in the decoding space.

C. Choice of 𝑘on Additional Models and Tasks

On Mistral-7B, increasing k consistently improves the pretrained model, whereas larger k does not necessarily add gains for the instruction-tuned model. Figure 6 presents accuracy as a function of k for both models.

  • Increasing k yields highly consistent improvement for the Mistral-7B pretrained model.
  • For the Mistral-7B instruction-tuned model, exploring more tokens at larger k does not necessarily produce further gains.The model is already fine-tuned with CoT data to generate CoTs for the first few decoding paths.
  • Figure 6 reports accuracy with respect to the choice of k over Mistral-7B.

D. Details on Experimental Settings

The experiments use task- and model-specific input, decoding, answer-extraction, and filtering procedures. Settings differ across PaLM-2 and Mistral models, while additional processing handles open-ended, invalid, or ill-formed outputs.

  • PaLM-2 settings: PaLM-2 CoT-decoding uses 256 input tokens and up to 128 decoding steps, while prompted baselines use longer inputs and outputs.Few-shot CoT prompting uses 1024 input tokens and 256 decoding steps; zero-shot CoT prompting also uses 256 decoding steps.
  • Input formatting: The default input format is “Q: [question]\nA:”, except multi-step arithmetic, which uses the original question format.For Mistral instruction-tuned models, questions instead use the model’s [INST] and [/INST] format.
  • Answer identification: Answer spans are extracted from final numerical outputs or available options for Mistral, while PaLM-2 continuations are aligned after “So the answer is”.This procedure operationalizes which generated text counts as the evaluated answer.
  • Answer processing: For open-ended outputs, Δ is averaged over continuation tokens; fixed yes/no tasks instead compare probability masses over affirmative and negative options.Math paths lacking the expected continuation are ignored, while invalid outputs are excluded from fixed-choice evaluation.
  • Response filtering: Zero-shot QA outputs are filtered when empty, maximally repeated, or ending in a question mark, using simple response-length and termination heuristics.These filters target unfinished or ill-formed generations.
  • Models and resources: Experiments cover PaLM-2, Mistral, and Gemma models using TPUs or A100 GPUs, with task runtimes ranging from hours to several days.Mistral and Gemma experiments use A100 GPUs with 40 GB RAM; PaLM-2 experiments use TPU v4.
Loading 2402.10200v2…