PaperlayerView exact revision on arXiv

Source-linked AI summary

Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation

Yu Wang, Shiwan Zhao, Zhihu Wang, Heyuan Huang, Ming Fan, Yubo Zhang, Zhixing Wang, Haijun Wang, Ting Liu

arXiv:2409.03271v1cs.AIcs.CLcs.HC

TL;DR

CoT reasoning can be unstable because generated paths vary in quality. SCoT elicits a strategy before generating reasoning steps, and experiments report improvements across reasoning datasets, with a few-shot extension using matched demonstrations.

  • Problem

    CoT paths vary in quality, creating a risk of erroneous final outcomes even when a path aligns with valid reasoning.

    Source: Introduction · p. 1

  • Method

    SCoT elicits a problem-solving strategy in a single prompt and uses it to guide CoT generation and the final answer.

    Source: Introduction · p. 2

  • Results

    21.05% on GSM8K and 24.13% on Tracking Objects are reported as increases with Llama3-8b, within experiments across eight reasoning datasets.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The paper concludes that both zero-shot and few-shot SCoT are effective approaches for improving reasoning-path quality in LLMs.

    Source: Conclusion · p. 7

    The authors identify evaluating SCoT on more complex problems and exploring further applications as future work.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

The Chain-of-Thought (CoT) paradigm has emerged as a critical approach for enhancing the reasoning capabilities of large language models (LLMs). However, despite their widespread adoption and success, CoT methods often exhibit instability due to their inability to consistently ensure the quality of generated reasoning paths, leading to sub-optimal reasoning performance. To address this challenge, we propose the \textbf{Strategic Chain-of-Thought} (SCoT), a novel methodology designed to refine LLM performance by integrating strategic knowledge prior to generating intermediate reasoning steps. SCoT employs a two-stage approach within a single prompt: first eliciting an effective problem-solving strategy, which is then used to guide the generation of high-quality CoT paths and final answers. Our experiments across eight challenging reasoning datasets demonstrate significant improvements, including a 21.05\% increase on the GSM8K dataset and 24.13\% on the Tracking\_Objects dataset, respectively, using the Llama3-8b model. Additionally, we extend the SCoT framework to develop a few-shot method with automatically matched demonstrations, yielding even stronger results. These findings underscore the efficacy of SCoT, highlighting its potential to substantially enhance LLM performance in complex reasoning tasks.

Introduction

The section motivates SCoT as a way to address variable CoT path quality by eliciting a problem-solving strategy before generating reasoning steps and answers. It reports improvements across reasoning tasks and presents automatically matched demonstrations as an extension.

Introduction

  • CoT paths vary in quality, and even a valid reasoning path may be followed by an erroneous final outcome.The section links this variability to reduced reliability in critical applications.Source: Introduction · p. 1
  • Other approaches can mitigate path-quality variability but may require up to 40 queries, multi-stage prompting, or external knowledge sources.SCoT is presented as avoiding multi-query approaches and additional knowledge sources.Source: Introduction · p. 2
  • SCoT elicits a problem-solving strategy before using it to guide CoT generation and the final answer, within a single prompt.The few-shot extension uses strategic knowledge to select relevant demonstrations.Source: Introduction · p. 2
  • 21.05% on GSM8K and 24.13% on Tracking Objects are reported as increases with Llama3-8b.The experiments span eight reasoning datasets across five domains.Source: Introduction · p. 2
  • The work contributes a two-stage strategy-guided CoT method, strategic-knowledge-based demonstration matching, and experiments across multiple reasoning domains.The experiments include mathematical, commonsense, physical, spatial, and multi-hop reasoning.Source: Introduction · p. 2

Related Work

The related work describes diverse strategies for problem solving and reviews methods that seek to improve CoT paths through voting, retrieval, tools, and prompt selection.

Strategic Diversity in Problem Solving

Enhancement of CoT Path

Method

The method defines strategic knowledge as a guide for stable reasoning, then uses it to shape CoT generation. A few-shot extension selects demonstrations according to the elicited strategy.

Strategic Knowledge

  • Strategic knowledge is a well-defined method or principle that guides reasoning toward a correct and stable solution.Its principles include a comprehensive approach and steps that are sufficiently detailed without being overly complex.Source: Strategic Knowledge · p. 3

Strategic Chain-of-Thought

  • SCoT elicits strategic knowledge before answering, identifying an effective approach and then applying it to solve the problem.Its prompt has five components—Role, Workflow, Rule, Initialization, and Task Input—and a three-step workflow within one prompt.Source: Strategic Chain-of-Thought · p. 3
  • Strategic knowledge varies by domain, favoring efficient solutions in mathematics, relevant formulas in physics, and appropriate decomposition in multi-hop reasoning.The section also describes developing an overarching method or workflow in other domains.Source: Strategic Chain-of-Thought · p. 3

Few-shot Strategic Chain-of-Thought

  • Few-shot SCoT builds a corpus from accurate zero-shot SCoT answers, then uses generated strategy to match demonstrations during a two-query inference process.Corpus construction assumes the strategies used in retained examples are correct and relevant.Source: Few-shot Strategic Chain-of-Thought · p. 3, §1 · p. 3

1. Strategic Knowledge

The figures identify the zero-shot and few-shot variants and show the prompt templates and a math-task workflow.

1. Strategic Knowledge

  • Figure 2 contrasts zero-shot SCoT with few-shot SCoT, which adds selected demonstrations; the few-shot details are omitted.Source: Figure 2 · p. 4
  • Figure 3 presents prompt templates for zero-shot and few-shot SCoT; the few-shot template integrates selected demonstrations into the prompt.Source: §1 · p. 3, Figure 3 · p. 4
  • Figure 4 illustrates a math-task workflow in which the prompt identifies and elicits a strategy before applying it to generate an answer.Source: Figure 4 · p. 4

Experimental Setup

The setup evaluates SCoT across reasoning datasets, model families, and prompting baselines.

Datasets and Tasks

  • Datasets span mathematical and physical, commonsense, multi-hop, and spatial reasoning; few-shot evaluation is limited to MathQA, AQuA, GSM8K, and ARC.Source: Datasets and Tasks · p. 4

Models

  • The tested models include Llama3, Llama2, Mistral-7B, Qwen2, and ChatGLM4-9B, with both 8B- and 70B-scale models represented.Source: Models · pp. 4–5

Baselines

  • Baselines are zero-shot prompting, Self-Consistency, and Step Back; Step Back is tested on five datasets, and accuracy averages three inferences per model.Source: Baselines · p. 5

Experimental Results

The experiments compare SCoT with existing prompting methods across datasets and models, then examine scale, prompt components, case studies, and output length.

Results across all Datasets

  • SCoT 1-shot−, with strategy-matched demonstrations, improves performance across most datasets over CoT 0-shot, while SCoT 1-shot achieves the best overall results.Source: Results across all Datasets · p. 5

Model Scale

  • SCoT improves accuracy at all three Llama2 sizes, with gains marginally shrinking as models grow; larger models more often generate strategic knowledge in zero-shot CoT.Source: Figure 5 · p. 5, Model Scale · p. 5

Results across all Datasets

Results across all Models

Ablation Study

  • Roles, workflows, and markdown formatting progressively raise CoT 0-shot accuracy, while more SCoT demonstrations slightly improve or preserve performance.Source: Ablation Study · p. 5, Table 3 · p. 6

Case Study

  • Cases show strategy prompting favoring inequality-based math, selecting a correct physics formula despite misleading wording, and using overall context in multi-hop reasoning.Source: Case Study · p. 6

Efficiency Analysis

Discussions

The paper tests whether SCoT prompt templates can be generated automatically and reports that these templates remain better than zero-shot CoT on AQuA.

Automatic SCoT

  • Qwen2-72B generated SCoT-based prompt templates from the SCoT concept, and the templates were tested on AQuA.Source: Automatic SCoT · p. 7
  • Automatically generated prompts score below manually crafted SCoT prompts but above zero-shot CoT on AQuA.Source: Automatic SCoT · p. 7
  • The AQuA comparison supports the feasibility of automatically generating SCoT-based prompt templates.Source: Automatic SCoT · p. 7

Conclusion

The conclusion presents SCoT as a framework for improving LLM reasoning paths, with both zero-shot and few-shot versions reported as effective. It identifies evaluation on more complex problems and further applications as future work.

Conclusion

  • SCoT uses strategic knowledge to support LLMs in autonomously generating reasoning paths; the paper also introduces a few-shot version with strategy-matched demonstrations.The conclusion reports experimental effectiveness for both zero-shot and few-shot SCoT.Source: Conclusion · p. 7
  • Evaluating SCoT on more complex problems and exploring further applications are identified as future research directions.Source: Conclusion · p. 7

Details of Experiments

The experiments use LLM-generated automated SCoT prompts, a set of models and datasets, and specified inference and evaluation procedures.

Details of Experiments

Models Details

  • The experiment uses ten models, described as nine public models and one private model.Source: Models Details · p. 10

Datasets Details

  • Eight publicly available datasets are used, with correctness checked against gold choices for multiple-choice tasks and exact answers for GSM8K.The datasets are MathQA, AQuA, GSM8K, MMLU, ARC, StrategyQA, CommonsenseQA, and Tracking Object.Source: Datasets Details · p. 10

Other Details

  • Most experiments use temperature 0 and top p 1; Self-Consistency uses temperature 0.5 and top p 0.5, and vllm is the inference framework.Source: Other Details · p. 10

Results

The results section reports accuracy averaged across three independent inference runs for most experiments, alongside a case study of elicited strategic knowledge across several reasoning domains.

All Results

  • Accuracy is the evaluation metric, with results averaged over three inference runs except for Self-Consistency, which receives one run because of computational cost.Accuracy and standard deviation results are presented in Tables 8 and 9.Source: All Results · p. 10

Case Study

  • The case study presents representative strategic-knowledge examples in math, physical, commonsense, multi-hop, and spatial reasoning.Source: Case Study · p. 10

Experimental Prompts

The experimental prompts and figures provide examples of standard CoT and SCoT prompts, path comparisons, and prompt variants. The referenced tables identify model, dataset, and accuracy scopes.

Experimental Prompts

Related papers

Loading 2409.03271v1…