PaperlayerView exact revision on arXiv

Source-linked AI summary

The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models

Matthew Renze, Erhan Guven

arXiv:2401.05618v3cs.CLcs.AI

TL;DR

The paper asks how conciseness affects CoT response length and problem-solving, and introduces CCoT by combining concise prompting with CoT. CCoT reduced response token length while maintaining comparable performance, with a math-accuracy penalty for GPT-3.5.

  • Problem

    The paper asks how conciseness affects CoT response length and problem-solving capabilities.

    Source: D. Concise Chain-of-Thought (CCoT) · p. 1

  • Method

    CCoT combines CoT with concise prompting to seek a correct solution using the shortest possible response.

    Source: D. Concise Chain-of-Thought (CCoT) · p. 1

  • Results

    CCoT reduced GPT response token length by 48.70% while maintaining comparable performance, but GPT-3.5 accuracy on math problems fell 27.69%.

    Source: §V · p. 5

  • Takeaways & Limitations

    In practice, CCoT reduced the per-token cost of solving multi-step problems by 22.67%, with savings also extending to energy consumption and response times.

    Source: §V · p. 5

    The study tested only GPT-3.5 and GPT-4 and a single type of CoT and CCoT prompt, so findings may not generalize to other models or prompt variations.

    Source: A. Limitations · p. 4

Abstract

from arXiv · show

In this paper, we introduce Concise Chain-of-Thought (CCoT) prompting. We compared standard CoT and CCoT prompts to see how conciseness impacts response length and correct-answer accuracy. We evaluated this using GPT-3.5 and GPT-4 with a multiple-choice question-and-answer (MCQA) benchmark. CCoT reduced average response length by 48.70% for both GPT-3.5 and GPT-4 while having a negligible impact on problem-solving performance. However, on math problems, GPT-3.5 with CCoT incurs a performance penalty of 27.69%. Overall, CCoT leads to an average per-token cost reduction of 22.67%. All code, data, and supplemental materials are available on GitHub at https://github.com/matthewrenze/jhu-concise-cot

I. INTRODUCTION

The introduction frames CoT as a step-by-step prompting approach associated with improved problem-solving, but one that increases response length. It presents concise prompting as a way to reduce verbosity while noting that it may affect performance.

A. Background

  • LLMs have become common in AI systems that automate solutions to real-world problems, making effective prompt engineering relevant to their use.The introduction names CoT as one technique developed to improve performance for particular use cases and problem domains.Source: A. Background · p. 1

B. Chain-of-Thought (CoT) Prompting

  • CoT prompts ask LLMs to reason step by step, a technique associated with improved performance on many problem-solving tasks.The section describes zero-shot and few-shot CoT as distinct implementations.Source: B. Chain-of-Thought (CoT) Prompting · p. 1
  • CoT can improve performance on some tasks by up to 80%, but longer responses increase its cost.The introduction notes that CoT’s cost grows in proportion to response length.Source: B. Chain-of-Thought (CoT) Prompting · p. 1

C. Concise Prompting

  • Concise prompting reduces response verbosity and may lower costs, energy use, response time, and communication burden.Zero-shot prompting asks for a concise response; few-shot prompting uses concisely written examples.Source: C. Concise Prompting · p. 1
  • Concise prompting may hurt performance on some tasks when the LLM needs more verbosity to fully elaborate its reasoning steps.The introduction says prompt engineers often encourage verbose CoT, while noting that this convention appears based on anecdotal rather than empirical evidence.Source: C. Concise Prompting · p. 1

D. Concise Chain-of-Thought (CCoT)

The paper introduces CCoT to investigate how conciseness affects CoT response length and problem-solving. It combines concise prompting with CoT and relates the resulting comparison to practical costs and open questions about which reasoning elements matter.

D. Concise Chain-of-Thought (CCoT)

  • CCoT combines concise prompting with CoT to investigate how conciseness affects response length and problem-solving.The method aims to produce a chain of thought leading to a correct solution with the shortest possible response length.Source: D. Concise Chain-of-Thought (CCoT) · p. 1
  • CCoT asks the LLM to think step by step and be concise, with few-shot examples that use concise solutions.The paper points to Figure 12 for an example prompt and examples.Source: D. Concise Chain-of-Thought (CCoT) · p. 1

E. Contribution

  • The study examines how concise prompting affects CoT, with practical implications for engineers building LLM-based software.If CCoT reduces response length, the paper says it can reduce costs, energy consumption, and response times; APIs are typically priced per token,.Source: E. Contribution · p. 2
  • If CCoT does not decrease performance, the authors argue engineers should prefer it over standard CoT.This recommendation is conditional on there being no performance penalty.Source: E. Contribution · p. 2
  • Reducing CoT length without affecting performance would raise questions about which tokens or aspects are necessary for problem solving and which are superfluous.The paper frames this as a theoretical implication for research on CoT reasoning in LLMs.Source: E. Contribution · p. 2

F. Prior Literature

  • The paper situates its work alongside earlier CoT prompting research, including few-shot, zero-shot, and APE approaches [3]–.It also discusses an earlier attempt to develop concise CoT prompting that was withdrawn due to technical issues [10],.Source: F. Prior Literature · p. 2
  • The authors report finding no prior publications specifically studying concise prompting or concise CoT prompting.They characterize concise-prompting guidance as largely drawn from prompt-engineering best practices rather than direct study.Source: F. Prior Literature · p. 2

II. METHODS

The study uses MCQA problems drawn from standard benchmarks and evaluates two GPT models. Its dataset combines questions across benchmarks, while the experiments use Microsoft Azure OpenAI Service.

A. Data

  • The test set comprises MCQA problems from standard LLM benchmarks, selected to span problem domains and difficulty levels.Ten source benchmarks were drawn from ARC, AGIEval, HellaSwag, and MedMCQA [12]–.Source: A. Data · p. 2
  • The researchers randomly selected 100 questions from each of ten benchmarks to create a 1,000-question exam.The data were preprocessed into a standard format.Source: A. Data · p. 2

B. Models

  • The experiments compared GPT-3.5 and GPT-4.The paper describes GPT-3.5 as publicly released as ChatGPT in November 2022, and GPT-4 as a more powerful GPT with more advanced capabilities [20],.Source: B. Models · p. 2
  • The models were accessed through Microsoft Azure OpenAI Service, which the authors say uses the same underlying foundational models as OpenAI’s platform –.The authors state that results should hold for either platform.Source: B. Models · p. 2

C. Prompts

The prompt comparison includes answer-only, verbose CoT, and concise CoT conditions, evaluated using response length and correct-answer accuracy. The authors use Mann–Whitney U tests because the data were not normally distributed.

C. Prompts

  • The answer-only prompt requests only the answer and uses a one-shot example, providing a baseline for minimum response length and task performance.Source: C. Prompts · p. 2
  • The standard CoT prompt asks the LLM to think step by step and uses a verbose one-shot reasoning example, serving as an upper bound for response length and task performance.Source: C. Prompts · p. 2
  • The CCoT prompt adds “Be concise” to standard CoT and uses a concise one-shot solution to enable comparison with answer-only and standard CoT prompts.The paper directs readers to Figures 10–12 for prompt samples.Source: C. Prompts · p. 2

D. Metrics

  • The comparison measures response length in output tokens and performance as correctly answered questions divided by total questions.The latter metric is used to compare the models’ ability to solve MCQA problems correctly.Source: D. Metrics · p. 2

E. Analysis

  • Mann–Whitney U tests compare the CoT and CCoT response-length distributions and their correct-answer-accuracy distributions.The tests were used to assess statistical significance.Source: E. Analysis · p. 3
  • The authors chose Mann–Whitney U rather than a t-test because Shapiro–Wilk tests found response-length and accuracy data were not normally distributed.The paper identifies nonnormality as violating the t-test’s normality assumption.Source: E. Analysis · p. 3

III. RESULTS

CCoT substantially shortened responses for both models, with the reduction holding across problem domains. Overall accuracy was not statistically meaningfully reduced, though GPT-3.5 showed a significant penalty on math exams.

A. Response Length

B. Performance

  • CCoT did not reduce overall problem-solving performance in a statistically meaningful way compared with standard CoT.GPT-3.5 accuracy decreased by 2.95% (p = 0.55); GPT-4 accuracy increased by 0.25% (p = 0.21), with neither difference statistically significant.Source: B. Performance · p. 3
  • 27.69%: GPT-3.5 CCoT reduced accuracy on AQUA-RAT and SAT Math; math was the only domain with a statistically significant decrease.GPT-4 showed no statistically significant math-performance decrease versus CoT, and CCoT significantly outperformed the answer-only prompt on math.Source: B. Performance · pp. 3–4

C. Cost Analysis

Using current per-token pricing to estimate the cost of solving 1,000 problems, CCoT reduced total costs for both models.

C. Cost Analysis

  • The cost estimate used current per-token pricing for GPT-3.5 and GPT-4 to calculate the cost of solving all 1,000 problems.The listed prices were $0.001 per 1,000 input tokens and $0.002 per 1,000 output tokens for GPT-3.5, and $0.03 and $0.06, respectively, for GPT-4.Source: C. Cost Analysis · p. 4
  • 21.85% for GPT-3.5 and 23.49% for GPT-4: CCoT reduced total costs for solving 1,000 problems.The authors state that these cost savings should scale linearly.Source: C. Cost Analysis · p. 4, Table I · p. 4

IV. DISCUSSION

The study’s implications are tempered by limits in model, prompt, domain, and GPT-4 accuracy coverage. Within those bounds, the authors connect concise responses to practical resource savings and questions about which CoT tokens matter.

A. Limitations

  • The study tested only two OpenAI GPT models, so its findings may not generalize to other proprietary or open-source LLMs.The authors name Llama 2, PaLM, and Claude as examples outside the tested scope.Source: A. Limitations · p. 4
  • Only one type each of CoT and CCoT prompt was tested, and other prompt variations may produce different results.Source: A. Limitations · p. 4
  • The study covered ten problem domains, so findings may not hold for other domains or sub-domains.Source: A. Limitations · p. 4
  • GPT-4’s median accuracy was 1.0, creating a ceiling effect that may have affected statistical analysis of performance.Source: A. Limitations · p. 4

B. Implications

  • Per-token API pricing makes shorter outputs a direct cost-saving opportunity, and the authors also identify potential energy and response-time savings.The study’s implications section also notes that each MCQA problem was solved ten times, making total costs roughly 10x the reported values.Source: B. Implications · p. 4
  • The authors argue that roughly halving output tokens while maintaining accuracy suggests only some CoT tokens contribute to performance, raising questions about which are necessary.Source: B. Implications · p. 4

C. Future Research

The conclusion reports substantially shorter CCoT responses with comparable performance overall, alongside a GPT-3.5 math penalty and lower per-token cost. The authors propose follow-up tests across models, prompts, tasks, and error types.

C. Future Research

  • The authors recommend testing other proprietary and open-source LLMs to assess whether results hold beyond GPT-3.5 and GPT-4.Source: C. Future Research · p. 5
  • They recommend evaluating more CCoT prompt variations, including stronger concision instructions and more concise few-shot examples.The proposed tests would examine whether these variants shorten responses without sacrificing accuracy.Source: C. Future Research · p. 5
  • The authors call for tests beyond the MCQA task type and the problem domains represented in their test set, including other math sub-domains.Source: C. Future Research · p. 5
  • They recommend a deeper error analysis to identify whether particular CoT error types affect model performance and could be mitigated.Source: C. Future Research · p. 5

V. CONCLUSION

  • The paper introduces CCoT as a prompt-engineering technique.Source: §V · p. 5
  • 48.70% shorter response token length accompanied comparable performance to standard CoT, but GPT-3.5 incurred a 27.69% accuracy reduction on math problems.GPT-4 incurred no performance penalty in any problem domain.Source: §V · p. 5
  • 22.67% lower per-token cost for solving multi-step problems was reported in practice, alongside reduced energy consumption and shorter response times.Source: §V · p. 5

APPENDIX

The appendix includes figures comparing response length and performance across domains, plus materials describing the multi-domain MCQA problem sets and a dataset-availability note.

A. Response Length for GPT-3.5

  • The GPT-3.5 response-length figure reports significant decreases with CCoT compared with CoT across all problem domains.Source: Figure 5 · p. 6

B. Response Length for GPT-4

  • The GPT-4 response-length figure likewise reports significant decreases with CCoT compared with CoT across all problem domains.Source: Figure 6 · p. 6

C. Performance for GPT-3.5

  • For GPT-3.5, verbose CoT significantly improved performance over CCoT on AQUA-RAT and SAT Math, with minimal impact in other domains.Source: Figure 7 · p. 6

D. Performance for GPT-4

  • For GPT-4, both verbose CoT and CCoT significantly improved math accuracy, with minimal impact on other problem domains.Source: Figure 8 · p. 7

E. Problem Sets

  • Table II describes the problem sets used to create a 1,000-question multi-domain MCQA exam.Source: Table II · p. 7
  • The LogiQA repository has no license file, although the paper and readme state that the dataset is freely available.Source: E. Problem Sets · p. 7

F. Sample MCQA Problem

The appendix provides a sample MCQA item and an answer-only prompt example. The item asks for a sphere’s diameter around an inscribed rectangular solid; the prompt example answers a city-choice question with B.

F. Sample MCQA Problem

  • The sample MCQA item asks for the diameter of a sphere containing an inscribed 3 x 4 x 15 rectangular solid, and lists answer choice B as correct.Source: F. Sample MCQA Problem · p. 7

G. Answer-Only Prompt

H. Verbose CoT Prompt

The appendix supplies a verbose CoT prompt and worked MCQ example, then a concise prompt and example that shorten the reasoning while retaining the answer-choice format.

H. Verbose CoT Prompt

  • The verbose prompt directs step-by-step reasoning, specifies an Action: Answer("[choice]") response, and requires a choice other than “None of the Above.”Source: H. Verbose CoT Prompt · p. 8
  • The verbose example asks for the capital of the state where Johns Hopkins University is located and offers four city choices.Source: H. Verbose CoT Prompt · p. 8
  • The worked verbose solution links Johns Hopkins University in Baltimore to Maryland, then identifies Annapolis as the state capital and selects B.Source: H. Verbose CoT Prompt · p. 8
  • Figure 11 presents a sample standard, verbose CoT system prompt and one-shot example.Source: Figure 11

I. Concise CoT Prompt

  • The concise prompt retains the answer-choice format and adds “Be concise,” with a shorter thought identifying Maryland’s capital as Annapolis and selecting B.Source: §I · p. 8
  • Figure 12 presents a sample CCoT system prompt and one-shot example.Source: Figure 12
Loading 2401.05618v3…