Source-linked AI summary
Chain of Draft: Thinking Faster by Writing Less
Silei Xu, Wenhao Xie, Lingxiao Zhao, Pengcheng He
TL;DR
LLMs can solve complex reasoning tasks with CoT, but verbose intermediate outputs increase token use, latency, and computational demands. This paper introduces CoD, which uses concise, information-dense drafts inspired by human reasoning. Across reasoning tasks, CoD maintains or improves accuracy while reducing latency and cost, though its performance gap versus CoT is larger on small models.
Problem
CoT supports complex multi-step reasoning but often generates verbose outputs that increase token usage, latency, and computational resource demands.
Method
CoD prompts LLMs to generate concise, dense-information intermediate reasoning outputs inspired by humans’ use of minimal drafts.
Results
Across arithmetic, common-sense, and symbolic reasoning benchmarks, CoD maintains or improves accuracy compared with CoT while reducing token usage and latency.
Takeaways & Limitations
CoD’s token efficiency reduces computational cost and makes reasoning more appealing for latency-sensitive, cost-sensitive, and large-scale deployments.
Takeaways & Limitations
On models with fewer than 3B parameters, CoD’s performance gap relative to CoT is more pronounced, despite reducing response tokens and improving over direct answers.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks through mechanisms like Chain-of-Thought (CoT) prompting, which emphasizes verbose, step-by-step reasoning. However, humans typically employ a more efficient strategy: drafting concise intermediate thoughts that capture only essential information. In this work, we propose Chain of Draft (CoD), a novel paradigm inspired by human cognitive processes, where LLMs generate minimalistic yet informative intermediate reasoning outputs while solving tasks. By reducing verbosity and focusing on critical insights, CoD matches or surpasses CoT in accuracy while using as little as only 7.6% of the tokens, significantly reducing cost and latency across various reasoning tasks. Our code and data are available at https://github.com/sileix/chain-of-draft.
1 Introduction
Chain-of-Draft (CoD) proposes concise intermediate reasoning inspired by human drafting, aiming to preserve accuracy while reducing verbosity, latency, and computational cost.
- Motivation: CoT improves complex-task reasoning but produces verbose outputs that increase inference-time computation and latency.Humans more often use concise drafts or shorthand notes to retain essential insights.
- Contribution: Chain-of-Draft generates concise, dense-information intermediate outputs instead of verbose reasoning steps.The approach prioritizes efficiency and minimalism while retaining the reasoning process.
- Motivation: CoD is motivated by humans’ practice of recording only critical information when solving mathematical, writing, and coding problems.This minimizes overhead while helping progress toward solutions.
- Results: CoD maintains or improves accuracy relative to CoT across arithmetic, common-sense, and symbolic reasoning benchmarks while reducing token usage and latency.The evaluation covers multiple benchmarks requiring multi-step reasoning.
- Contributions: The paper reports reduced latency and cost without sacrificing accuracy, and discusses implications for LLM design, deployment, and real-world usability.These are presented as central empirical and practical contributions.
2 Related Work
Related work develops structured reasoning and latency-reduction methods, but these approaches can retain computational costs, require parallelizable tasks, reduce interpretability or accuracy, or rely on global token-budget estimates.
- Structured Reasoning Frameworks: Structured reasoning methods such as CoT, self-consistency, and ReAct improve reasoning capabilities but can substantially increase token usage and resource consumption.Overthinking may cause unnecessary computation even on simple tasks.
- Latency Reduction: Streaming reduces perceived latency by showing partial outputs, but does not fully reduce total latency or computational cost.It is also often unsuitable when intermediate reasoning is not intended for users.
- Latency Reduction: Skeleton-of-Thought lowers latency through outlines and parallel decoding, but does not reduce computational cost and is limited to effectively parallelizable questions.Another approach uses fast draft tokens followed by validation.
- Latency Reduction: Coconut reduces latency and cost but can reduce accuracy on complex tasks and loses natural-language interpretability, while excluding black-box models.The passage specifically cites reduced performance on GSM8k.
- Budget-Aware Reasoning: CCoT uses a fixed global reasoning budget, whereas TALE estimates budgets dynamically but adds an LLM call and assumes accurate complexity prediction.Both approaches can face limitations when tasks need reflection, self-correction, or external knowledge retrieval.
- CoD’s Distinction: CoD instead uses a per-step budget with unlimited reasoning steps, making it more adaptable to structured reasoning techniques.This contrast is presented as a distinction from global-budget methods.
3 Chain-of-Draft Prompting
Chain-of-Draft compresses multi-step reasoning into minimal, abstract representations that preserve essential operations, transparency, and correctness while reducing verbosity and token use.
- Method: CoD addresses CoT verbosity by replacing exhaustive intermediate explanations with concise drafts inspired by humans’ essential-result notes.The method targets multi-step reasoning tasks such as mathematical or logical puzzles.
- Prompting Strategies: Standard prompting directly outputs an answer, which can lack derivational transparency and provide little help with intermediate results.The passage associates this setup with hallucination during multi-step reasoning.
- Prompting Strategies: CoT supplies detailed, interpretable reasoning but includes irrelevant details that inflate token count and response latency.The arithmetic example illustrates this verbosity around contextual details that are unnecessary for solving the problem.
- Chain-of-Draft Prompting: CoD distills reasoning into minimal abstract representations, focusing on the essential mathematical operation while omitting irrelevant context.The example represents the solution as a concise equation.
- Chain-of-Draft Prompting: In the arithmetic example, CoD expresses the solution as 20 - x = 12; x = 20 - 12 = 8.The representation preserves the intermediate operation and final answer in compact form.
4 Experiments
The experiments compare Standard prompting, Chain-of-Thought, and Chain-of-Draft across arithmetic, commonsense, and symbolic reasoning tasks using GPT-4o and Claude 3.5 Sonnet. CoD generally preserves or improves accuracy while reducing token usage, latency, and cost, with weaker performance in zero-shot and small-model settings.
- Evaluation setup: The study evaluates arithmetic, commonsense, and symbolic reasoning using GSM8K, BIG-bench date and sports understanding, and coin-flip tasks.GPT-4o and Claude 3.5 Sonnet are tested with Standard prompting, CoT, and CoD.
- Evaluation setup: CoD limits each intermediate reasoning step to at most five words while retaining step-by-step prompting.Standard prompting requests only the final answer, whereas CoT uses unrestricted step-by-step reasoning.
- Arithmetic reasoning: 91% accuracy for both models on GSM8K requires about 40 tokens per response with CoD, versus over 95% accuracy and approximately 200 tokens with CoT.CoD reduces average output token count by 80%; latency falls by 76.2% for GPT-4o and 48.4% for Claude 3.5 Sonnet.
- Commonsense reasoning: 92.4% fewer tokens are produced by CoD than CoT for Claude 3.5 Sonnet on sports understanding, reducing average output from 189.4 to 14.3 tokens.Across commonsense tasks, CoD significantly reduces latency and cost and outperforms CoT in accuracy in various cases.
- Symbolic reasoning: 100% accuracy is achieved by both models with CoT and CoD on coin-flip reasoning, while CoD reduces tokens relative to CoT by 68% for GPT-4o and 86% for Claude 3.5 Sonnet.The coin-flip test set contains 250 synthesized examples following the original task design.
- Limitations: In zero-shot GSM8K, CoD improves Claude 3.5 Sonnet over direct answering by only 3.6%, and its token savings are less significant than in few-shot evaluation.The authors hypothesize that models lack sufficient CoD-style reasoning patterns in training data without few-shot guidance.
- Limitations: For models with fewer than 3B parameters, CoD reduces tokens and improves accuracy over direct answering, but its performance gap relative to CoT is more pronounced.The authors anticipate that additional CoD-formatted fine-tuning data could improve these models’ CoD reasoning accuracy.
5 Discussion
Chain of Draft targets overlooked inference latency and computational cost by replacing lengthy reasoning with concise drafts. The paper presents CoD as an efficient alternative that maintains or improves accuracy while reducing verbosity.
- CoD substantially reduces reasoning latency while achieving comparable or superior accuracy to standard Chain-of-Thought prompting.It uses concise reasoning drafts instead of lengthy intermediate steps to speed response generation without sacrificing correctness.
- Compacting reasoning steps reduces input tokens for few-shot prompting and shortens output length, lowering computational cost.This token efficiency is especially relevant to large-scale deployments and applications with strict budget constraints.
- CoD suggests that effective reasoning can maintain depth without requiring lengthy outputs, while future work may combine it with other latency-reduction methods or compact-reasoning training.The discussion identifies adaptive parallel reasoning, multi-pass validation, and training with compact reasoning data as directions for further optimization.