PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching

Simon A. Aytes, Jinheon Baek, Sung Ju Hwang

arXiv:2503.05179v4cs.CLcs.AIcs.LG

TL;DR

CoT-based reasoning can improve complex-task performance but often produces verbose intermediate outputs that increase token and computational costs. SoT uses cognitively inspired, task-specific sketching paradigms with dynamic routing to generate shorter structured traces. Across diverse evaluations, it reduces token usage by up to 84% with minimal accuracy loss and sometimes improves accuracy.

  • Problem

    CoT and related reasoning prompts improve multi-step reasoning but increase token usage and computational overhead, creating an efficiency challenge for constrained deployment.

  • Method

    SoT combines Conceptual Chaining, Chunked Symbolism, and Expert Lexicons with a lightweight router that dynamically selects a task-suitable reasoning paradigm.

  • Results

    Across 18 reasoning datasets and multiple languages and modalities, SoT reduces output token usage by up to 84% while preserving accuracy in most tasks and improving performance in some.

  • Takeaways & Limitations

    SoT offers a practical alternative to verbose prompting for resource-constrained reasoning while retaining compact, semantically faithful traces.

  • Takeaways & Limitations

    Fixed exemplars per paradigm may limit adaptability to subtle variations within task types, domains, and queries.

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have enabled strong reasoning capabilities through Chain-of-Thought (CoT) prompting, which elicits step-by-step problem solving, but often at the cost of excessive verbosity in intermediate outputs, leading to increased computational overhead. We propose Sketch-of-Thought (SoT), a prompting framework that integrates cognitively inspired reasoning paradigms with linguistic constraints to reduce token usage while preserving reasoning accuracy. SoT is designed as a flexible, modular approach and is instantiated with three paradigms--Conceptual Chaining, Chunked Symbolism, and Expert Lexicons--each tailored to distinct reasoning tasks and selected dynamically at test-time by a lightweight routing model. Across 18 reasoning datasets spanning multiple domains, languages, and modalities, SoT achieves token reductions of up to 84% with minimal accuracy loss. In tasks such as mathematical and multi-hop reasoning, it even improves accuracy while shortening outputs.

1 Introduction

Chain-of-Thought reasoning can improve complex-task performance but often generates verbose intermediate outputs that increase token usage and computational overhead. Sketch-of-Thought addresses this with cognitively inspired, dynamically selected sketching paradigms, reducing tokens while preserving or improving accuracy.

  • CoT improves reasoning by encouraging models to decompose problems into intermediate natural-language steps.
  • Verbose CoT outputs increase token usage and computational overhead, limiting suitability for latency- or budget-constrained deployment.
  • SoT produces concise, structured reasoning steps that preserve essential logic while avoiding full-sentence elaboration.Its representations are inspired by symbolic sketches, mathematical notation, and expert shorthand.
  • Three cognitively motivated paradigms—Conceptual Chaining, Chunked Symbolism, and Expert Lexicons—target distinct reasoning tasks through training-free prompts.A lightweight router selects the most suitable paradigm from the query’s structure at inference time.
  • 84% is the maximum reported reduction in output token usage, with no significant accuracy loss and performance improvements in some domains.The evaluation spans 18 reasoning datasets across mathematical, commonsense, logical, multi-hop, scientific, and medical domains.
  • SoT is presented as a concise, structured prompting framework that maintains or improves accuracy while reducing token usage across diverse datasets, models, languages, and modalities.

2 Method

The method restructures intermediate reasoning into compact sketches rather than verbose natural-language traces. Three task-aligned paradigms are combined with a lightweight router that selects a prompt-exemplar pair dynamically from the query.

  • 2.1 Preliminary: Prompted reasoning generates an intermediate trace and final answer, while CoT expresses the trace as natural-language steps.
  • 2.2 Sketch-of-Thought (SoT): SoT compresses the reasoning process itself, eliciting concise structured steps instead of compressing the input prompt.
  • 2.2 Sketch-of-Thought (SoT): SoT prompts produce sketched steps with the same logical content as CoT steps but fewer tokens, enforcing linguistic constraints and cognitive structuring.Formally, the sketched sequence is [ŝ1, ŝ2, . . . , ŝm, a], with |ŝ| < |s|.
  • 2.2 Sketch-of-Thought (SoT): Conceptual Chaining links essential concepts into concise logical sequences for commonsense, multi-hop, logical, and scientific reasoning.Its examples use connected concepts such as Seoul → South Korea → Won.
  • 2.2 Sketch-of-Thought (SoT): Chunked Symbolism organizes numerical and symbolic reasoning into compact steps, while Expert Lexicons use domain-specific shorthand for technical tasks.
  • 2.3 Adaptive Paradigm Selection: A lightweight router selects the paradigm dynamically using semantic and structural features of each query.The routing process is PSoT = ROUTER(q), with DistilBERT used as the base model.

3 Experimental Setup

The evaluation compares SoT with established reasoning prompts across diverse reasoning categories, models, and input settings. Accuracy and intermediate output-token count are the primary metrics.

  • 3.1 Datasets: 15 datasets cover mathematical, commonsense, logical, multi-hop, scientific, and medical reasoning categories.
  • 3.1 Datasets: Multilingual evaluation uses MMLU and professionally translated MMMLU, while multimodal evaluation uses GQA and image-based ScienceQA.
  • 3.2 Baselines: CoT, CCoT-45, and CoD provide the principal baselines, using natural-language reasoning, a 45-word global limit, and five-word per-step limits, respectively.
  • 3.3 Implementation Details: The model suite includes Qwen-2.5 variants, LLaMA-3.1-8B, LLaMA-3.2-11B, GPT-4o, Claude Sonnet 3.5, and multimodal Qwen-2.5-VL-7B.
  • 3.3 Implementation Details: Table 1 reports average Acc and Tkn values by reasoning type, plus overall token reduction (Red.) and accuracy change (∆).
  • 3.3 Implementation Details: Accuracy and output token count are the two primary metrics, with tokens measured over generated intermediate reasoning.Exact match is used for multiple-choice, yes/no, and numeric tasks; open-ended correctness is assessed with GPT-4o.

4 Results and Discussion

Across models, tasks, languages, and modalities, SoT substantially compresses reasoning outputs while generally preserving or improving accuracy. Its adaptive paradigms and routing support favorable efficiency–performance trade-offs, though technical and fine-grained visual tasks show greater variability.

  • Overall Performance: Over 74% average token reduction relative to CoT typically comes with accuracy deviations within 1% across evaluated models.The pattern holds across open-weight and proprietary models, with statistically insignificant accuracy impact reported across runs.
  • Model-wise Trends: 82.30% accuracy on Qwen-2.5-32B exceeds CoT’s 82.24% while reducing output tokens by 74.36%.GPT-4o and Claude Sonnet 3.5 likewise remain close to CoT accuracy while reducing token usage by 76% and 68%, respectively.
  • Paradigm-Task Performance: 86.94% mathematical accuracy versus CoT’s 84.17% accompanies compression from 222 to 88 average tokens on Qwen-2.5-32B.Chunked Symbolism is identified as the dominant paradigm for this category and as effective for concise arithmetic reasoning.
  • Paradigm-Task Performance: 92.00% commonsense accuracy at 34 tokens compares with CoT’s 91.48% at 177 tokens in Qwen-2.5-32B.Conceptual Chaining is described as the prevailing strategy for commonsense and multi-hop reasoning.
  • Paradigm-Task Performance: Expert Lexicons often use half as many tokens as CoT while preserving competitive accuracy on variable domain-specialized tasks.Paradigm–task alignment results report the expected-dominant paradigm outperforming alternatives in accuracy, while token efficiency can favor another paradigm.
  • Cross-setting Evaluation: SoT improves the efficiency–performance trade-off over CoD and CCoT, while extending to ensemble, multilingual, and multimodal settings with task-dependent variation.Multilingual tests reduce output length by over 80% with accuracy decreases from -0.33% to -1.33%; multimodal results range from a 6.60% gain on ScienceQA to a 2.50% loss on GQA.

5 Related Work

Prior token-efficient reasoning methods shorten outputs through fixed constraints, summarization, or prompt compression, while structured strategies expand reasoning via search or sampling. These approaches can improve efficiency or accuracy but may sacrifice clarity, performance, or computational efficiency.

  • Token-Efficient Reasoning: Fixed constraints and two-stage summarization reduce reasoning-trace length, but rely on step, word, or surface-level compression heuristics.Concise Chain-of-Thought, Constrained CoT, and SCOTT represent three such approaches.
  • Structured Reasoning Strategies: Tree-of-Thoughts and Graph-of-Thoughts restructure reasoning as searches over graph-structured intermediate steps.These methods alter the generation process rather than only shortening the output.
  • Structured Reasoning Strategies: Self-Consistency samples multiple reasoning paths and selects the majority answer to improve stability.The method expands inference by considering several candidate reasoning trajectories.
  • Token-Efficient Reasoning: Chain-of-Draft reduces length with densely packed natural language, but can reduce clarity and substantially hurt performance on complex reasoning tasks.This trade-off distinguishes it from approaches that aim to preserve both compactness and reasoning quality.
  • Token-Efficient Reasoning: CoT-Influx and LLMLingua compress input exemplars, while cascaded and compute-adaptive methods reduce computation selectively.These techniques target prompt length or inference allocation rather than the structure of reasoning traces themselves.

6 Conclusion

The conclusion presents Sketch-of-Thought as a cognitively informed prompting framework for compact, semantically faithful reasoning. Across broad evaluations, it reduces token usage while generally preserving accuracy and offers practical applicability, including in resource-constrained settings.

  • 6 Conclusion: Sketch-of-Thought reduces reasoning token usage by up to 84% while preserving accuracy in most tasks and incurring minor tradeoffs in others.The framework generates compact reasoning traces intended to remain semantically faithful.
  • 6 Conclusion: Extensive experiments across 18 reasoning datasets, multiple languages, and multimodal tasks demonstrate SoT’s broad applicability.The conclusion frames the evidence as spanning diverse evaluation settings.
  • 6 Conclusion: SoT is compatible with ensemble prompting strategies, reinforcing its practical utility in resource-constrained settings.The conclusion identifies this compatibility as an additional practical benefit.
  • 6 Conclusion: SoT reframes efficiency as a reasoning-design challenge rather than solely a surface-level compression problem.This framing motivates further work on scalable, cognitively informed prompting.

Limitations and Future Work

The paper identifies limits in SoT’s current fixed-paradigm exemplars, three-paradigm scope, and multilingual evaluation coverage. Future work could improve adaptability through retrieval, add paradigms for downstream tasks, and evaluate low-resource languages.

  • Limitations and Future Work: Fixed exemplars preserve stylistic consistency and interpretability but may limit adaptability to subtle variations within a task type.Retrieving exemplars dynamically from a larger pool is proposed as a way to improve flexibility.
  • Limitations and Future Work: The current evaluation focuses on three cognitively grounded, prompt-based paradigms, although the framework can incorporate additional paradigms.Adding paradigms requires updating the sketching pool and retraining the routing module.
  • Limitations and Future Work: Additional reasoning paradigms could help adapt SoT to downstream tasks such as code generation.The paper presents this as a future direction rather than a demonstrated result.
  • Limitations and Future Work: The multilingual experiments cover widely spoken languages, leaving evaluation in low-resource languages as future work.The paper characterizes low-resource-language evaluation as an open direction.

Ethics Statement

The ethics statement describes the use of public datasets and licensed models, alongside a caution that compressed reasoning may reduce interpretability in high-stakes applications. It also notes that the router’s GPT-4o-generated annotations may contain underlying-model biases.

  • Ethics Statement: All datasets used are publicly available, cited, and used according to their intended uses and licensing terms.The statement also specifies subsets where applicable and identifies Hugging Face access for the datasets.
  • Ethics Statement: All models are used in accordance with their respective licenses.This applies to both the general model-use statement and the implementation details for open- and closed-weight models.
  • Ethics Statement: Compressing intermediate outputs may affect interpretability in high-stakes settings such as healthcare or legal analysis.The paper recommends caution where full transparency of reasoning steps may be essential.
  • Ethics Statement: The router was trained with GPT-4o-generated annotations and may reflect biases present in that underlying model.The paper recommends further evaluation before deployment in sensitive or high-stakes settings.
  • Ethics Statement: Experiments used instruction-tuned checkpoints without parameter fine-tuning or additional adaptation of the language models.The implementation used bfloat16 where supported and FlashAttention2 for inference acceleration.

A.4 Reproducibility

The reproducibility setup standardizes prompts, exemplars, decoding, tokenization, formatting, and evaluation procedures, while documenting router design and labeling validation. Explicit formatting instructions were necessary to achieve near-zero malformed outputs.

  • Experimental Controls: All experiments used random seed 42, temperature 0.5, fixed few-shot exemplars, and each model checkpoint’s default tokenizer for token counts.
  • Prompt Format: Each paradigm prompt contains Role & Objective, Application Steps, Rules & Directives, and Closing Statement sections.The sections provide cognitive motivation, procedures, output constraints, and reminders for concise structured reasoning.
  • Formatting Compliance: Explicit formatting instructions reduced malformed or noncompliant outputs to near zero across paradigms and model variants.Exemplars alone often failed to enforce structural tags and the expected output format.
  • Exemplar Construction: SoT exemplars were separately constructed for each paradigm, with manually selected questions representing typical target use cases.Candidate responses were first generated with Qwen-2.5-32B before manual selection.
  • Exemplar Construction: Baseline exemplars used the same procedure and drew from the three reasoning categories represented in SoT to support fair coverage.One exemplar was selected per category for CoT, CCoT, and CoD.
  • Router Validation: GPT-4o-generated router labels showed substantial agreement with human-majority labels, with Cohen’s k = 0.95 and inter-human agreement k = 0.72.The human study used 30 samples and nine annotators.

C.3 Routing Distribution by Dataset

The router’s dominant paradigm matched the expected paradigm in all 15 analyzed datasets, with task domains determining the prevailing reasoning style. Ambiguous cases showed limited overlap and conservative fallback toward general-purpose reasoning.

  • Overall Distribution: 100% agreement was observed between dominant and expected paradigms across all 15 datasets.The distribution analysis covered 150 samples per dataset from the primary experiments.
  • Dataset Patterns: Conceptual Chaining dominated commonsense, logical, and multi-hop datasets where relational inference was critical.Examples include StrategyQA, HotPotQA, and Reclor.
  • Dataset Patterns: Chunked Symbolism was used exclusively for mathematical tasks, where symbolic notation provided the clearest compression benefit.Examples include GSM8K, AQUA, and SVAMP; DROP showed some cross-paradigm routing.
  • Dataset Patterns: Expert Lexicons was most common in domain-specific datasets such as PubMedQA and MedQA.Occasional Conceptual Chaining assignments reflected conservative fallback when confidence was low.
  • Examples: The router’s expected paradigm assignments were illustrated with representative examples from OpenbookQA, GSM8K, and PubMedQA.These examples pair each query with its assigned paradigm.
  • Routing Consequences: Correct paradigm selection consistently yielded the highest accuracy, while routing errors primarily affected accuracy rather than token efficiency.Output token usage remained relatively stable across paradigms because prompts and exemplar lengths were fixed.

D.1 Per-Dataset Results

The paper reports per-dataset accuracy and token counts across prompting and ensemble-style methods, including SoT within Multi-Agent Debate, Self-Consistency, and Self-Refine settings. In debate experiments, SoT produced nearly the same number of rounds per query as CoT.

  • Primary Results: Per-dataset accuracy and token counts were averaged across three runs for the primary experiments.Full results were reported across model families and prompting strategies.
  • Multi-Agent Debate: SoT was incorporated into Multi-Agent Debate while preserving paradigm-specific prompts and adding prompts for iterative cross-agent revision.The setup used multiple agents answering and revising responses over debate rounds.
  • Multi-Agent Debate: Each debate run used three agents and at most three rounds, terminating early on consensus or using majority vote otherwise.
  • Multi-Agent Debate: 1.11 average rounds-per-query for SoT was similar to CoT’s 1.14 in the debate experiments.The paper reports that SoT did not notably affect the number of debate rounds.
  • Comparison Methods: Self-Consistency generated three reasoning outputs per sample and selected the most frequent answer, with random selection when all three differed.
  • Comparison Methods: Self-Refine applied critique and refinement prompts to an initial response generated with the router-selected SoT paradigm.The refinement loop asked the model to revise its concise, structured reasoning using the critique.

E Output Examples

SoT outputs use compact, paradigm-specific structures rather than shortened natural-language chains, with Conceptual Chaining linking concepts, Chunked Symbolism organizing calculations, and Expert Lexicons using technical shorthand. Representative traces show shorter, logically structured responses than baseline prompting.

  • Comparative Traces: SoT responses are more compact than CoT, CoD, and CCoT while maintaining logical structure and semantic completeness.CoD and CCoT shorten natural-language reasoning but remain natural-language approaches.
  • Chunked Symbolism: A Chunked Symbolism example computes Lily’s 50 friends, Amy’s 70 friends, and their total of 120 through explicit variable assignments and arithmetic.
  • Conceptual Chaining: A Conceptual Chaining example represents a commonsense answer as green_parts → chloroplasts → absorb_light → photosynthesis, yielding option B.
  • Conceptual Chaining: Conceptual Chaining links essential concepts in ordered steps using minimal words, especially for commonsense, multi-hop, and fact-based reasoning.Its prompts emphasize extracting key concepts and avoiding full sentences.
  • Chunked Symbolism: Chunked Symbolism represents numerical reasoning with variables, equations, and small computational steps while minimizing words.The paradigm is motivated by chunking and targets mathematical, symbolic, and technical calculations.
  • Expert Lexicons: Expert Lexicons compresses reasoning into high-information expressions using domain-specific shorthand, technical symbols, and field-specific abbreviations.The paradigm targets technical, symbolic, and formulaic reasoning where few-token communication is useful.
Loading 2503.05179v4…