Source-linked AI summary
C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
Yu Kang, Xianghui Sun, Liangyu Chen, Wei Zou
TL;DR
Long CoT can improve LLM reasoning but adds decoding cost, while naive shortening can reduce effectiveness. C3oT compresses CoT, trains on both lengths with conditions, and generates shorter CoT at inference, matching longer-CoT performance across four datasets while reducing length by more than 50%.
Problem
CoT improves LLM reasoning but often generates longer-than-needed intermediate text, increasing decoding costs in latency-sensitive applications; shortening it can diminish reasoning ability.
Method
C3oT compresses longer CoT into shorter interpretable CoT, trains LLMs on both forms using distinct conditions, and uses the short condition during inference.
Results
C3oT performs on par with longer-CoT training across four arithmetic and commonsense datasets while significantly shortening generated CoT; the paper reports compression by up to more than 50%.
Takeaways & Limitations
C3oT provides a way to reduce generated-CoT length and inference cost without compromising effectiveness within the evaluated reasoning tasks.
Takeaways & Limitations
C3oT requires compressed training CoT to retain sufficient information and interpretability; excessive compression can reduce performance.
Abstract
from arXiv · showhide
Generating Chain-of-Thought (CoT) before deriving the answer can effectively improve the reasoning capabilities of large language models (LLMs) and significantly improve the accuracy of the generated answer. However, in most cases, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs. Furthermore, existing research has discovered that shortening the reasoning steps in CoT, even while preserving the key information, diminishes LLMs' abilities. These phenomena make it difficult to use LLMs and CoT in many real-world applications that only require the final answer and are sensitive to latency, such as search and recommendation. To reduce the costs of model decoding and shorten the length of the generated CoT, this paper presents $\textbf{C}$onditioned $\textbf{C}$ompressed $\textbf{C}$hain-of-$\textbf{T}$hought (C3oT), a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT while maintaining key information and interpretability, a conditioned training method to train LLMs with both longer CoT and shorter CoT simultaneously to learn the corresponding relationships between them, and a conditioned inference method to gain the reasoning ability learned from longer CoT by generating shorter CoT. We conduct experiments over four datasets from arithmetic and commonsense scenarios, showing that the proposed method is capable of compressing the length of generated CoT by up to more than 50% without compromising its effectiveness.
Introduction
C3oT addresses the tension between CoT’s reasoning benefits and the decoding costs of lengthy intermediate reasoning. It compresses CoT while using conditioned training and inference to preserve effectiveness.
- Motivation: CoT improves LLM reasoning, but lengthy intermediate reasoning increases decoding costs in latency-sensitive applications such as search and recommendation.Models trained with reasoning processes, explanation traces, and step-by-step thoughts generally perform better.
- Motivation: Shortening CoT can significantly diminish reasoning ability even when key information is preserved.Prior work links higher reasoning complexity and more reasoning steps with better performance on multi-step reasoning tasks.
- C3oT: C3oT compresses longer CoT into shorter, interpretable CoT, trains on both forms with distinct conditions, and uses conditioned inference to generate shorter reasoning.The framework aims to retain essential information while learning the relationship between longer and shorter CoT.
- Evaluation: Experiments cover four arithmetic and commonsense datasets, with performance on par with longer-CoT training while generated CoT is significantly shorter.The datasets are GSM8K, MathQA, ECQA, and StrategyQA.
- Contributions: C3oT is presented as a framework for drastically shortening generated CoT without loss of effectiveness, supported by experiments, ablations, and extension analyses.The contribution claims reduced model inference cost and broad evaluation across math and commonsense reasoning tasks.
Related Work
Prior work shows that longer CoT generally improves reasoning but leaves output decoding costly. C3oT instead targets shorter generated CoT while preserving effectiveness.
- Inference acceleration: Inference output decoding can cost more than input processing, making the additional tokens generated by CoT consequential for LLM inference cost.Accelerating CoT generation has received less attention than reducing input-token processing.
- Prior approaches: Implicit-CoT avoids explicit CoT generation through hidden-state reasoning but significantly sacrifices performance relative to explicit CoT.It is described as the main prior attempt to accelerate CoT generation.
- Positioning: C3oT uses conditioned training and inference to generate shorter CoT while retaining reasoning capabilities learned from longer CoT.Quantization and pruning are described as orthogonal techniques that can be combined with this approach.
- CoT length: Existing studies consistently find that lengthening intermediate CoT reasoning enhances LLM capabilities across multiple tasks.Higher reasoning complexity, characterized by more reasoning steps, improves multi-step reasoning performance.
- C3oT: C3oT differs by compressing generated CoT while aiming to preserve its effectiveness.The method addresses the relationship between CoT length and effectiveness rather than eliminating explicit reasoning entirely.
Method
C3oT constructs paired longer and compressed CoT examples, trains an LLM with length-specific conditions, and invokes the short condition at inference. The compressor preserves key information and interpretability while removing redundancy.
- Problem Statement: A compressor F maps each detailed reasoning trace r_long to a shorter trace r_short that retains key information.The training objective uses the compressed and original traces to align answers derived from both forms.
- Compressor: GPT-4 serves as the CoT compressor, producing condensed reasoning traces intended to preserve core information and interpretability.The paper also examines different compressor models in its analysis.
- Conditioned Training: Each instruction is represented with both longer and shorter CoT, distinguished by separate conditions in a class-conditioned dataset.The shorter and longer traces are controlled by condition c_i and need not appear as paired samples during fine-tuning.
- Conditioned Training: Distinct prompts request either a detailed thought process or the briefest possible thought process, creating long- and short-CoT training examples.The sample transformation changes the rationale from a worked arithmetic explanation to a concise answer rationale.
- Conditioned Training: Conditioned training is implemented as ordinary supervised fine-tuning after randomly shuffling the class-conditioned dataset.The method does not require an additional mechanism to explicitly inform the LLM how corresponding traces relate.
- Conditioned Inference: At inference, the short-CoT prompt condition is reused so the trained model generates shorter reasoning while deriving the final answer efficiently.The paper’s inference prompt explicitly asks for as brief a thought process as possible.
Experiment
C3oT is evaluated across four reasoning datasets using accuracy and compression rate, with comparisons against longer and shorter CoT baselines. Results show that shortening CoT alone reduces effectiveness, whereas C3oT remains competitive while reducing generated-CoT length.
- Evaluation Setup: Four datasets span mathematical reasoning (GSM8K and MathQA) and commonsense reasoning (ECQA and StrategyQA).The evaluation follows the datasets’ original splits except StrategyQA, whose training set was further split because test labels were inaccessible.
- Evaluation Metrics: Accuracy uses exact match, while compression rate measures the reduction in generated-CoT length relative to Long CoT.A larger compression rate indicates lower inference cost; a rate of 1 corresponds to generating no intermediate CoT.
- Main Results: C3oT consistently outperforms Implicit-CoT in accuracy across GSM8K, MathQA, ECQA, and StrategyQA, while Implicit-CoT reaches 100% compression.Implicit-CoT avoids explicitly generating CoT but significantly sacrifices performance.
- Main Results: Short CoT significantly diminishes effectiveness compared with Long CoT, even when GPT-4 preserves key information and interpretability during compression.This result is consistent with previous findings that shortening reasoning steps can reduce reasoning ability.
- Main Results: C3oT achieves similar or better performance than Long CoT across datasets, with a slight mathematical-task lag and stronger performance on commonsense tasks.The difference is related to the reasoning ability required by each task.
- Compression Analysis: Compression rate depends mainly on the dataset rather than the task domain when the compressor and preservation requirements remain fixed.Datasets with more detailed and redundant original CoT permit higher achievable compression rates.
Analysis
The analysis tests C3oT across compressor choices, expanded reasoning traces, training compression rates, mixed conditions, and adaptive compression. Results show that conditioned training remains effective when compressed traces preserve sufficient information, while excessive compression limits performance.
- Class-conditioned policy: C3oT outperforms the unconditioned ablation in both accuracy and compression rate across all datasets.Without distinct conditions, models cannot reliably distinguish longer from shorter CoT and become confused by their differing lengths.
- Compressor comparison: LLaMA-2 compressors cause a minor, nonsignificant accuracy decrease and compression rates over 15% lower than GPT-4 compressors.The results suggest stronger compressors produce more concise compressed CoT while preserving key information remains comparatively easy.
- Expanded CoT: C3oTExpansion improves over C3oT while maintaining a similar compression rate, although expanding CoT also improves the Long CoT baseline.The expanded-CoT experiment uses five reasoning-step expansion methods and combines expanded and compressed traces for conditioned training.
- Compression-rate analysis: About 50% compression is used for GPT-4-compressed GSM8K training CoT, but C3oT accuracy decreases as training-set compression increases.Figure 2 compares accuracy against training-set compression or the corresponding inference condition on GSM8K.
- Compression-rate analysis: C3oT approaches Long CoT only when compressed training CoT retains sufficient key information and interpretability; over-compression causes performance decline.Highly compressed traces can lose grammatical structure and become uninterpretable, reducing C3oT’s advantage over Short CoT.
- Mixed conditions: Mixed Conditions outperform C3oT across training-set compression rates and surpass Long CoT at 50% compression, while enabling multiple inference compression levels.Different initial prompt tokens condition shorter CoT at distinct compression levels.
- Adaptive compression: C3oTAdapt significantly outperforms C3oTGPT-4 in both accuracy and compression rate by learning to select efficient CoT lengths for varying question complexity.The appropriate compression rate is defined as the highest rate at which the model still answers accurately.
Conclusion
C3oT compresses longer Chain-of-Thought into shorter reasoning while preserving effectiveness, using a compressor, conditioned training, and conditioned inference. Evaluations across arithmetic and commonsense datasets show comparable performance with substantially shorter generated CoT.
- C3oT combines a compressor, conditioned training, and conditioned inference to shorten generated Chain-of-Thought.The framework is based on a class-conditioned policy.
- Across four arithmetic and commonsense datasets, C3oT performs on par with models trained using original longer CoT.
- C3oT significantly shortens generated CoT, supporting application of reasoning-capable models in time-sensitive real-world settings.
Prompt Templates
The prompt templates define formats for simplifying, expanding, and compressing thought processes while preserving or adding specified reasoning content. They also include templates for questions, original thought processes, answers, and strategy instructions.
- Simplification: The templates pair a question with an original thought process and final answer before requesting a simplified thought process.
- Simplification: A second simplification template requests shortening the thought process as much as possible while retaining only information needed to solve the question.
- Compression: A separate heading introduces compression prompts with a specified compression rate, followed by question and instruction placeholders.
- Expansion: The expansion template instructs the model to expand the original thought process without removing its content.
- Expansion: Listed expansion strategies include interpreting important question words, rereading the question, repeating the current state, self-verification, and making equations.
Dataset Details
The passage introduces GSM8K as a dataset for evaluating arithmetic reasoning and mathematical step formulation, and begins describing its CoT token statistics.
- GSM8K: GSM8K contains 8,792 math reasoning problems, with 7,473 training examples and 1,319 test examples.
- GSM8K: The dataset evaluates arithmetic reasoning and the formulation of mathematical steps using language.
- CoT Statistics: The training set’s original CoT averages 124 tokens, while the passage begins reporting a shorter average after GPT-4 compression.
Cases Study
The section presents figures documenting cases involving different compressors, CoT expansion, and different compression rates.
- Different Compressors: Figures 3 and 4 present cases of different compressors.
- CoT Expansion: Figures 5 and 6 present cases of CoT expansion.
- Compression Rates: Figures 7 and 8 present cases of different compression rates.