PaperlayerView exact revision on arXiv

Source-linked AI summary

C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

Yu Kang, Xianghui Sun, Liangyu Chen, Wei Zou

arXiv:2412.11664v1cs.CLcs.LG

TL;DR

Long CoT can improve LLM reasoning but adds decoding cost, while naive shortening can reduce effectiveness. C3oT compresses CoT, trains on both lengths with conditions, and generates shorter CoT at inference, matching longer-CoT performance across four datasets while reducing length by more than 50%.

Abstract

from arXiv · show

Generating Chain-of-Thought (CoT) before deriving the answer can effectively improve the reasoning capabilities of large language models (LLMs) and significantly improve the accuracy of the generated answer. However, in most cases, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs. Furthermore, existing research has discovered that shortening the reasoning steps in CoT, even while preserving the key information, diminishes LLMs' abilities. These phenomena make it difficult to use LLMs and CoT in many real-world applications that only require the final answer and are sensitive to latency, such as search and recommendation. To reduce the costs of model decoding and shorten the length of the generated CoT, this paper presents $\textbf{C}$onditioned $\textbf{C}$ompressed $\textbf{C}$hain-of-$\textbf{T}$hought (C3oT), a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT while maintaining key information and interpretability, a conditioned training method to train LLMs with both longer CoT and shorter CoT simultaneously to learn the corresponding relationships between them, and a conditioned inference method to gain the reasoning ability learned from longer CoT by generating shorter CoT. We conduct experiments over four datasets from arithmetic and commonsense scenarios, showing that the proposed method is capable of compressing the length of generated CoT by up to more than 50% without compromising its effectiveness.

Introduction

C3oT addresses the tension between CoT’s reasoning benefits and the decoding costs of lengthy intermediate reasoning. It compresses CoT while using conditioned training and inference to preserve effectiveness.

Motivation

  • CoT improves LLM reasoning, but lengthy intermediate reasoning increases decoding costs in latency-sensitive applications such as search and recommendation.Models trained with reasoning processes, explanation traces, and step-by-step thoughts generally perform better.Source: Introduction · p. 1
  • Shortening CoT can significantly diminish reasoning ability even when key information is preserved.Prior work links higher reasoning complexity and more reasoning steps with better performance on multi-step reasoning tasks.Source: Introduction · p. 1

C3oT

  • C3oT compresses longer CoT into shorter, interpretable CoT, trains on both forms with distinct conditions, and uses conditioned inference to generate shorter reasoning.The framework aims to retain essential information while learning the relationship between longer and shorter CoT.Source: Introduction · pp. 1–2

Evaluation

  • Experiments cover four arithmetic and commonsense datasets, with performance on par with longer-CoT training while generated CoT is significantly shorter.The datasets are GSM8K, MathQA, ECQA, and StrategyQA.Source: Introduction · p. 2

Contributions

  • C3oT is presented as a framework for drastically shortening generated CoT without loss of effectiveness, supported by experiments, ablations, and extension analyses.The contribution claims reduced model inference cost and broad evaluation across math and commonsense reasoning tasks.Source: Introduction · p. 2

Related Work

Prior work shows that longer CoT generally improves reasoning but leaves output decoding costly. C3oT instead targets shorter generated CoT while preserving effectiveness.

Inference acceleration

  • Inference output decoding can cost more than input processing, making the additional tokens generated by CoT consequential for LLM inference cost.Accelerating CoT generation has received less attention than reducing input-token processing.Source: LLMs Inference Acceleration · p. 2

Prior approaches

  • Implicit-CoT avoids explicit CoT generation through hidden-state reasoning but significantly sacrifices performance relative to explicit CoT.It is described as the main prior attempt to accelerate CoT generation.Source: LLMs Inference Acceleration · p. 2

Positioning

  • C3oT uses conditioned training and inference to generate shorter CoT while retaining reasoning capabilities learned from longer CoT.Quantization and pruning are described as orthogonal techniques that can be combined with this approach.Source: LLMs Inference Acceleration · p. 2

CoT length

  • Existing studies consistently find that lengthening intermediate CoT reasoning enhances LLM capabilities across multiple tasks.Higher reasoning complexity, characterized by more reasoning steps, improves multi-step reasoning performance.Source: Chain-of-Thought Analyzing · p. 2

C3oT

  • C3oT differs by compressing generated CoT while aiming to preserve its effectiveness.The method addresses the relationship between CoT length and effectiveness rather than eliminating explicit reasoning entirely.Source: Chain-of-Thought Analyzing · p. 2

Method

C3oT constructs paired longer and compressed CoT examples, trains an LLM with length-specific conditions, and invokes the short condition at inference. The compressor preserves key information and interpretability while removing redundancy.

Problem Statement

  • A compressor F maps each detailed reasoning trace r_long to a shorter trace r_short that retains key information.The training objective uses the compressed and original traces to align answers derived from both forms.Source: Problem Statement · p. 3

Compressor

  • GPT-4 serves as the CoT compressor, producing condensed reasoning traces intended to preserve core information and interpretability.The paper also examines different compressor models in its analysis.Source: Conditioned Compressed… · p. 3

Conditioned Training

  • Each instruction is represented with both longer and shorter CoT, distinguished by separate conditions in a class-conditioned dataset.The shorter and longer traces are controlled by condition c_i and need not appear as paired samples during fine-tuning.Source: Conditioned Compressed… · p. 3
  • Distinct prompts request either a detailed thought process or the briefest possible thought process, creating long- and short-CoT training examples.The sample transformation changes the rationale from a worked arithmetic explanation to a concise answer rationale.Source: Conditioned Compressed… · p. 3
  • Conditioned training is implemented as ordinary supervised fine-tuning after randomly shuffling the class-conditioned dataset.The method does not require an additional mechanism to explicitly inform the LLM how corresponding traces relate.Source: Conditioned Compressed… · p. 3

Conditioned Inference

  • At inference, the short-CoT prompt condition is reused so the trained model generates shorter reasoning while deriving the final answer efficiently.The paper’s inference prompt explicitly asks for as brief a thought process as possible.Source: Conditioned Compressed… · p. 3

Experiment

C3oT is evaluated across four reasoning datasets using accuracy and compression rate, with comparisons against longer and shorter CoT baselines. Results show that shortening CoT alone reduces effectiveness, whereas C3oT remains competitive while reducing generated-CoT length.

Evaluation Setup

  • Four datasets span mathematical reasoning (GSM8K and MathQA) and commonsense reasoning (ECQA and StrategyQA).The evaluation follows the datasets’ original splits except StrategyQA, whose training set was further split because test labels were inaccessible.Source: Settings · p. 3

Evaluation Metrics

  • Accuracy uses exact match, while compression rate measures the reduction in generated-CoT length relative to Long CoT.A larger compression rate indicates lower inference cost; a rate of 1 corresponds to generating no intermediate CoT.Source: Settings · p. 4

Main Results

  • C3oT consistently outperforms Implicit-CoT in accuracy across GSM8K, MathQA, ECQA, and StrategyQA, while Implicit-CoT reaches 100% compression.Implicit-CoT avoids explicitly generating CoT but significantly sacrifices performance.Source: Main Results · p. 4
  • Short CoT significantly diminishes effectiveness compared with Long CoT, even when GPT-4 preserves key information and interpretability during compression.This result is consistent with previous findings that shortening reasoning steps can reduce reasoning ability.Source: Main Results · p. 4
  • C3oT achieves similar or better performance than Long CoT across datasets, with a slight mathematical-task lag and stronger performance on commonsense tasks.The difference is related to the reasoning ability required by each task.Source: Main Results · pp. 4–5

Compression Analysis

  • Compression rate depends mainly on the dataset rather than the task domain when the compressor and preservation requirements remain fixed.Datasets with more detailed and redundant original CoT permit higher achievable compression rates.Source: Main Results · p. 5

Analysis

The analysis tests C3oT across compressor choices, expanded reasoning traces, training compression rates, mixed conditions, and adaptive compression. Results show that conditioned training remains effective when compressed traces preserve sufficient information, while excessive compression limits performance.

Class-conditioned policy

  • C3oT outperforms the unconditioned ablation in both accuracy and compression rate across all datasets.Without distinct conditions, models cannot reliably distinguish longer from shorter CoT and become confused by their differing lengths.Source: What is the contribution of the… · p. 5

Compressor comparison

  • LLaMA-2 compressors cause a minor, nonsignificant accuracy decrease and compression rates over 15% lower than GPT-4 compressors.The results suggest stronger compressors produce more concise compressed CoT while preserving key information remains comparatively easy.Source: What is the impact of different… · p. 5

Expanded CoT

  • C3oTExpansion improves over C3oT while maintaining a similar compression rate, although expanding CoT also improves the Long CoT baseline.The expanded-CoT experiment uses five reasoning-step expansion methods and combines expanded and compressed traces for conditioned training.Source: Is C3oT effective for expanded CoT? · pp. 5–6

Compression-rate analysis

  • About 50% compression is used for GPT-4-compressed GSM8K training CoT, but C3oT accuracy decreases as training-set compression increases.Figure 2 compares accuracy against training-set compression or the corresponding inference condition on GSM8K.Source: What is the impact of training sets… · p. 6, Figure 2 · p. 6
  • C3oT approaches Long CoT only when compressed training CoT retains sufficient key information and interpretability; over-compression causes performance decline.Highly compressed traces can lose grammatical structure and become uninterpretable, reducing C3oT’s advantage over Short CoT.Source: What is the impact of training sets… · pp. 6–7

Mixed conditions

  • Mixed Conditions outperform C3oT across training-set compression rates and surpass Long CoT at 50% compression, while enabling multiple inference compression levels.Different initial prompt tokens condition shorter CoT at distinct compression levels.Source: What is the impact of mixed conditions… · p. 7

Adaptive compression

  • C3oTAdapt significantly outperforms C3oTGPT-4 in both accuracy and compression rate by learning to select efficient CoT lengths for varying question complexity.The appropriate compression rate is defined as the highest rate at which the model still answers accurately.Source: Can C3oT select the appropriate… · p. 7

Conclusion

C3oT compresses longer Chain-of-Thought into shorter reasoning while preserving effectiveness, using a compressor, conditioned training, and conditioned inference. Evaluations across arithmetic and commonsense datasets show comparable performance with substantially shorter generated CoT.

  • C3oT combines a compressor, conditioned training, and conditioned inference to shorten generated Chain-of-Thought.The framework is based on a class-conditioned policy.Source: Conclusion · p. 7
  • Across four arithmetic and commonsense datasets, C3oT performs on par with models trained using original longer CoT.Source: Conclusion · p. 7
  • C3oT significantly shortens generated CoT, supporting application of reasoning-capable models in time-sensitive real-world settings.Source: Conclusion · p. 7

Prompt Templates

The prompt templates define formats for simplifying, expanding, and compressing thought processes while preserving or adding specified reasoning content. They also include templates for questions, original thought processes, answers, and strategy instructions.

Simplification

  • The templates pair a question with an original thought process and final answer before requesting a simplified thought process.Source: Prompt Templates · p. 10
  • A second simplification template requests shortening the thought process as much as possible while retaining only information needed to solve the question.Source: Prompt Templates · p. 10

Compression

  • A separate heading introduces compression prompts with a specified compression rate, followed by question and instruction placeholders.Source: Prompt Templates · p. 10

Expansion

  • The expansion template instructs the model to expand the original thought process without removing its content.Source: Prompt Templates · p. 10
  • Listed expansion strategies include interpreting important question words, rereading the question, repeating the current state, self-verification, and making equations.Source: Prompt Templates · p. 10

Dataset Details

The passage introduces GSM8K as a dataset for evaluating arithmetic reasoning and mathematical step formulation, and begins describing its CoT token statistics.

GSM8K

  • GSM8K contains 8,792 math reasoning problems, with 7,473 training examples and 1,319 test examples.Source: Dataset Details · p. 11
  • The dataset evaluates arithmetic reasoning and the formulation of mathematical steps using language.Source: Dataset Details · p. 11

CoT Statistics

  • The training set’s original CoT averages 124 tokens, while the passage begins reporting a shorter average after GPT-4 compression.Source: Dataset Details · p. 11

Cases Study

The section presents figures documenting cases involving different compressors, CoT expansion, and different compression rates.

Different Compressors

CoT Expansion

Compression Rates

Related papers

Loading 2412.11664v1…