Source-linked AI summary
Compressed Chain of Thought: Efficient Reasoning Through Dense Representations
Jeffrey Cheng, Benjamin Van Durme
TL;DR
Explicit CoT reasoning can improve language-model performance but has high generation latency. CCoT instead generates variable-length, contentful continuous tokens that compress reasoning chains, and experiments show accuracy gains with controllable efficiency trade-offs.
Problem
CoT improves reasoning but requires long discrete-token generations, creating a high-latency inference trade-off.
Method
CCoT generates variable-length continuous contemplation tokens that compress language-based reasoning chains and can be applied to pretrained decoder-only LLMs.
Results
With r = 0.10, CCoT improved accuracy by 9 points over the no-contemplation baseline for a 0.4-second generation-time increase; with r = 0.05, improvement remained 6 points for around 0.15 seconds.
Takeaways & Limitations
CCoT provides contentful contemplation tokens as an alternative to explicit reasoning chains while allowing performance-efficiency adjustment through the compression ratio.
Takeaways & Limitations
The framework assumes a pretrained causal decoder-only language model with trained CCOTφ, DECODEψ, and end-predictor modules.
Abstract
from arXiv · showhide
Chain-of-thought (CoT) decoding enables language models to improve reasoning performance at the cost of high generation latency in decoding. Recent proposals have explored variants of contemplation tokens, a term we introduce that refers to special tokens used during inference to allow for extra computation. Prior work has considered fixed-length sequences drawn from a discrete set of embeddings as contemplation tokens. Here we propose Compressed Chain-of-Thought (CCoT), a framework to generate contentful and continuous contemplation tokens of variable sequence length. The generated contemplation tokens are compressed representations of explicit reasoning chains, and our method can be applied to off-the-shelf decoder language models. Through experiments, we illustrate how CCoT enables additional reasoning over dense contentful representations to achieve corresponding improvements in accuracy. Moreover, the reasoning improvements can be adaptively modified on demand by controlling the number of contemplation tokens generated.
1. Introduction
CoT improves language-model reasoning but incurs high generation latency. CCoT addresses this trade-off by generating compressed contemplation tokens and supports adaptive performance-efficiency control.
- CoT decomposes complex questions into sequential reasoning steps, improving reasoning capabilities across tasks but increasing generation latency.GPT-4o took 21.37 seconds with CoT versus 2.81 seconds without it for the same answer.
- Prior contemplation-token methods introduce additional inference-time computation through shorter token sequences instead of fully explicit reasoning chains.These tokens may be contentful or noncontentful and provide online memory for additional computation.
- CCoT generates contentful contemplation tokens as compressed representations of language-based reasoning chains.The framework trains against gold hidden states from full reasoning traces.
- CCoT adapts pretrained decoder-only language models through LoRA finetuning and variable compression ratios.Controlling the number of generated tokens adjusts the performance-efficiency trade-off during inference.
- The paper evaluates pretrained decoder-only LLMs on GSM8K for performance and throughput and studies their relation to prior contemplation-token methods.It also extends theoretical results concerning the computational capacity of CCoT contemplation tokens.
2. Related Work
Prior work explores extra inference-time computation through discrete or continuous tokens, parallel or autoregressive generation, and contextual compression. CCoT differs by autoregressively generating contentful continuous representations of reasoning chains and unknown text.
- Distillation of Knowledge Chains: Reasoning-chain distillation compresses explicit computations into hidden states, while CCoT grounds its generated continuous tokens in text.The grounding could support later decoding of the compressed reasoning chain for human inspection.
- Chain of Thought: CoT uses discrete language tokens for sequential reasoning, producing long sequences with substantial generation costs.Figure 1 contrasts this with CCoT’s shorter sequence of continuous embeddings.
- Distillation of Knowledge Chains: CCoT differs from related latent-token work by adapting a 7B model with approximately 9,000 GSM8K instances versus an unreleased augmented dataset of approximately 400,000 instances for a 1.5B model.The paper presents this contrast as evidence suggesting better scaling and data efficiency.
- Filler (Pause) Tokens: Prior contemplation-token methods mainly use noncontentful tokens that can be decoded in parallel, increasing computational width during inference.These methods include pause, memory, filler, and thinking tokens.
- Filler (Pause) Tokens: CCoT tokens are contentful compressed reasoning representations decoded autoregressively, providing computational depth as well as width.The cited discussion contrasts CCoT with noncontentful parallel filler tokens.
- Contextual Compression: Context compression encodes known context into representations that are attended to but not generated, whereas CCoT autoregressively generates representations of a priori unknown content.The paper describes contextual compression as orthogonal to contemplation tokens.
- Chain of Thought: CoT originated as prompting with hand-crafted demonstrations and was later elicited zero-shot through instructions to think step by step.Related innovations target both CoT efficiency and performance.
- Comparison of Contemplation Tokens: Table 1 compares contemplation-token methods by contentfulness, discrete-versus-continuous format, inference generation, and additional notes.These dimensions organize distinctions across the related methods.
3. Contemplation Tokens
Decoder-only language models process queries through Transformer blocks and generate answers autoregressively. Contemplation tokens add inference-time computation; CCoT compresses reasoning-chain representations into shorter contentful sequences that can preserve downstream performance while reducing answer-decoding length.
- Preliminaries and Notation: A decoder-only language model embeds an input sequence, applies Transformer blocks, and uses the final hidden states to generate the next-token distribution.The model can compute hidden states under different weights for concatenated embedding sequences.
- Contemplation Tokens: Contemplation tokens are additional inference-time tokens that provide memory and computation before the model decodes an answer.Given query q, the model can generate tokens t and attend to [q; t] instead of attending only to q.
- Contemplation Tokens: Contentful contemplation tokens are semantically meaningful tokens or hidden states derived from semantically meaningful tokens, unlike filler tokens.CoT reasoning chains exemplify contentful tokens, whereas filler tokens are noncontentful.
- Motivation: Explicit CoT improves reasoning but adds autoregressive decoding cost because every reasoning-chain token requires an extra model pass.The reasoning chain has m tokens, so generating it incurs the cost of m additional passes.
- Compressing Reasoning Chains: Selecting a shortened subset of reasoning-chain hidden states can preserve downstream performance, motivating direct generation of compressed representations.Conditioning on [x_1:n; z_1:k] is reported to yield lossless downstream performance, with k much smaller than the original chain length m.
- Compressing Reasoning Chains: CCoT directly generates compressed reasoning representations rather than first decoding the full chain, reducing the answer-stage sequence length without retaining the chain-generation cost.The proposed module generates z directly as contentful representations of reasoning chains.
4. Approach
CCoT generates compressed contemplation tokens by approximating selected hidden states from full reasoning chains, then conditions answer decoding on them. Its inference procedure uses continuous intermediate-layer states, variable-length autoregressive generation, and a separately trained decoder.
- 4. Approach: CCoT trains modules that generate contemplation tokens and decode answers conditioned on the query and those tokens.Both modules are initialized from the pretrained language model; CCOT generates the contemplation sequence, while DECODE produces the answer.
- 4. Approach: A compression ratio r sets k = ⌈r · m⌉ contemplation tokens, interpolating between full reasoning chains at r = 1 and answer-only training at r = 0.Here m is the reasoning-chain length and k is the compressed sequence length.
- 4. Approach: CCoT approximates a size-k subset of precomputed hidden states from the full reasoning chain rather than generating the entire chain.A scorer selects the hidden-state subset, which supplies gold labels for training the compressed-token generator.
- 4. Approach: The generator uses an intermediate hidden layer of the previous contemplation token as the next continuous input, preserving autoregressive computational depth without discrete token decoding.The initial input is the corresponding intermediate hidden state of the query’s last token.
- 4. Approach: CCOTφ is trained layer by layer with scaled mean squared error, while later decoder finetuning conditions answer prediction on generated contemplation tokens.Unfreezing all CCOT parameters during decoder training reduced downstream performance, whereas unfreezing layers after the autoregressive layer improved it.
- 4. Approach: At inference, an end classifier determines when to stop generating contemplation tokens, with a maximum of h = 200r tokens.The stated cap would prematurely terminate less than 3% of the long-tailed reasoning-chain distribution.
5. Experiments
The experiments evaluate CCOT on GSM8K against no-contemplation, full-reasoning, and PAUSE baselines using accuracy and decode time. CCOT improves accuracy over the no-contemplation baseline with modest generation-time increases, while PAUSE provides only nominal gains.
- 5. Experiments: CCOT is evaluated on GSM8K using compression ratios r = 0.05 and r = 0.10, with calculator annotations removed from training reasoning chains.The compression ratio is selected during training, and CCOTφ approximates hidden states at that fixed ratio.
- 5. Experiments: The baselines span r = 0.0, which outputs answers directly, and r = 1.0, which generates the full explicit reasoning chain.PAUSE is also evaluated at r = 0.05 and r = 0.10 using appended learned special tokens.
- 5. Experiments: Table 2 reports exact-match accuracy and average wall-clock decode time on a single Nvidia A100 GPU.Higher accuracy indicates better performance, while lower decode time indicates better efficiency.
- 5. Experiments: 9 points of accuracy improvement over the no-contemplation baseline at r = 0.10 requires only a 0.4-second increase in generation time.At r = 0.05, CCOT still improves accuracy by 6 points with a generation-time increase of around 0.15 seconds.
- 5. Experiments: PAUSE contemplation tokens decode faster but produce only nominal performance improvements, whereas CCOT’s dense contentful tokens improve reasoning over the no-token baseline.The authors hypothesize that GSM8K requires more sequential than parallel computation.
6. Further Discussion
CCoT’s adaptive and autoregressive contemplation tokens trade decoding efficiency for additional computation, with performance shaped by compression, layer choice, and hidden-state approximation. Theoretical analysis links autoregressive contemplation tokens to tasks requiring computation deeper than the model’s Transformer stack.
- Varying r: Accuracy plateaus beyond approximately r = 0.2 even as increasing r raises decode time.The authors hypothesize that approximation noise eventually outweighs the signal from additional contemplation tokens.
- Varying l: The best performance occurs near l ≈ L/2, while choices near the embedding or final layer fail to learn good φ weights.Intermediate-layer hidden states are hypothesized to encode global information suitable for autoregressive contemplation-token decoding.
- Subset selection: A better hidden-state scorer could potentially approach lossless performance with only a fraction of full-chain decoding.Evenly spaced token selection performed similarly to the learned scorer, while decoding from gold hidden states was lossless at small r.
- Computational capacity: Autoregressive contemplation tokens add sequential computation, making them relevant to depth-bottlenecked tasks such as multi-hop question answering and sequential games.Generating m tokens adds O(mL) sequential operations in an L-layer model.
- Computational capacity: Under prior assumptions, a 2-layer Transformer can implement some depth-D tasks only when it autoregressively decodes contemplation tokens.The stated theorem applies when D > 2 and concerns tasks involving M independent computations.
7. Conclusion
CCoT generates contentful contemplation tokens autoregressively as compressed reasoning representations. The framework improves the performance-efficiency tradeoff through an adaptive compression ratio and motivates reasoning in continuous space.
- Contribution: CCoT generates contentful, autoregressively decoded contemplation tokens as an alternative to explicit reasoning chains.The framework unifies terminology for tokens that introduce additional computation during language-model inference.
- Implication: Adaptive compression ratios let users trade reasoning performance against efficiency by controlling the number of generated contemplation tokens.The conclusion frames this as an efficiency-performance tradeoff.
- Implication: The work demonstrates the potential of contentful contemplation tokens as a continuous-space reasoning paradigm.
A. Varying the autoregressive layer
The autoregressive layer l determines how CCoT generates contemplation tokens from hidden states. The accompanying GSM8K table reports accuracy for varying l at r = 0.05, with NONE as the no-contemplation baseline.
- Autoregressive layer: CCoT uses the hidden state at layer l and index i as the next input embedding at index i + 1.Here l = 0 is the embedding layer, while l = L is the final layer before the model head.
- Table 3: Table 3 compares GSM8K accuracy across autoregressive-layer choices at r = 0.05 and against the NONE baseline.NONE denotes inference without decoded contemplation tokens.
B. Further Theoretical Considerations
The theoretical section formalizes how contemplation tokens expand Transformer computation, especially for tasks requiring sequential operations. Under stated assumptions, autoregressive decoding supports function classes with greater computational depth than parallel or standard inference.
- Comparison: The paper observes that tasks solvable with parallel contemplation tokens are also solvable autoregressively, while the latter covers a broader problem class.
- Task class: The analysis extends prior K = 2 tasks to function classes requiring M operations of depth K followed by aggregation.Examples include triplet sums, multi-hop question answering, and recursive problems.
- Assumptions: The theory assumes bounded hidden-state information, specified Transformer operations, and block capacity independent of input length.These conditions are given in Assumptions B.2–B.4.
- Theorem B.5: Under the assumptions, standard 2-layer inference represents F_M,2 only for M ≤ N, while parallel contemplation tokens extend this to M ≤ T.
- Theorem B.5: Autoregressive decoding extends representable functions to F_M,K for K > 2 when MK ≤ T.The result formalizes the computational benefit of sequentially decoded contemplation tokens.
- Mechanism: Autoregressive inputs save intermediate computation steps, allowing later Transformer passes to compose additional operations.A depth-K operation requires sequential prefix operations of depths K − 1 through 2, using extra contemplation tokens.