Source-linked AI summary
SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs
Yige Xu, Xu Guo, Zhiwei Zeng, Chunyan Miao
TL;DR
Existing CoT methods can be inefficient or constrained by discrete decoding, while continuous-space approaches may require full-model fine-tuning and risk catastrophic forgetting. SoftCoT uses a fixed assistant model to generate soft thoughts, projects them into the LLM’s representation space, and trains only the projection module, improving reasoning across five benchmarks while preserving the backbone.
Problem
CoT can require costly thought sampling, while continuous-space methods may require full-model fine-tuning that risks catastrophic forgetting in capable instruction-tuned LLMs.
Method
SoftCoT uses a lightweight assistant model to generate instance-specific soft thought tokens, maps them into the LLM’s representation space, and tunes the projection module while freezing the backbone.
Results
SoftCoT consistently improves accuracy across five reasoning datasets spanning mathematical, commonsense, and symbolic reasoning, including ASDiv-Aug.
Takeaways & Limitations
SoftCoT provides a lightweight continuous-space reasoning approach that uses supervised, parameter-efficient fine-tuning while mitigating catastrophic forgetting in state-of-the-art LLMs.
Takeaways & Limitations
SoftCoT has been evaluated on LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct, so scalability to extremely large LLMs remains empirically unvalidated.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) reasoning enables Large Language Models (LLMs) to solve complex reasoning tasks by generating intermediate reasoning steps. However, most existing approaches focus on hard token decoding, which constrains reasoning within the discrete vocabulary space and may not always be optimal. While recent efforts explore continuous-space reasoning, they often require full-model fine-tuning and suffer from catastrophic forgetting, limiting their applicability to state-of-the-art LLMs that already perform well in zero-shot settings with a proper instruction. To address this challenge, we propose a novel approach for continuous-space reasoning that does not require modifying the LLM. Specifically, we employ a lightweight fixed assistant model to speculatively generate instance-specific soft thought tokens as the initial chain of thoughts, which are then mapped into the LLM's representation space via a trainable projection module. Experimental results on five reasoning benchmarks demonstrate that our method enhances LLM reasoning performance through supervised, parameter-efficient fine-tuning. Source code is available at https://github.com/xuyige/SoftCoT.
1 Introduction
CoT improves reasoning but can propagate errors and require costly thought sampling, while continuous-space methods reduce decoding length yet often rely on full-model fine-tuning. SoftCoT addresses these constraints with assistant-generated soft thoughts, projection into the LLM space, and frozen-backbone training, improving reasoning across five benchmarks.
- Motivation: CoT quality affects performance because autoregressive intermediate thoughts can propagate errors, motivating methods that sample or organize multiple reasoning paths.Self-consistency, Tree-of-Thought, and Graph-of-Thought address this challenge but require extensive thought sampling.
- Motivation: Continuous-space methods replace discrete intermediate token sequences with latent representations, but their full-model fine-tuning can incur high costs and catastrophic forgetting.These constraints limit transfer to capable instruction-tuned LLMs.
- Motivation: Fine-tuning LLaMA3.1-8B with a language-modeling objective degraded performance relative to zero-shot CoT, suggesting catastrophic forgetting in continuous-space reasoning.The observation was made for the objective used by Coconut and CCoT.
- Approach: SoftCoT uses a small assistant model to generate instance-specific thought tokens and a trainable projection module to map them into the LLM’s representation space.The assistant’s last-layer hidden states provide soft thoughts, avoiding information loss from autoregressive vocabulary decoding.
- Evaluation: SoftCoT is evaluated on five reasoning benchmarks and two state-of-the-art LLM architectures, including mathematical, commonsense, and symbolic reasoning tasks.The evaluation includes the harder ASDiv-Aug dataset.
- Results: SoftCoT consistently improves accuracy on public and augmented datasets while mitigating catastrophic forgetting associated with full-model fine-tuning.The method is presented as a lightweight approach that preserves the LLM’s prior knowledge.
2 Related Works
Related work spans discrete CoT prompting, structured reasoning paths, and external enhancement methods. Continuous-space approaches reduce reliance on discrete decoding, but Coconut and CCoT use language-modeling fine-tuning objectives that motivate SoftCoT’s alternative design.
- Discrete CoT: CoT research includes prompt construction, topological variants such as trees and graphs, and enhancement methods such as decomposition and self-consistency.These categories organize recent methods for improving chain-of-thought reasoning.
- Discrete CoT: Most existing CoT methods rely on discrete token-by-token generation, which constrains reasoning within the discrete language space.This limitation motivates exploration of continuous representations.
- Method comparison: Figure 1 compares SoftCoT with vanilla Chain-of-Thought and Coconut.Use the figure to compare the three reasoning approaches named in its caption.
- Continuous-space reasoning: Coconut and CCoT perform reasoning in continuous representation space using content-rich or compressed continuous thought tokens.Heima further uses a single continuous vector for compressed reasoning tokens in multimodal tasks.
- Continuous-space reasoning: Coconut and CCoT rely on a language-modeling objective for supervised fine-tuning, distinguishing their training setup from SoftCoT’s projected soft-thought approach.SoftCoT maps assistant-generated soft thoughts into the LLM representation space through a projection module.
3 Methodology
SoftCoT combines instance-specific soft thoughts from a frozen assistant model with a trainable projection and CoT reasoning module, enabling parameter-efficient continuous-space reasoning without modifying the backbone LLM.
- 3.2 Overview of the SoftCoT Framework: SoftCoT generates reasoning steps and final answers by combining task instructions, instance-specific soft thoughts, and word embeddings.The framework has soft thought generation, projection, and CoT reasoning components.
- 3.3 Prompt Tuning for CoT Reasoning: The backbone LLM remains frozen while a relatively small assistant model generates prompts, reducing computational costs and supporting stability across domains.The approach uses fixed task-specific instructions together with instance-specific guidance.
- 3.3 Prompt Tuning for CoT Reasoning: Soft thoughts replace discrete assistant tokens to avoid vocabulary constraints and preserve gradient information during optimization.Hard-token decoding constrains the representation space and prevents gradients from backpropagating through the decoding process.
- 3.4 Soft Thought Tokens for CoT Reasoning: The assistant model processes instructional context, the reasoning question, and placeholder tokens to produce instance-specific soft thoughts.The extracted final-layer hidden states corresponding to the placeholder tokens serve as the soft thoughts.
- 3.4 Soft Thought Tokens for CoT Reasoning: A trainable projection layer maps assistant hidden states from the assistant embedding space into the LLM embedding space.The mapping has dimensionality Linearθ: R^dassist → R^dLLM and is intended to preserve relevant semantic information.
- Parameter-Efficient Training: Supervised training uses a standard language modeling objective over annotated reasoning steps and final answers.The model is trained autoregressively across rationale and answer spans.
4 Experiments
Experiments evaluate SoftCoT across five datasets spanning mathematical, commonsense, and symbolic reasoning, using zero-shot, hard-token, continuous-space, and LoRA baselines.
- 4.1 Datasets: Five datasets cover mathematical, commonsense, and symbolic reasoning.The datasets are GSM8K, ASDiv, AQuA, StrategyQA, and Date Understanding.
- 4.1 Datasets: ASDiv-Aug augments ASDiv by replicating instances and randomly replacing numerical values to test reasoning beyond memorized patterns.The augmented dataset is intended to expose novel instances and is released publicly.
- Baselines: Zero-shot CoT is included to assess whether supervised fine-tuning causes performance degradation.The experiments also use zero-shot CoT-Unk and Zero-Shot Assist-CoT to evaluate projection tuning and soft thoughts.
- Baselines: Coconut and LoRA fine-tuning provide continuous-reasoning and traditional parameter-efficient baselines, respectively.Coconut is adapted to larger LLaMA3.1 and Qwen2.5 models with LoRA fine-tuning.
5 Results and Discussions
SoftCoT consistently improves reasoning performance across five benchmarks and multiple LLM backbones while preserving the benefits of parameter-efficient fine-tuning. Additional analyses show gains with self-consistency, compact soft thoughts, and varying assistant-model scale.
- Comparison with Baselines: SoftCoT consistently outperforms baselines across five mathematical, commonsense, and symbolic reasoning datasets.The evaluation includes public datasets and the augmented ASDiv-Aug benchmark.
- Generalization to Other LLM Backbones: SoftCoT improves Qwen2.5-7B-Instruct performance across all reasoning datasets, indicating robustness across backbone models.The experiments also report benefits in zero-shot domain transfer on Date Understanding.
- The Number of Thought Tokens: Six soft thought tokens achieve optimal ASDiv-Aug performance, while Zero-Shot Assist-CoT requires 24 tokens for similar effectiveness.The reported hard-to-soft token ratio is approximately 4:1.
- The Number of Thought Tokens: SoftCoT consistently outperforms Zero-Shot CoT across tested soft-thought lengths, whereas insufficient hard thoughts can underperform zero-shot CoT.The authors attribute the latter pattern to insufficiently informative assistant-generated thoughts.
- Varying the Size of Assistant Model: Changing assistant-model scale has limited impact on final accuracy in both Zero-Shot Assist-CoT and SoftCoT.This observation is reported for experiments on Qwen2.5 using GSM8K.
- Self Consistency: SoftCoT’s gains remain complementary to self-consistency, allowing both methods to be combined for enhanced reasoning performance.The paper characterizes SoftCoT as introducing an independent improvement mechanism rather than duplicating self-consistency.
6 Conclusion
SoftCoT is an efficient soft chain-of-thought approach that improves LLM reasoning while avoiding backbone modification. It combines speculative soft-thought generation, representation mapping, and frozen-LLM tuning.
- SoftCoT uses an assistant model to generate soft thought tokens, a projection module to map them into the LLM’s representation space, and the LLM to reason with them.The assistant generates all soft thought tokens in a single forward pass, while only the projection module is tuned.
- SoftCoT freezes the backbone LLM to mitigate catastrophic forgetting during reasoning-oriented adaptation.
- Experiments across five datasets and three reasoning-task types support SoftCoT’s effectiveness for LLM reasoning.Additional experiments on multiple LLMs and with self-consistency examine its robustness and adaptability.
Limitations
The paper identifies two limitations: SoftCoT supplements rather than replaces reasoning search, and its scalability beyond tested model sizes remains unvalidated.
- SoftCoT does not fully replace the reasoning path because decoding remains a crucial search process in chain-of-thought reasoning.Soft thought tokens enrich the probability space for exploration rather than serving as the search mechanism itself.
- SoftCoT’s scalability to extremely large LLMs remains an open question requiring further empirical validation.The method was evaluated on LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct, while larger models exist in both families.
A Implementation Details
Training uses Hugging Face Transformers on a single NVIDIA A100-80G GPU, with the projection module optimized for 10 epochs using AdamW. Reproduction materials include logs and checkpoints.
- All models are trained on a single NVIDIA A100-80G GPU using Hugging Face Transformers.
- The projection module is trained for 10 epochs with AdamW, a 1e-5 learning rate, and 0.01 weight decay.
- Batch sizes are 8 for Qwen and 16 for LLaMA, selected according to GPU memory usage.
- The released reproduction includes training logs, intermediate checkpoints, and evaluation logs.The source code is also available for additional implementation details.
B Experiments on Qwen3-Series
On Qwen3-8B, SoftCoT remains effective across mathematical, commonsense, and symbolic reasoning evaluations. Its benefits also remain compatible with self-consistency sampling.
- Qwen3-8B evaluation: SoftCoT remains consistently effective on the cutting-edge Qwen3-8B architecture, supporting robustness across different LLM architectures.
- Mathematical reasoning: Table 6 compares Qwen3-8B baselines on mathematical reasoning datasets using N = 1 or N = 10 reasoning chains.N = 1 omits self-consistency; N = 10 uses 10 return sequences and majority voting.
- Commonsense and symbolic reasoning: Table 7 compares Qwen3-8B baselines on commonsense and symbolic reasoning datasets, reporting average accuracy and standard variance across 5 random seeds.DU denotes the Date Understanding dataset.
- Combination with self-consistency: SoftCoT remains orthogonal to self-consistency on Qwen3-8B, allowing their benefits to be compounded without redundancy or interference.
- Overall finding: The findings support SoftCoT as a model-agnostic strategy that adapts to new LLM releases while enhancing reasoning performance and flexibility.
C Instruction Templates
The examples compare Zero-Shot CoT, Zero-Shot Assist-CoT, and SoftCoT instruction templates for GSM8K. Assist-CoT uses 24 assistant-generated tokens, while SoftCoT uses four soft thought-token positions delimited by special tokens before the LLM produces step-by-step reasoning.
- Zero-Shot CoT: The examples define Zero-Shot CoT as a direct problem-solving prompt followed by concise or step-by-step reasoning instructions and a required final-answer format.The template distinguishes simple problems from complex ones and requires the answer to conclude with a boxed result.
- Zero-Shot Assist-CoT: Zero-Shot Assist-CoT asks a fixed assistant model to generate 24 informative tokens containing useful numbers or operations for the reasoning problem.The assistant prompt prioritizes highly relevant tokens and excludes uninformative sentence structure or serial numbering.
- Zero-Shot Assist-CoT: The large reasoning model receives the problem alongside assistant-generated prompts, which it may use as reference or ignore before solving with the same structured answer format.The example explicitly allows the model to disregard incorrect assistant prompts and return a step-by-step solution ending in the final answer.
- SoftCoT: SoftCoT requests 4 informative assistant-model tokens and inserts reserved special tokens to mark the beginning and ending of soft thought tokens.The end-of-text token serves as the placeholder for soft thought tokens, while the reserved tokens delimit them.
- SoftCoT: SoftCoT places the four-token soft-thought sequence between reserved markers before the large reasoning model receives the problem and produces its answer.The example shows one beginning reserved token, four placeholder positions, and one ending reserved token.