PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space

Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, Xin Eric Wang

arXiv:2505.15778v1cs.CLcs.AI

TL;DR

Standard CoT confines LLM reasoning to discrete, sequential language tokens, limiting abstract representation and exploration of multiple plausible paths. Soft Thinking introduces training-free probability-weighted concept tokens in a continuous space, and experiments report higher pass@1 accuracy, lower token usage, and interpretable reasoning than standard CoT.

  • Problem

    Standard CoT reasons with discrete tokens and commits to one sequential branch, limiting abstract concept representation and exploration of multiple plausible reasoning paths.

  • Method

    Soft Thinking is a training-free method that forms continuous concept tokens by probability-weighting the entire vocabulary’s token embeddings during intermediate reasoning.

  • Results

    Across mathematical and coding benchmarks, Soft Thinking consistently improves pass@1 accuracy and reduces token usage versus standard CoT, with gains up to 2.48 points and reductions up to 22.4%.

  • Takeaways & Limitations

    Soft Thinking provides an alternative reasoning paradigm that retains interpretable, concise reasoning without additional training or architectural modifications.

  • Takeaways & Limitations

    Because models are trained on discrete token sequences, continuous concept tokens place inference in an out-of-distribution regime that can cause instability or generation collapse, especially for long reasoning chains or divergent inputs.

Abstract

from arXiv · show

Human cognition typically involves thinking through abstract, fluid concepts rather than strictly using discrete linguistic tokens. Current reasoning models, however, are constrained to reasoning within the boundaries of human language, processing discrete token embeddings that represent fixed points in the semantic space. This discrete constraint restricts the expressive power and upper potential of such reasoning models, often causing incomplete exploration of reasoning paths, as standard Chain-of-Thought (CoT) methods rely on sampling one token per step. In this work, we introduce Soft Thinking, a training-free method that emulates human-like "soft" reasoning by generating soft, abstract concept tokens in a continuous concept space. These concept tokens are created by the probability-weighted mixture of token embeddings, which form the continuous concept space, enabling smooth transitions and richer representations that transcend traditional discrete boundaries. In essence, each generated concept token encapsulates multiple meanings from related discrete tokens, implicitly exploring various reasoning paths to converge effectively toward the correct answer. Empirical evaluations on diverse mathematical and coding benchmarks consistently demonstrate the effectiveness and efficiency of Soft Thinking, improving pass@1 accuracy by up to 2.48 points while simultaneously reducing token usage by up to 22.4% compared to standard CoT. Qualitative analysis further reveals that Soft Thinking outputs remain highly interpretable and readable, highlighting the potential of Soft Thinking to break the inherent bottleneck of discrete language-based reasoning. Code is available at https://github.com/eric-ai-lab/Soft-Thinking.

1 Introduction

Soft Thinking addresses the expressive and path-exploration limits of discrete, sequential CoT by reasoning with probability-weighted concept tokens in a continuous space. Across mathematical and coding benchmarks, it improves accuracy, reduces token usage, and preserves readable intermediate reasoning.

  • Motivation: Standard CoT restricts reasoning to discrete language tokens and commits to one sequential reasoning branch at each step.These constraints can limit abstract concept representation and lead to incorrect paths or wasted tokens when multiple trajectories are plausible.
  • Method: Soft Thinking replaces discrete token selection with probability-weighted aggregation over the vocabulary to construct continuous concept tokens.The resulting embeddings retain the next-step distribution and support more nuanced semantic representations.
  • Efficiency: Cold Stop monitors output entropy and terminates reasoning to improve efficiency and mitigate generation collapse under out-of-distribution inputs.The mechanism addresses repetition and other collapse risks caused by concept tokens that may be unseen during training.
  • Method: Continuous concept tokens preserve information about multiple possible reasoning paths and enable their implicit parallel exploration.The method operates in a continuous concept space formed as a convex combination of token embeddings.
  • Results: 2.48 points is the maximum reported pass@1 improvement, while token usage falls by up to 22.4% versus standard CoT.Evaluations use mathematical and coding benchmarks with Llama and Qwen architectures at 32B and 70B parameter sizes.
  • Results: Soft Thinking’s intermediate reasoning remains highly readable, interpretable, and informative while improving effectiveness and efficiency.These qualitative findings accompany the reported benchmark gains.

2 Related Work

Related work studies reasoning beyond ordinary token sequences through hidden representations, interventions, latent paths, and discrete planning variables. These approaches motivate continuous-space reasoning while using different mechanisms for exposing or controlling intermediate reasoning.

  • Chain-of-Thought Reasoning: Chain-of-Thought methods include prompt-based learning, supervised fine-tuning, and reinforcement learning optimization for multi-step reasoning.
  • Chain-of-Thought Reasoning: Longer reasoning chains can improve performance under inference-time scaling laws but also increase computational cost.
  • Continuous Space Reasoning: Prior work decoded intermediate reasoning variables from hidden representations in datasets for two-hop reasoning tasks.
  • Continuous Space Reasoning: Other studies intervened on hidden states to manipulate reasoning outcomes and observed parallel latent reasoning paths.
  • Continuous Space Reasoning: Latent planning approaches predict discrete planning tokens before generating reasoning steps.

3 Methodology

Soft Thinking replaces discrete intermediate-token sampling with probability-weighted concept tokens in a continuous concept space, preserving information about multiple reasoning paths. It uses entropy-based Cold Stop and approximates path summation through iterative linearization while keeping final answer generation discrete.

  • Soft Thinking: Soft Thinking replaces discrete one-hot intermediate tokens with full probability distributions called concept tokens.Each concept token is injected through its embedding into the next reasoning step.
  • Soft Thinking: Soft Thinking preserves multiple possible reasoning paths instead of committing to one sampled token at each intermediate step.The full vocabulary distribution is retained at every step, supporting parallel path exploration.
  • Continuous Concept Space: The continuous concept space consists of probability-weighted mixtures, or convex combinations, of vocabulary token embeddings.This representation captures abstract and fine-grained semantics beyond individual vocabulary items.
  • Reasoning Process: Only intermediate thinking uses continuous concept tokens; final answer tokens are sampled discretely after the end-of-thinking condition.This separates soft reasoning from ordinary discrete answer generation.
  • Cold Stop: Cold Stop terminates intermediate reasoning after sustained low entropy, using confidence to improve efficiency and prevent generation collapse.When entropy remains below threshold τ for k consecutive steps, the method inserts an end-of-thinking token.
  • Theoretical Analysis: The theoretical analysis derives Soft Thinking as an iterative linear approximation to standard CoT’s full path summation.It replaces exponential trajectory summations with evaluation at concept-token representations through recursive linearization.

4 Experiments & Results

Soft Thinking is evaluated across mathematical and coding benchmarks, model families, and baselines using accuracy and generation length. It consistently improves Pass@1 while reducing token usage, with analyses attributing these gains to richer path exploration and concise reasoning.

  • 4.1 Experiment Setup: The evaluation covers eight mathematics and programming benchmarks using QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and DeepSeek-R1-Distill-Llama-70B.The models span 32B and 70B scales, Qwen and LLaMA architectures, and reinforcement-learning or supervised-distillation training.
  • 4.1 Experiment Setup: Soft Thinking is compared with Standard CoT Thinking and Standard Greedy CoT Thinking using Pass@1 accuracy and tokens generated for correct solutions.The setup uses 16 samples per problem for Standard CoT and one sample with temperature 0 for greedy CoT.
  • 4.3 Results and Analysis: 2.48 percentage points: QwQ-32B’s average mathematical Pass@1 rises from 83.84% with CoT Thinking to 86.32% with Soft Thinking, with gains across math and coding benchmarks.On AIME2024, the improvement reaches 6.45 percentage points; coding gains include 0.48 points for QwQ-32B.
  • 4.3 Results and Analysis: 22.4%: Soft Thinking reduces mathematical token usage for DeepSeek-R1-Distill-Qwen-32B relative to standard CoT, with reductions also reported for other models and coding tasks.Mathematical reductions are 11.6% for QwQ-32B and 17.9% for DeepSeek-R1-Distill-Llama-70B; coding reductions range from 16.1% to 19.1%.
  • 4.3 Results and Analysis: Soft Thinking preserves accuracy while using richer concept tokens to encode abstract or composite ideas, reducing unnecessary reasoning detours and redundant steps.The analysis contrasts this with greedy decoding, which reduces tokens but can lose accuracy when reduced path diversity causes premature convergence.
  • 4.4 Qualitative Results: In a multiplication example, both methods reach the correct answer, while Soft Thinking produces a 96-token explanation versus 157 tokens for standard CoT.The visualization selects the highest-probability token at each Soft Thinking step for readability and interpretability.
  • 4.4 Qualitative Results: During exploratory reasoning, token distributions are more uniform, whereas precise calculations produce nearly one-hot distributions, associating exploration with textual elements and exact computation with numerical components.This qualitative pattern is shown through intermediate probability distributions in Figure 4.

5 Conclusion

Soft Thinking is a training-free framework for reasoning in a continuous concept space through probabilistically weighted concept tokens. Experiments show improved pass@1 accuracy, shorter generation, and interpretable reasoning without additional training or architectural changes.

  • 5 Conclusion: Soft Thinking replaces traditional discrete tokens with probabilistically weighted concept tokens in a continuous concept space.The method aggregates information across the entire vocabulary at each reasoning step.
  • 5 Conclusion: Soft Thinking implicitly explores multiple reasoning paths in parallel, leading to higher accuracy and greater token efficiency.
  • 5 Conclusion: Experiments on mathematical and coding benchmarks consistently improve pass@1 accuracy and reduce generation length without additional training or architectural modifications.
  • 5 Conclusion: Qualitative analyses show that Soft Thinking's reasoning process remains interpretable and concise.

A.1 Limitation

Soft Thinking's training-free use of continuous concept tokens places current language models in an out-of-distribution regime. This can cause instability or generation collapse, particularly during long reasoning chains or on inputs unlike training data.

  • A.1 Limitation: Because models are trained on discrete token sequences, continuous concept tokens are unseen during pre-training and fine-tuning.
  • A.1 Limitation: Feeding concept-token mixtures during inference creates an out-of-distribution regime that can produce instability or generation collapse.
  • A.1 Limitation: Instability is especially possible when reasoning chains become long or inputs diverge from the model's training data.
  • A.1 Limitation: Cold Stop mitigates these issues by terminating reasoning when the model is sufficiently confident, but it does not fundamentally resolve the OOD problem.
  • A.1 Limitation: Future work should train models with concept tokens to improve robustness and generalization under soft-thinking paradigms.

A.2 Benchmarks

The evaluation covers four mathematical and three programming benchmark tasks. The mathematical suite spans diverse and difficult reasoning problems, while the coding suite assesses functional correctness and programming ability.

  • A.2 Benchmarks: The evaluation covers four mathematical benchmark tasks and three programming benchmark tasks.
  • A.2 Benchmarks: The mathematical benchmarks include Math500, AIME 2024, GSM8K, and GPQA-Diamond.
  • A.2 Benchmarks: Math500 contains 500 MATH problems spanning seven mathematical disciplines.
  • A.2 Benchmarks: The coding benchmarks are HumanEval, MBPP, and LiveCodeBench.
  • A.2 Benchmarks: HumanEval measures functional correctness through executed unit tests, while MBPP assesses basic programming logic from simple specifications.

A.3 Implementation of Soft Thinking on SGLang

The appendix describes engineering modifications to SGLang v0.4.6.post1 that support Soft Thinking. It identifies core code changes, affected files, and high-level snippets explaining the revised reasoning flow.

  • A.3 Implementation of Soft Thinking on SGLang: Soft Thinking is implemented through engineering modifications to the SGLang inference engine, version v0.4.6.post1.
  • A.3 Implementation of Soft Thinking on SGLang: The implementation description covers core code changes, affected files, and high-level code snippets.
  • A.3 Implementation of Soft Thinking on SGLang: The snippets clarify the new reasoning flow introduced for Soft Thinking.

A.3.1 Overview of Modifications

Soft Thinking modifies SGLang’s inference pipeline to represent intermediate reasoning as probability distributions over tokens rather than single discrete token ids. The changes add configuration, distributional sampling, weighted embeddings, concept-token propagation, and entropy-based Cold Stop.

  • A.3.1 Overview of Modifications: Soft thinking introduces an inference mode whose intermediate reasoning steps use full token probability distributions instead of discrete token ids.This requires changes to SGLang’s input/output interface, sampling, embedding, and state management.
  • A.3.1 Overview of Modifications: The sampler outputs top-k probability distributions as concept tokens rather than a single sampled token.The modified sampler provides the distributions and indices needed for subsequent embedding computation.
  • A.3.1 Overview of Modifications: The embedding layer computes concept representations through probability-weighted interpolation of token embeddings.This converts the sampler’s distributional output into embeddings consumed by the model.
  • A.3.1 Overview of Modifications: The forward pipeline is adapted to accept and propagate concept tokens through the model.Concept-token inputs are processed as part of the standard inference flow.
  • A.3.1 Overview of Modifications: Cold Stop adds entropy-based early stopping for intermediate reasoning steps.It is introduced to improve efficiency and address generation collapse such as repetition from OOD inputs.

A.3.2 Key Files and Logic Changes

The implementation adds soft-thinking controls across sampling, embedding, model forwarding, state tracking, and stopping logic. Together, these changes support distributional reasoning and entropy-based early stopping with minimal overhead to the standard inference pipeline.

  • A.3.2 Key Files and Logic Changes: Configuration flags enable soft thinking and expose parameters controlling its behavior.The implementation adds options such as enable_soft_thinking, --max-topk, and --think-end-str.
  • A.3.2 Key Files and Logic Changes: The sampler emits top-k probabilities and indices for concept tokens instead of only discrete token ids.Standard discrete sampling remains available, while entropy is calculated for Cold Stop.
  • A.3.2 Key Files and Logic Changes: The weighted-forward path computes the next representation from probability-weighted token embeddings.It receives top-k probabilities and indices, forms a weighted sum, and normalizes the probabilities.
  • A.3.2 Key Files and Logic Changes: The model forward pass accepts concept-token inputs through top-k probabilities and indices, including tensor-parallel execution paths.These inputs are used when available instead of relying solely on input embeddings.
  • A.3.2 Key Files and Logic Changes: Cold Stop terminates reasoning after sufficiently low entropy persists for the configured length threshold, then switches to answer mode.The mechanism inserts the designated end-of-thinking token when the stopping condition is met.
  • A.3.2 Key Files and Logic Changes: These modifications enable continuous, distributional reasoning and entropy-based early stopping with minimal overhead to the standard inference pipeline.The implementation also tracks soft-thinking state and exposes entropy and length thresholds as per-batch parameters.
Loading 2505.15778v1…