LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking

Motivation

• CoT relies on discrete natural-language tokens, which may limit alternative-solution exploration and reasoning beyond language.Soft Thinking instead aims to support abstract, high-bandwidth signals and parallel reasoning paths.Source: §1 · p. 1

Findings

• LLMs predominantly follow the highest-probability token in Soft Thinking, suppressing alternative paths through a greedy feedback loop.This top-1 dominance prevents Soft Tokens from fully delivering their potential for parallel reasoning.Source: §1 · p. 2, Figure 1 · p. 2

Method

• Stochastic Soft Thinking introduces controlled randomness, especially through Gumbel-Softmax, to mitigate greedy behavior and improve reasoning performance.The framework uses techniques including Dirichlet sampling and Gumbel-Softmax.Source: §1 · p. 2

Contribution

• The paper presents an in-depth analysis showing that Soft Thinking does not inherently support parallel reasoning and is dominated by top-1 token signals.This analysis addresses the limited understanding of Soft Thinking’s underlying mechanisms.Source: §1 · pp. 1–2

Implications

• Stochastic Soft Thinking shows stronger exploration potential than conventional CoT and supports test-time scaling for Soft Thinking.These findings motivate future work on improving Soft Thinking with Reinforcement Learning.Source: §1 · p. 2

2.1 DISCRETE THINKING

• In discrete Thinking, the LLM predicts each next token from the query and previously generated tokens, then switches to the final answer after an end-of-thinking token.The generated sequence contains an intermediate reasoning path and a final answer.Source: §2.1 · pp. 2–3

2.2 SOFT THINKING

• A Soft Token is the generated probability distribution over the LLM vocabulary rather than a single selected token.Soft Thinking therefore bypasses the final token-selection step during intermediate reasoning.Source: §2.2 · p. 3
• The Soft Token embedding is a weighted sum of token embeddings, producing a continuous input for the next model step.This construction keeps the representation within the original input-space manifold as a convex combination.Source: §2.2 · p. 3
• In practice, Soft Thinking replaces discrete-token CoT only during intermediate reasoning, then uses discrete decoding for the readable final answer.Reasoning terminates when the top-1 Soft Token is the end-of-thinking token.Source: §2.2 · p. 3
• Soft Thinking has been proposed as a latent search tree that could preserve multiple possibilities and perform implicit parallel search, but empirical support for this hypothesis was absent.The paper motivates its analysis by testing whether this proposed property appears in actual LLM behavior.Source: §2.3 · p. 3

EXPERIMENT SETTING

• The experiments evaluate Deepseek-R1-Distill-Qwen-32B, QwQ-32B, and Skywork-OR1-32B on eight diverse benchmarks.Source: §3.1 · pp. 3–4
• For AIME’24/’25 and AMC’23, the study reports Average@32, while the remaining benchmarks use Pass@1.Source: §3.1 · p. 4

EXPERIMENTAL RESULTS

• Table 1 reports results for mathematics, knowledge question-answering, and code generation, with colors indicating improvement, comparability, or decline relative to Token CoT.Source: Table 1 · p. 4

• Vanilla Soft Thinking generally underperforms discrete Token Thinking across all tested LLMs, without improving accuracy.The evaluation covers math, knowledge question-answering, and code benchmarks.Source: §3.2 · p. 4, Table 1 · p. 4

SINGLE-THREADED REASONING

• LLMs generally process Soft Tokens through the dominant component instead of maintaining multiple semantic trajectories in parallel.The paper names this hypothesis “LLMs are Single-Threaded Reasoners.”Source: §4.1 · p. 5

OUTPUT PROBABILITY OF SOFT THINKING

• In a QwQ-32B analysis of nearly 10^6 decoding steps from AIME’24 and AIME’25, the authors compare predictions from the full Soft Token, top-1 token, and top-2 token using JS divergence.Source: §4.1 · p. 5, §4.2 · p. 5
• The soft input’s next-token prediction closely matches the highest-probability token input but differs sharply from the second-highest token input.This indicates that Soft Thinking behavior is dominated by the top-1 token.Source: Figure 3 · p. 5, §4.2 · p. 5

DECODING HIDDEN STATES BY LOGIT LENS

• Logit Lens analysis shows both token paths emerge during the first 2–3 layers, after which the first token’s path rises toward 1.0 while the second declines.The forward process therefore progressively prunes the second reasoning path.Source: §4.3 · pp. 5–6

GREEDY PITFALL

• The resulting top-1 dominance creates a positive feedback loop that entrenches the model in its most confident reasoning path, termed the “Greedy Pitfall.”Source: §4.4 · p. 6

5.1 STOCHASTIC SOFT THINKING

• Stochastic Soft Thinking samples valid, unbiased, non-collapsed probability distributions to inject controllable randomness into Soft Thinking.The proposed stochastic soft tokens are designed to preserve predictive information while remaining soft rather than one-hot.Source: §5.1 · p. 7

5.1.1 DIRICHLET SAMPLING

• Dirichlet sampling uses scaled concentration parameters γp because directly using the model distribution concentrates samples near simplex corners and produces nearly one-hot vectors.The scaling parameter controls the concentration of the sampled stochastic soft token.Source: §5.1.1 · p. 7

5.1.2 GUMBEL-SOFTMAX TRICK

• Gumbel-Softmax adds Gumbel noise to logits and replaces argmax with temperature-controlled softmax, producing continuous differentiable stochastic samples.Its samples interpolate between one-hot categorical distributions and continuous categorical densities.Source: §5.1.2 · p. 7

5.2 EXPERIMENTAL RESULTS

• Gumbel-Softmax is the only stochastic approach reported to improve beyond discrete Token Thinking, while both stochastic methods outperform vanilla Soft Thinking.The results cover eight benchmarks, with detailed comparisons presented in Table 2.Source: §5.2 · pp. 7–8, Table 2 · p. 7

5.3 BALANCING RANDOMNESS AND SOFTNESS

• Gumbel-Softmax adjusts softness through temperature while maintaining adequate randomness, whereas Dirichlet sampling cannot balance the two properties effectively.Dirichlet samples become nearly one-hot at low γ and approach the original distribution with limited randomness as γ increases.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 THEORETICAL JUSTIFICATION

• The Gumbel-Max construction extends to ordered sampling without replacement, and softmax with top-k renormalization relaxes this discrete ranking for stochastic soft tokens.The theorem preserves proportional categorical probabilities when selecting multiple items without replacement.Source: §5.4 · p. 9

6.1 WELL-DEFINED POLICY RATIO

• RL policy-gradient methods require stochastic exploration, but deterministic latent CoT generation provides no room for it.The paper contrasts COCONUT’s deterministic hidden-state generation with the exploration requirements of policy-gradient training.Source: §6.1 · p. 9
• Gumbel-Softmax samples Stochastic Soft Tokens with a well-defined probability density function, unlike hidden-state sampling.This property supports a principled policy ratio for latent trajectories.Source: §6.1 · p. 9
• The resulting well-defined policy ratio enables scalable RL training to further enhance Soft Thinking.The formulation uses the model’s output distribution over sampled token sequences.Source: §6.1 · p. 9

6.2 STRONGER EXPLORATION POTENTIAL

• Stochastic Soft Thinking consistently outperforms Discrete Token Thinking across Pass@k values from 1 to 32 on MATH500 for Qwen2.5 models from 0.5B to 7B parameters.The authors identify this result as evidence that stochastic soft rollouts may be promising for RL training, while leaving full RL integration for future work.Source: §6.2 · p. 10

7 LIMITATION AND FUTURE WORKS

• Supervised fine-tuning can cause distributional shift and catastrophic forgetting of knowledge acquired during pretraining.This may help explain why COCONUT and related methods struggle to match token-based CoT when compressing multiple tokens into one embedding.Source: §7 · p. 10
• Although RL can enhance reasoning without forgetting, its effectiveness remains constrained by the capabilities of the base model.The paper presents this as a scope boundary for using RL to improve Soft Thinking.Source: §7 · p. 11
• Existing LLMs cannot generate Soft Thinking traces with parallel reasoning characteristics, likely because Transformer and next-token objectives impose a strong inductive bias.The authors argue that relying solely on RL may therefore be insufficient for parallel reasoning, motivating multi-token prediction pretraining and new architectures.Source: §7 · pp. 10–11
• Soft Thinking retains advantages beyond parallel reasoning because Soft Tokens convey enriched information, leaving this potential for future work.The authors propose harnessing that information to improve reasoning even though parallel traces are absent.Source: §7 · p. 11

8 RELATED WORK

• CoT enables step-by-step reasoning through natural language but may be inefficient because it relies on discrete text tokens.Continuous-space reasoning has therefore been studied through implicit transformer reasoning and related approaches.Source: §8 · p. 11
• Training-free Soft Thinking and Mixture-of-Inputs methods use vocabulary distributions to align hidden-state outputs with input embeddings during continuous-space reasoning.These approaches bridge output and input representation spaces without requiring discrete intermediate tokens.Source: §8 · p. 11
• Existing theoretical frameworks propose that Soft Thinking can perform implicit parallel search, but empirical evidence for these hypotheses remains absent.The paper positions its behavioral analysis as addressing this gap.Source: §8 · p. 11

9 CONCLUSION

• LLMs predominantly rely on the most influential component of soft inputs, so vanilla Soft Thinking resembles greedy decoding rather than simultaneous multi-path reasoning.This feedback loop limits the benefit of transmitting richer information through Soft Tokens.Source: §9 · p. 11
• Stochastic Soft Thinking introduces controlled randomness to disrupt greedy feedback, with Gumbel-Softmax identified as an effective randomization approach.The conclusion cites theoretical proof and experimental results supporting this choice.Source: §9 · p. 11
• The behavioral analysis establishes a foundation for further reinforcement-learning training of Soft Thinking models.The paper frames this as a direction enabled by deeper understanding of latent reasoning behavior.Source: §9 · p. 11

Search and reinforcement learning

• The related methods also include A* sampling, proximal policy optimization, and reinforcement-learning analyses of reasoning capacity.These references connect search and policy-optimization techniques to reasoning-model development.Source: R Duncan Luce et al. Individual choice… · pp. 14–15

Reasoning approaches

• Prior work explores alternative reasoning structures, including parallel reasoning processes, deliberate search over thought trees, and reasoning by superposition.These works frame reasoning as potentially branching or distributed rather than strictly sequential.Source: R Duncan Luce et al. Individual choice… · pp. 14–15

Systems and models

• Additional references concern efficient structured-language-model execution, logit-lens interpretability, and open models such as DeepSeekMath and QwQ-32B.These resources support implementation, internal-behavior analysis, and mathematical-reasoning comparisons.Source: R Duncan Luce et al. Individual choice… · pp. 14–15

Continuous reasoning

• Recent studies investigate continuous or latent reasoning through soft thinking, compressed chain-of-thought, dynamic latent compression, and generation beyond discrete token sampling.Together, these references situate reasoning in continuous concept spaces or latent representations.Source: R Duncan Luce et al. Individual choice… · p. 15

Evaluation and analysis

• The evaluation literature covers out-of-distribution robustness, latent multi-hop reasoning, mathematical reasoning, and benchmark construction for graduate-level question answering.These works provide methodological and benchmarking context for assessing reasoning behavior.Source: R Duncan Luce et al. Individual choice… · pp. 14–15

Mathematical reasoning

• AIME’24&’25 includes 30 challenging problems from each of the 2024 and 2025 American Invitational Mathematics Examinations.These datasets assess deep mathematical insight and creative problem-solving.Source: A DATASET DETAILS · p. 16
• MATH500 contains 500 multi-step problems across algebra, combinatorics, geometry, number theory, and precalculus.The benchmark targets mathematical reasoning beyond basic computation or factual recall.Source: A DATASET DETAILS · p. 16
• AMC’23 contains 40 middle- and high-school competition problems emphasizing logical reasoning, mathematical creativity, and conceptual understanding.It evaluates moderately challenging mathematical tasks.Source: A DATASET DETAILS · p. 16

Science reasoning

• GPQA-Diamond contains 198 expert-validated multiple-choice questions in biology, chemistry, and physics.Its items range from advanced undergraduate to postgraduate difficulty, with strong discrimination between experts and non-experts.Source: A DATASET DETAILS · p. 16

Programming evaluation

• HumanEval, MBPP, and LiveCodeBench evaluate functional code synthesis, basic programming skills, and interactive coding workflows, respectively.HumanEval uses unit tests, MBPP contains 974 crowd-sourced tasks, and LiveCodeBench emphasizes editing, debugging, and code completion.Source: A DATASET DETAILS · p. 16