PaperlayerView exact revision on arXiv

Source-linked AI summary

SIM-CoT: Supervised Implicit Chain-of-Thought

Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, Dahua Lin

arXiv:2509.20317v2cs.CLcs.AI

TL;DR

Implicit CoT is token-efficient but trails explicit CoT and can become unstable as its latent reasoning budget grows. SIM-CoT adds training-time step-level supervision through an auxiliary decoder, improving accuracy and stability while retaining efficient inference. It surpasses explicit CoT on GPT-2 and narrows the gap on larger models.

  • Problem

    Implicit CoT methods are token-efficient but less accurate than explicit CoT, and increasing their reasoning tokens can cause unstable training and collapse.

  • Method

    SIM-CoT uses an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, then removes the decoder at inference.

  • Results

    +2.1% over SFT-CoT on GPT-2, +8.2% over Coconut, and +3.0% over CODI on LLaMA-3.1 8B are reported for SIM-CoT.

  • Takeaways & Limitations

    SIM-CoT improves in-domain accuracy and out-of-domain stability across implicit CoT methods and model scales while maintaining fast inference efficiency.

Abstract

from arXiv · show

Implicit Chain-of-Thought (CoT) methods offer a token-efficient alternative to explicit CoT reasoning in Large Language Models (LLMs), but a persistent performance gap has limited their adoption. We identify a core latent instability issue when scaling the computational budget of implicit CoT: as the number of reasoning tokens increases, training often becomes unstable and collapses. Our analysis shows that this instability arises from latent representations becoming homogeneous and losing semantic diversity, caused by insufficient step-level supervision in current implicit CoT methods. To address this, we propose SIM-CoT, a plug-and-play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space. SIM-CoT employs an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information. The auxiliary decoder is removed at inference, preserving the efficiency of implicit CoT with no added overhead. It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis. SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2\% on GPT-2 and CODI by +3.0\% on LLaMA-3.1 8B. It further surpasses the explicit CoT baseline on GPT-2 by 2.1\% with 2.3$\times$ greater token efficiency, while closing the performance gap on larger models like LLaMA-3.1 8B. Code: https://github.com/InternLM/SIM-CoT

1 INTRODUCTION

Implicit CoT improves token efficiency but remains less accurate than explicit CoT, especially as scaling latent tokens can destabilize training. SIM-CoT addresses this with step-level supervision that improves accuracy, stability, interpretability, and generalization while preserving efficient inference.

  • Motivation: Implicit CoT methods are faster and more token-efficient than explicit CoT but remain less accurate, limiting their broader application.
  • Motivation: Increasing implicit tokens can initially improve accuracy but may instead trigger unstable training and collapse.
  • Method: SIM-CoT uses an auxiliary decoder to align each implicit token with its corresponding explicit reasoning step during training.The decoder is removed at inference, so the module adds virtually no extra computational cost.
  • Results: +8.2% on GPT-2 over Coconut, +4.3% over CODI, and +2.1% over the explicit SFT-CoT baseline demonstrate SIM-CoT's main accuracy gains.The method also improves CODI by 3.4% on LLaMA-3.2 1B, 1.5% on LLaMA-3.2 3B, and 3.0% on LLaMA-3.1 8B.
  • Contributions: SIM-CoT provides interpretability by projecting each latent token onto the explicit reasoning vocabulary for per-step visualization and diagnosis.
  • Results: SIM-CoT improves in-domain accuracy and generalizes effectively to out-of-domain datasets across GPT-2 and LLaMA 3 models.

2 ANALYSIS OF IMPLICIT COT: THE LATENT INSTABILITY ISSUE

The analysis shows that scaling implicit tokens can cause accuracy collapse because latent representations lose information, diversity, and geometric separation. SIM-CoT is motivated by this failure pattern and uses finer-grained supervision than existing approaches.

  • Latent Instability Issue: 12.5% accuracy is reached when Coconut is scaled to five latent tokens, after accuracy improves from one to four tokens.The subsequent fluctuations show sensitivity to the number of latent tokens.
  • Information Loss: Increasing latent tokens generally reduces number, operator, and answer accuracy, with operator accuracy declining especially sharply during failure.
  • Framework Comparison: Coconut and CODI use answer-level or trajectory-level supervision, whereas SIM-CoT aligns implicit latents with step-level reasoning through a decoder.
  • Shifted Distance: Collapsed models show sharply reduced latent-to-latent distance and increased distance from the vocabulary embedding space.
  • Semantic Homogenization: Failed models produce homogeneous decoded content, mainly numbers, whereas normal models retain diverse and meaningful tokens.
  • Summary: The combined collapse pattern reflects lost diversity and stability, causing information loss and failure of the reasoning process.

3 METHODOLOGY

SIM-CoT constructs a fixed sequence of implicit latent reasoning states, supervises each state with its corresponding textual reasoning step during training, and retains only explicit answer decoding at inference.

  • 3.2 IMPLICIT PHASE: LATENT CONSTRUCTION BY LAST HIDDEN STATES: SIM-CoT fixes K implicit reasoning steps, autoregressively generates latent states z1:K, then switches to explicit decoding for the final answer.The latent states are appended to the context before answer generation.
  • 3.4 TRAINING-TIME DECODER AND STEP-LEVEL SUPERVISION: A training-only auxiliary decoder conditions on each latent zk and generates its corresponding textual reasoning step sk.The decoder uses teacher forcing to predict the step tokens.
  • 3.5 OBJECTIVES: The auxiliary decoder is discarded at inference, leaving the runtime as implicit latent generation followed by direct answer decoding.Gradients from step supervision shape the latent states during training, while answer supervision trains the base model for inference.
  • 3.5 OBJECTIVES: Training combines textual-step supervision through the decoder with final-answer supervision through the base LLM.The two cross-entropy losses are combined in the overall objective.
  • 3.4 TRAINING-TIME DECODER AND STEP-LEVEL SUPERVISION: Step-level supervision grounds each latent zk to a specific reasoning step, encouraging the latent sequence to carry fine-grained semantics.The loss is computed over textual step tokens rather than the latent vector itself.

4 EXPERIMENT

Experiments evaluate SIM-CoT on in-domain and out-of-domain mathematical reasoning across GPT-2 and LLaMA backbones, comparing accuracy, stability, efficiency, scalability, and interpretability. SIM-CoT consistently improves implicit reasoning baselines while retaining efficient inference and supporting larger latent capacities.

  • Main Results: +8.2 points: SIM-CoT improves over Coconut on GPT-2 in-domain GSM8k-Aug, while also exceeding SFT-CoT by +2.1 points.It additionally improves over CODI by +0.6 points in the same setting.
  • Main Results: +3.4 points: SIM-CoT improves CODI-based results on LLaMA 3.2 1B and reaches 96% of SFT-CoT accuracy.The reported comparison is on the in-domain benchmark.
  • Main Results: +4.3 points: SIM-CoT’s average out-of-domain gain over SFT-CoT with Coconut as the backbone, and it improves CODI by +1.0 point.The out-of-domain benchmarks are GSM-Hard, MultiArith, and SVAMP.
  • Inference Efficiency: 2.3× and 2.2×: SIM-CoT speeds up GPT-2 inference over SFT-CoT on in-domain and out-of-domain benchmarks, respectively.The auxiliary decoder is used during training, while inference retains implicit-CoT efficiency.
  • Number of Implicit Tokens: SIM-CoT provides more stable training and consistent gains over Coconut as the number of implicit latents increases.The ablation evaluates configurations up to eight implicit latents across in-domain and out-of-domain benchmarks.
  • Interpretability: SIM-CoT reuses its training decoder to project each latent step into interpretable expressions, enabling visualization of intermediate reasoning.Each latent corresponds to one reasoning step in the described generation process.

5 RELATED WORK

Related work contrasts explicit CoT’s established reasoning strategies with implicit CoT methods that reduce output length while retaining multi-step reasoning. Prior implicit approaches differ in whether they internalize knowledge, modify architecture, construct latents without training, or model latent reasoning autoregressively.

  • Explicit Chain-of-Thought: Explicit CoT research includes self-consistency, least-to-most prompting, reflection, external tools, and step-level supervision.These methods structure or augment verbalized intermediate reasoning, but explicit CoT increases inference cost with long outputs.
  • Implicit Chain-of-Thought: Implicit CoT reduces output length while retaining multi-step reasoning through knowledge internalization, architectural changes, training-free latent construction, and autoregressive latent reasoning.Coconut is identified as using answer-level supervision.

6 CONCLUSION

SIM-CoT provides step-level supervision for implicit reasoning, improving performance, stability, and interpretability while preserving fast inference. Its benefits extend across implicit-CoT approaches and larger backbones.

  • 6 CONCLUSION: SIM-CoT applies step-level supervision to latent tokens and surpasses explicit and implicit baselines on GPT-2.It is described as a training-based implicit reasoning method with minimal inference overhead.
  • 6 CONCLUSION: SIM-CoT consistently improves performance over existing implicit reasoning methods on larger LLaMA backbones while maintaining fast inference efficiency.The method also improves training stability as the number of latent tokens increases.
  • 6 CONCLUSION: SIM-CoT can integrate with Coconut, CODI, and training-free approaches, while enabling latent-token visualization through explicit-vocabulary projections.The appendix also discusses scaling behavior, decoder size, and soft-thinking combinations.

A ADDITIONAL ANALYSIS ON SCALING TO LARGER BACKBONES

SIM-CoT scales effectively to larger LLaMA backbones, delivering gains over CODI and outperforming SFT-CoT on selected tasks while remaining comparable on others.

  • A ADDITIONAL ANALYSIS ON SCALING TO LARGER BACKBONES: SIM-CoT improves over CODI by +1.5 points on GSM8k-Aug and +1.6 points on SVAMP on LLaMA 3.2 3B.Performance remains comparable on GSM-Hard and MultiArith.
  • A ADDITIONAL ANALYSIS ON SCALING TO LARGER BACKBONES: +3.0 points on GSM8k-Aug, +1.3 points on SVAMP, and +0.8 points on MultiArith over CODI on LLaMA 3.1 8B.Accuracy remains stable on GSM-Hard.
  • A ADDITIONAL ANALYSIS ON SCALING TO LARGER BACKBONES: On LLaMA 3.1 8B, SIM-CoT reaches 100.0 versus 98.3 on MultiArith and 79.4 versus 73.1 on SVAMP against SFT-CoT.The methods remain similar on GSM-Hard.
  • A ADDITIONAL ANALYSIS ON SCALING TO LARGER BACKBONES: These results indicate that SIM-CoT provides consistent gains on larger backbones while reducing reliance on trajectory-level supervision.

B ADDITIONAL ANALYSIS ON DECODER SIZES

Decoder scaling affects SIM-CoT unevenly: a moderate decoder improves performance, whereas larger decoders provide no further benefit and can slightly degrade results.

  • B ADDITIONAL ANALYSIS ON DECODER SIZES: A 1B-scale decoder consistently improves performance across all evaluated benchmarks, unlike 3B and 8B decoders.The larger decoder replacements slightly degrade performance.
  • B ADDITIONAL ANALYSIS ON DECODER SIZES: Moderate decoder scaling may enhance reasoning, while excessively large decoders can create optimization or representation-alignment difficulties.The authors also suggest that matching 1B encoder and decoder representations may facilitate learning.
  • B ADDITIONAL ANALYSIS ON DECODER SIZES: The soft-thinking experiments combine a training-free vocabulary-mixture mechanism with SIM-CoT and evaluate the hybrid on in-domain and out-of-domain mathematical benchmarks.The evaluation uses GPT-2 with GSM8k-Aug, GSM-Hard, MultiArith, and SVAMP.

C.2 RESULTS

Soft thinking augments continuous latent reasoning by mixing vocabulary embeddings, and its combination with SIM-CoT improves performance across all reported benchmarks. The formulation preserves continuous representations while adding vocabulary-based semantics.

  • C.2 RESULTS: +0.2 on GSM8k-Aug, +0.1 on GSM-Hard, +0.7 on MultiArith, and +0.1 on SVAMP for SIM-CoT with soft thinking.For Coconut, gains occur on GSM-Hard and MultiArith, with a slight decrease on SVAMP.
  • C.3 FORMULATION: Soft thinking maps a continuous latent token into vocabulary probabilities, then constructs a weighted mixture of vocabulary embeddings.The mixture forms a soft token containing multiple semantic hypotheses rather than one discrete token.
  • C.3 FORMULATION: The hybrid latent combines the original continuous representation with the soft-thinking embedding using weights that balance both components.This grounds learned latent representations in vocabulary semantics while retaining continuous reasoning.
  • C.3 FORMULATION: Table 4 compares decoder sizes on in-domain and out-of-domain benchmarks and reports latent-token distances under different configurations.The distance analysis includes failed cases and SIM-CoT results.
  • C.2 RESULTS: The hybrid approach improves in-domain and out-of-domain benchmarks by combining training-based supervision with vocabulary-based latent construction.It encourages exploration of diverse intermediate states instead of a single deterministic path.
  • RELATED WORK: Implicit CoT replaces long textual traces with continuous latent computation, while SIM-CoT aligns each latent with its corresponding textual reasoning step during training.The auxiliary decoder is discarded at inference, leaving decoding cost unchanged from standard implicit CoT.

E IMPLEMENTATION AND TRAINING DETAILS

Training uses model-specific optimization settings and curricula for Coconut, CODI, and SIM-CoT, with larger backbones receiving adjustments intended to improve stability.

  • Training uses Adam with β1 = 0.9, β2 = 0.999, weight decay 0.1, and early stopping with patience 3.
  • GPT-2 and LLaMA 1B use a 1 × 10−4 learning rate with a curriculum that progressively replaces explicit steps with implicit latents.Training continues for 15 additional epochs after reaching the maximum latent-step count.
  • LLaMA 3B uses 3 × 10−4 for 8 epochs, whereas LLaMA 8B uses 1 × 10−4 for 6 epochs to address larger-model optimization sensitivity.
  • CODI reproduction follows prior reported schedules: 3 × 10−3 for 40 epochs on GPT-2 and 8 × 10−4 for 10 epochs on LLaMA 1B.These settings are intended to preserve comparability and isolate the proposed method’s effect.
  • Table 6 summarizes training hyperparameters across the evaluated models.

F TRAINING AND INFERENCE DETAILS

The training and inference pipeline gradually increases implicit reasoning steps, uses GSM8K-derived data, and supports both efficient decoding and interpretable latent-step visualization.

  • Training and inference: The curriculum gradually increases the number of implicit steps from K(0) = 0 until reaching Kmax, after which the count remains fixed.Each latent corresponds to two implicit tokens, and Δe controls the update interval.
  • Training and inference: Inference removes the auxiliary decoder and generates the final answer after K implicit steps, with total decoding length T + K + La.Because K is moderate, the cost remains comparable to other implicit methods and can be lower than long explicit CoT trajectories.
  • Training data: GSM8K-Aug expands 8.5k GSM8K training problems to roughly 385k examples, with most problems requiring two to four reasoning steps and a long tail of harder cases.
  • Step-level supervision: SIM-CoT aligns each implicit latent with an explicit reasoning step through auxiliary decoding during training, enforcing consistency with step annotations.
  • Latent diagnostics: Latent geometry is evaluated using average pairwise distance and distance to the vocabulary center; larger pairwise distance indicates better separation.These diagnostics are not used during training but indicate latent diversity and stability.
  • Interpretability: The optional decoder projects latent tokens into human-readable expressions, enabling visualization of ordered intermediate steps and their dependencies.GSM8K examples show decoded latent sequences matching the logic of explicit reasoning.
  • Latent diagnostics: Figure 6 contrasts separated normal latents, collapsed failed latents, and structured SIM-CoT latents that recover separation and stability.

I.2 DISTANCE ANALYSIS

Distance analysis shows that failed five-latent representations collapse and drift from the vocabulary space, whereas SIM-CoT preserves both separation and stability.

  • Distance analysis: 32.81 is the SIM-CoT Dist. value, exceeding 4.21 in the failed five-latent case and indicating stronger separation among latent representations.With increasing latent count, normal Dist. rises from 20.30 to 28.34 before the failed configuration collapses.
  • Distance analysis: 29.80 is the SIM-CoT Dist. to VC value, contrasting with 39.39 for failed five-latent training and indicating more stable proximity to the vocabulary space.The failed case drifts away from the vocabulary manifold, while SIM-CoT balances grounding and diversity.
  • Distance analysis: Figure 6 visually shows normal tokens as separated and grounded, failed tokens as collapsed and outward-drifting, and SIM-CoT tokens as structured.
  • Distance analysis: The combined distance pattern supports a balance between latent diversity and stability when scaling to more implicit reasoning tokens.
  • Future directions: Future work includes multimodal supervision, branched implicit reasoning, RLHF integration, and stronger theoretical foundations.
  • Interpretability: Redundant trailing latents are omitted from visualizations when they repeat the final prediction rather than add new reasoning steps.
  • Interpretability: Decoded case studies show latent steps matching explicit reasoning structure, with final predictions aligned to ground-truth answers.
Loading 2509.20317v2…