PaperlayerView exact revision on arXiv

Source-linked AI summary

SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens

Yinhan He, Wendy Zheng, Yaochen Zhu, Zaiyi Zheng, Lin Su, Sriram Vasudevan, Qi Guo, Liangjie Hong, Jundong Li

arXiv:2510.24940v3cs.CL

TL;DR

Verbose CoT increases reasoning time, while existing implicit CoT methods face challenges preserving semantic alignment and reducing the cost of generating each reasoning token. SemCoT combines a contrastively trained sentence transformer with a lightweight implicit reasoning generator. Across multiple datasets and LLMs, the paper reports superior efficiency and effectiveness compared with state-of-the-art baselines.

  • Problem

    Existing implicit CoT methods struggle to preserve alignment between implicit and ground-truth reasoning, and may not address the cost of generating each reasoning token.

    Source: §1 · p. 2

  • Method

    SemCoT uses a contrastively trained sentence transformer to guide a lightweight language model that generates implicit reasoning while optimizing answer accuracy.

    Source: §1 · pp. 2–3, §3.3 · p. 5

  • Results

    Across multiple datasets and LLMs, SemCoT reports superior efficiency and effectiveness compared with state-of-the-art baselines.

    Source: §4.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The paper concludes that SemCoT advances the computational accessibility of advanced reasoning for real-world applications.

    Source: §6 · p. 10

    The approach treats semantic alignment in embedding space as a practical alternative because translating implicit reasoning into human language is difficult to formulate explicitly and human language is discrete.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

The verbosity of Chain-of-Thought (CoT) reasoning hinders its mass deployment in efficiency-critical applications. Recently, implicit CoT approaches have emerged, which encode reasoning steps within LLM's hidden embeddings (termed ``implicit reasoning'') rather than explicit tokens. This approach accelerates CoT by reducing the reasoning length and bypassing some LLM components. However, existing implicit CoT methods face two significant challenges: (1) they fail to preserve the semantic alignment between the implicit reasoning (when transformed to natural language) and the ground-truth reasoning, resulting in a significant CoT performance degradation, and (2) they focus on reducing the length of the implicit reasoning; however, they neglect the considerable time cost for an LLM to generate one individual implicit reasoning token. To tackle these challenges, we propose a novel semantically-aligned implicit CoT framework termed SemCoT. In particular, for the first challenge, we design a contrastively trained sentence transformer that evaluates semantic alignment between implicit and explicit reasoning, which is used to enforce semantic preservation during implicit reasoning optimization. To address the second challenge, we introduce an efficient implicit reasoning generator by finetuning a lightweight language model using knowledge distillation. This generator is guided by our sentence transformer to distill ground-truth reasoning into semantically aligned implicit reasoning, while also optimizing for accuracy. SemCoT is the first approach that enhances CoT efficiency by jointly optimizing token-level generation speed and preserving semantic alignment with ground-truth reasoning. Extensive experiments demonstrate the superior performance of SemCoT compared to state-of-the-art methods in both efficiency and effectiveness. Our code can be found at https://github.com/YinhanHe123/SemCoT/.

1 Introduction

Explicit Chain-of-Thought can be verbose and time-consuming, while existing implicit CoT methods face challenges preserving alignment with ground-truth reasoning and reducing per-token generation cost. SemCoT addresses both by combining semantic-alignment training with a lightweight implicit reasoning generator.

1 Introduction

  • Implicit CoT replaces explicit reasoning tokens with a small number of first-layer hidden embeddings and can avoid tokenization and unembedding steps.The paper presents implicit CoT as an efficiency strategy because it shortens reasoning and bypasses some decoding components.Source: §1 · p. 2
  • Existing implicit CoT methods struggle to align implicit reasoning with ground-truth reasoning and can leave the high cost of generating each reasoning token unaddressed.The text notes that generating one token can cost approximately 0.1 seconds for DeepSeek-R1.Source: §1 · p. 2
  • SemCoT uses a customized sentence transformer to assess semantic alignment and guide a lightweight language model that generates implicit reasoning while optimizing answer accuracy.The generator is trained to distill ground-truth reasoning into semantically aligned implicit reasoning.Source: §1 · pp. 2–3
  • The paper evaluates SemCoT and state-of-the-art implicit CoT baselines across multiple LLMs and real-world NLP tasks.The stated evaluation tests both effectiveness and efficiency.Source: §1 · p. 3

2 Preliminaries and Problem Definition

The paper defines implicit reasoning and formalizes its goal as reducing the time needed to generate it while retaining answer accuracy relative to traditional CoT.

2 Preliminaries and Problem Definition

  • Implicit reasoning tokens are embedding vectors that encode reasoning information without requiring explicit-token generation; the framework denotes the query as Q, answer as Y, and reasoning as R or Z.The white-box LLM is Fg, the lightweight generator is Iψ, and the customized sentence transformer is Cϕ.Source: §2 · p. 3
  • The problem is to minimize the time for a white-box LLM to generate implicit reasoning before producing the answer, subject to maximal retention of original CoT answer accuracy.Source: §2 · p. 3

3 Methodology

SemCoT assesses semantic alignment between implicit and ground-truth reasoning, then trains a lightweight generator to produce efficient implicit reasoning that supports correct answers. Its training combines semantic-alignment and answer-prediction objectives, and inference feeds the generated embeddings to the LLM.

3.2 Implicit vs. Ground-truth Reasoning Semantic Alignment Assessment

  • The customized sentence transformer is contrastively trained to place ground-truth reasoning near condensed aligned reasoning and apart from in-batch negatives.It compares semantic embeddings; its motivation is that direct vector distances in LLM embedding space may not reliably measure sentence meaning.Source: §3.2 · pp. 4–5

3.3 Token-level Efficient Implicit Reasoning Generation

  • A lightweight generator produces implicit reasoning by transforming the last-layer embeddings of k special <CoT> tokens into the LLM’s embedding space.The generator uses a pruned or distilled model selected for semantic properties resembling those of the original LLM.Source: §3.3 · p. 5
  • The generator is trained with answer-prediction and semantic-alignment losses, with λ controlling the emphasis on alignment.The prediction loss uses the LLM’s probabilities for answer tokens conditioned on the query, implicit reasoning, and preceding answer tokens.Source: §3.3 · pp. 5–6

3.4 Training and Inference Strategy

  • Training first optimizes the sentence transformer with contrastive loss, then freezes it while optimizing the generator; inference passes generated implicit reasoning to the LLM to produce an answer.Source: §3.4 · p. 6

4 Experimental Evaluations

The evaluation compares SemCoT with efficient-CoT baselines across multiple datasets and two LLMs, examining effectiveness, efficiency, component contributions, parameter sensitivity, and semantic consistency.

4 Experimental Evaluations

  • The evaluation asks whether SemCoT retains CoT effectiveness while improving efficiency, which components contribute, how token count and hyperparameters affect performance, and whether its reasoning is semantically aligned.It uses five datasets spanning mathematical, commonsense, and symbolic reasoning, and tests Llama-2-7b-chat-hf and Mistral-7B-Instruct-v0.2.Source: §4 · p. 6, §4.1 · p. 6

4.1 Experiment Settings

  • Across five datasets and two LLMs, answer accuracy measures effectiveness and average wall-clock answer-generation time measures efficiency; evaluation uses one implicit reasoning token.Training uses five implicit tokens, and the lightweight generator uses corresponding distilled or sheared models.Source: §4.1 · pp. 6–7

4.2 Effectiveness & Efficiency of SemCoT

  • SemCoT achieves the highest answer accuracy on most datasets and nearly the fastest implicit-reasoning processing time, giving the best overall efficiency–effectiveness trade-off.On some datasets it is slightly slower than SoftCoT while achieving much higher accuracy.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Ablation Study

  • Removing any SemCoT component generally reduces performance; removing semantic-alignment loss hurts more than replacing the custom transformer with mean-pooled cosine similarity.Replacing the lightweight generator with a LoRA-finetuned original LLM is less effective, potentially because of catastrophic forgetting.Source: §4.3 · p. 8

4.4 Parameter Sensitivity

  • As λ varies, accuracy generally increases except at λ = 0.5, while inference time remains consistent; accuracy decreases as the number of implicit tokens increases.The authors report similar token-count findings on other datasets.Source: §4.4 · pp. 8–9

4.5 Case Study

  • For three SVAMP queries and 20 semantically equivalent variants per query, SemCoT’s first-token embeddings cluster more tightly than COCONUT’s, suggesting robustness to surface-level query differences.The comparison uses PCA as a proxy for semantic alignment because directly decoding implicit reasoning into human language is challenging.Source: §4.5 · p. 9, Figure 5 · p. 9

5 Related Work

Related work covers implicit CoT efficiency methods and sentence-level semantic similarity methods. The paper identifies limited attention to token-generation speed and to comparing LLM embeddings with natural-language text.

Efficient CoT with Implicit Reasoning

  • Implicit-CoT approaches include progressive encoding, partial reasoning compression, small-LM generation, and self-distillation, but the authors say they give little attention to token-level speed and semantic alignment.Examples include CCoT [3], Token Assorted [43], SoftCoT [53], and CODI [41].Source: §5 · p. 9

Semantic Textual Similarity

  • Sentence-level methods such as SBERT [38], Universal Sentence Encoder, and InferSent [6] produce comparable sentence representations, but cannot compare LLM embeddings directly with natural-language text.The passage also describes the progression from lexical measures to word-level and sentence-level semantic representations.Source: §5 · pp. 9–10

6 Conclusion

The conclusion presents SemCoT as a framework for accelerating Chain-of-Thought reasoning while preserving semantic alignment between implicit and explicit reasoning.

6 Conclusion

  • Experiments across multiple datasets and LLMs show superior efficiency and effectiveness relative to state-of-the-art baselines.The conclusion frames this as making advanced reasoning more computationally accessible for real-world applications.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist answer is yes: it says the abstract and introduction accurately reflect the paper’s contributions and scope.

NeurIPS Paper Checklist

  • The checklist affirms that the paper’s stated focus on semantic alignment and token-level generation speed matches its reported contributions and results.The checklist also says the abstract and introduction accurately reflect the paper’s scope.Source: NeurIPS Paper Checklist · p. 14

2. Limitations

The reproducibility checklist reports that the paper provides training and test details, makes code and data available, and reports variability across runs. It also states that the work has no formal theoretical results requiring proofs.

2. Limitations

  • The authors state that the methodology is algorithmic and empirical, without formal theoretical results requiring proofs.The checklist marks the theory-assumptions-and-proofs item as not applicable.Source: §2 · pp. 14–15
  • The paper reports open access to code and standard publicly available evaluation datasets.The checklist justification points to an anonymized code repository.Source: §2 · p. 16
  • The checklist says necessary training and test details are specified, including dataset splits, architectures, hyperparameters, and optimizer settings.It also identifies PyTorch and Huggingface as the implementation frameworks.Source: §2 · p. 16
  • Experimental tables and figures report standard deviations across multiple runs.The checklist marks the experiment-significance item Yes and says error bars are reported.Source: §2 · p. 16
  • The checklist states that the research conforms to the NeurIPS Code of Ethics and involves no human-subject research.It says the evaluation uses existing benchmark datasets rather than newly collected human-annotated data.Source: §2 · pp. 17, 19, 20

A Limitations

The authors identify deployment overhead, evaluation scope, architecture coverage, and interpretability as limitations of SemCoT. They also describe potential benefits from efficiency alongside risks of misuse and speed-prioritized errors.

A Limitations

  • The customized sentence transformer adds pre-deployment training overhead, which may challenge resource-constrained environments.Evaluation centers on standard mathematical, commonsense, and symbolic benchmarks; specialized domains and extremely long reasoning chains remain unexplored.Source: A Limitations · p. 21
  • Generalization beyond the tested Llama-2 and Mistral architectures remains to be investigated, and latent reasoning tokens reduce human interpretability.The authors also note that some reasoning patterns may make the implicit-versus-explicit trade-off less favorable.Source: A Limitations · p. 21

B Broader Impact

  • The authors associate more efficient reasoning with lower computational costs and energy use, and potentially broader access in resource-constrained settings.They also identify possible use at scale for misinformation or automated cyberattacks, premature deployment, and errors if speed outweighs quality in specific contexts.Source: B Broader Impact · p. 21

C Implementation Details

The implementation details cover benchmark examples and metadata, a one-token evaluation setting, and experimental configuration. The paper also reports its hardware and asset-license information.

C Implementation Details

  • Table 3 reports benchmark dataset sizes for train and test sets and their reasoning types.Table 2 provides sample questions and answers from benchmark datasets.Source: C Implementation Details · p. 21, Table 2 · p. 22, Table 3 · p. 24
  • Experiments use NVIDIA H100 80GB GPUs running CUDA 12.4, and the paper lists licenses for the adopted and lightweight language models.The listed licenses include the Llama 2 Community License Agreement and Apache 2.0.Source: C Implementation Details · p. 23
  • Table 1 measurements average accuracy and time across three independent rounds; training allots five implicit reasoning tokens to baselines and SemCoT.The implementation uses PyTorch and Huggingface, with hyperparameter settings listed in the repository.Source: C Implementation Details · p. 23
  • The evaluation limits methods to one implicit reasoning token to avoid confounds such as directly encoding the answer or excessively encoding model knowledge.The paper includes sample input-and-reasoning pairs in Table 5 and sample benchmark questions and answers in Table 2.Source: Table 2 · p. 22, C Implementation Details · p. 24

D.1 Ablation Study

Supplementary ablations report that SemCoT performs better than its three variants in almost all dataset-and-model configurations. Supplementary parameter analysis reports observations consistent with the main paper's Section 4.4.

D.1 Ablation Study

  • SemCoT performs better than all three variants in almost all configurations across five datasets and both tested LLMs.The supplementary ablations cover Llama-2-7b-chat-hf and Mistral-7B-Instruct-v0.2; Figures 6 and 7 show the respective results.Source: Figure 6 · p. 24, §D.1 · p. 24, Figure 7 · p. 26

D.2 Parameter Analysis

  • Supplementary parameter analysis across adopted datasets and LLMs yields observations generally consistent with Section 4.4.Figure 8 shows Llama-2-7b-chat-hf accuracy and inference time as the number of implicit tokens varies; the prose identifies Figure 9 as the Mistral results.Source: Figure 8 · p. 26, §D.2 · p. 26

D.3 Case Study: Semantic Alignment Analysis

The case study examines whether semantically aligned queries produce consistent and distinguishable implicit reasoning. SemCoT is reported to maintain low variation across aligned queries while separating samples and reasoning domains in implicit-reasoning space.

D.3 Case Study: Semantic Alignment Analysis

  • The case study randomly selects three samples per dataset and generates semantically aligned queries for each sample twenty times.The authors assess whether aligned queries induce aligned reasoning, whether variants and different samples remain distinguishable, and whether semantic domains separate.Source: §D.3 · pp. 26–27
  • Figure 9 analyzes accuracy and inference time as the number of implicit tokens varies for Mistral-7B-Instruct-v0.2.The caption specifies the quantities varied and measured but gives no numerical outcomes.Source: Figure 9 · p. 27
  • SemCoT alone maintains low implicit-reasoning variance under semantically aligned queries while separating samples and mathematical reasoning from other domains.The analysis compares implicit reasoning across aligned-query variants, samples, and semantic domains.Source: §D.3 · p. 27, Figure 12 · p. 29, Figure 13 · p. 29
  • Figures 10–15 show case studies for SoftCoT, SemCoT, and COCONUT, with each method represented on Llama-2-7b-chat-hf [46] and Mistral-7B-Instruct-v0.2 [19].Source: Figure 10 · p. 28, Figure 11 · p. 28, Figure 12 · p. 29
Loading 2510.24940v3…