PaperlayerView exact revision on arXiv

Source-linked AI summary

Sep-Latent: Separator Tokens as Natural Compression Anchors for Implicit Reasoning

Yinlong Xu, Jianfeng Huang, Zhengdong Zhou, Mingjie Lin, Fuyu Lv, Dan Ou, Hongxia Xu, Jian Wu

arXiv:2610.22924v1cs.AIcs.CL

TL;DR

Explicit CoT brings inference latency and computational overhead, while fixed-length latent methods lack flexibility and risk semantic homogenization. Sep-Latent compresses reasoning into separator-token anchors through attention decay and adaptively terminates latent thinking. It reports 56.8% accuracy on GSM8k using 5.4 tokens on average while outperforming prior implicit methods across math and commonsense benchmarks.

  • Problem

    Explicit CoT incurs inference latency and computational overhead, while existing implicit methods often use fixed numbers of latent tokens and risk semantic homogenization.

    Source: §1 · p. 1

  • Method

    Sep-Latent compresses reasoning into separator-token anchors through attention decay, aligns explicit and implicit pathways with distillation, and adaptively terminates latent thinking based on query complexity.

    Source: §1 · pp. 1–2

  • Results

    56.8% accuracy on GSM8k using only 5.4 tokens on average, with Sep-Latent significantly outperforming previous implicit methods across math and commonsense benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present Sep-Latent as an efficient framework for adaptive internalized reasoning that compresses explicit chains into structured latent space without textual-generation overhead.

    Source: §7 · p. 8

    Sep-Latent does not yet surpass explicit SFT-CoT in accuracy, and textual chains remain a more stable supervision signal for some complex logical tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning through explicit Chain-of-Thought (CoT), but the associated computational overhead and token verbosity remain significant barriers to efficiency. While implicit reasoning in continuous latent spaces offers a promising alternative, existing methods like Coconut and CODI often rely on fixed-length latent sequences and suffer from representational collapse. In this paper, we propose Sep-Latent, an adaptive implicit reasoning framework that utilizes separator tokens as structural semantic anchors. By introducing an Attention-Decay mechanism, we force the model to aggregate and compress reasoning logic into these anchors during training. We then align explicit and implicit pathways through multi-level distillation across all layers to ensure logical consistency. Crucially, Sep-Latent enables dynamic reasoning depth by adaptively terminating the latent thinking phase based on query complexity. Experimental results on the GSM8k-Aug benchmark show that Sep-Latent achieves an accuracy of 56.8\% with an average of only 5.4 tokens, significantly outperforming prior fixed-length implicit models in efficiency while narrowing the performance gap with explicit CoT. Further analysis via PCA and symbolic variant testing confirms that our method ensures semantic grounding and exhibits superior robustness to perturbations.

1 Introduction

The introduction positions Sep-Latent against costly explicit CoT and fixed-length implicit methods that risk semantic homogenization, proposing separator-guided compression with attention decay and adaptive latent exits. It reports 56.8% accuracy on GSM8k using 5.4 tokens on average and gains over prior implicit methods across math and commonsense benchmarks.

  • Explicit CoT incurs inference latency and computational overhead, while fixed-length implicit methods lack flexibility and risk semantic homogenization.Source: §1 · p. 1
  • Increasing separator-token attention preserves reasoning accuracy and perplexity with negligible variation, motivating Attention-Decay compression into structural anchors.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Online-policy distillation uses compact on-policy latent trajectories to improve reasoning stability while reducing training-memory overhead.Source: §1 · p. 2
  • Supervision on an exit token lets Sep-Latent adapt reasoning depth to query complexity and current reasoning states.Source: §1 · p. 2
  • 56.8% accuracy on GSM8k using only 5.4 tokens on average, with Sep-Latent significantly outperforming previous implicit methods across math and commonsense benchmarks.Source: §1 · p. 2

2 Related Work

Related work covers implicit reasoning through hidden-state alignment, staged internalization, and distillation of explicit reasoning into latent representations. It also reviews reasoning-process distillation, which this paper extends with joint explicit and implicit optimization.

2.1 Implicit Reasoning

  • ICoT (Deng et al., 2023) aligns teacher and student hidden states, while Coconut (Hao et al., 2024) distills explicit reasoning traces into continuous latent space.Source: §2.1 · p. 2
  • The paper contrasts prior implicit-reasoning approaches with attention decay that organizes latent reasoning through structural guidance and semantic compression constraints.Source: §2.1 · p. 3

2.2 Online-Policy Knowledge Distillation

  • Prior reasoning-process distillation includes System 2 Distillation (Yu et al., 2024), which alternates teacher and student roles, while this paper jointly trains explicit and implicit modes for semantic consistency.Source: §2.2 · p. 3

3 Method

Sep-Latent transitions from explicit CoT to implicit reasoning by compressing information into separator tokens and aligning the two paths. Its method combines progressive attention decay, continuous latent inputs, and training objectives for linguistic supervision and representation alignment.

3.1 Notation

  • The notation defines the vocabulary, hidden-state dimension, embedding and language-model head, and represents the input query and explicit reasoning steps as token sequences.It also defines the answer sequence and the model’s attention layers and parameters.Source: §3.1 · p. 3

3.2 Explicit CoT with Attention-Decay

  • Attention-Decay suppresses attention to step-specific tokens so separator hidden states aggregate and propagate necessary reasoning information as “summary latents.”The explicit sequence places separators around the reasoning steps, and the figure describes information gradually compressing into <sep> during training.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Implicit Reasoning Alignment

  • Explicit reasoning steps are replaced by continuous latent tokens, each generated by projecting the previous token’s last-layer hidden state into the current step’s embedding space.This recursive construction bypasses vocabulary constraints while maintaining autoregressive dependencies.Source: §3.3 · p. 4
  • Implicit-path hidden states are aggregated across all layers to align hierarchical reasoning features with the compressed knowledge from the explicit path.The implicit sequence replaces textual reasoning steps with continuous latent tokens.Source: §3.3 · p. 4

3.4 Training Objectives

  • Training combines cross-entropy supervision on both paths with token-level distillation at structural anchors and answer positions.The explicit path is supervised on reasoning steps and the answer, while the implicit path is supervised only on the final answer; distillation aligns separator and latent states as well as terminal states.Source: §3.4 · pp. 4–5

4 Experiments

The experiments train on GSM8k-Aug without natural-language reasoning chains and evaluate mathematical, commonsense, logical, and symbolic-perturbation tasks against representative explicit and implicit baselines.

4.1 Training Datasets

  • GSM8k-Aug expands the original GSM8k to approximately 385k examples with GPT-4, and its training data removes natural-language reasoning chains.This is intended to make models internalize intermediate reasoning as mathematical calculations rather than discrete text.Source: §4.1 · p. 5

4.2 Evaluation Benchmarks

  • Evaluation covers in-domain GSM8k, out-of-domain mathematical tasks, logical and commonsense benchmarks, and GSM-SYM symbolic perturbations.MultiArith, SVAMP, and GSM-Hard test mathematical shifts; LogiQA, CSQA, and StrategyQA assess logical and commonsense reasoning.Source: §4.2 · p. 5

4.3 Baselines

  • Baselines span explicit CoT and direct-answer fine-tuning, plus Coconut, CODI, and SIM-CoT implicit-reasoning methods.Experiments use Llama3-1B/3B/8B models.Source: §4.3 · p. 5

5 Main Results

Sep-Latent is evaluated on in-domain and shifted mathematical reasoning, commonsense tasks, and efficiency. It reports competitive accuracy with fewer reasoning tokens than existing implicit methods.

5.1 In-Domain Mathematical Reasoning

  • 56.8% accuracy on GSM8k-Aug exceeds SIM-CODI’s 52.3% and nears SFT-CoT’s 56.2%.The comparison is reported for in-domain performance on the GSM8k-Aug test set.Source: §5.1 · pp. 5–6, Table 1 · p. 6

5.2 Out-of-Domain Reasoning

  • Sep-Latent consistently outperforms SFT-CoT across the GSM-SYM symbolic-variant benchmarks.The paper attributes the result to robustness against superficial perturbations in problem statements.Source: Figure 3 · p. 6, §5.2 · p. 6
  • Sep-Latent reaches 13.1% accuracy on GSM-Hard and 62.0% on SVAMP, while MultiArith accuracy is 95.7%, near SFT-CoT’s 96.7%.The authors report superior generalization and resilience to structural perturbations on these out-of-domain math tasks.Source: §5.2 · p. 6
  • 61.3% average accuracy on commonsense tasks exceeds CODI’s 59.6% and SIM-CODI’s 60.5%, and nearly matches SFT-Direct’s 61.5%.Sep-Latent also reaches 47.1% on LogiQA, the highest accuracy among the evaluated methods.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Efficiency

  • Sep-Latent achieves competitive 56.8% accuracy with substantially lower training cost than implicit methods such as Coconut and CODI.The efficiency comparison considers training time and GPU memory.Source: §5.3 · p. 7
  • Sep-Latent uses 5.4–5.7 tokens on average at inference, versus 11.9–13.4 for existing implicit methods and over 23 for SFT-CoT.The authors link lower token use to adaptive termination through a learned termination marker.Source: §5.3 · p. 7

6 Ablation Studies

The ablations examine scaling, the Attention-Decay mechanism, and latent-token representations. Results report low token use at larger scales, a role for decay in separator distillation, and structured, vocabulary-grounded latent tokens.

6.1 Scaling to Larger Backbone

  • At larger scales, Sep-Latent reaches 64.7% accuracy on Llama3-3B and 68.8% on Llama3-8B, while token use remains 5.5–5.7.The 3B result exceeds CODI’s 60.8% and SIM-CODI’s 62.3%.Source: §6.1 · p. 7

6.2 Necessity of Attention-Decay

  • Distillation without Attention-Decay produces a significant performance drop, while decay without distillation changes attention flow but has limited accuracy impact.The reported synergy supports Attention-Decay as a structural basis for separator distillation to represent and transfer reasoning logic.Source: §6.2 · pp. 7–8

6.3 Distributions of Latent Tokens

  • PCA visualizations show Sep-Latent tokens close to the vocabulary manifold, with separated distributions indicating distinct reasoning computations.The authors describe these representations as semantically grounded and structured rather than collapsed.Source: Figure 4 · p. 8, §6.3 · p. 8

7 Conclusions

Sep-Latent compresses explicit reasoning into a structured latent space using separator tokens and Attention-Decay, with dual-path distillation to maintain accuracy. It adaptively adjusts reasoning depth, averaging 5.4 tokens on GSM8k.

  • Sep-Latent uses Attention-Decay and separator tokens as semantic anchors to compress explicit reasoning chains into a structured latent space.Dual-path distillation is described as maintaining accuracy without textual-generation overhead.Source: §7 · p. 8
  • Dual-path distillation supports an implicit pathway that maintains accuracy without the overhead of generating textual reasoning.Source: §7 · p. 8
  • 5.4 tokens is the average consumption on GSM8k, while Sep-Latent dynamically adjusts latent reasoning depth.The conclusion also reports robustness against representational collapse.Source: §7 · p. 8

Limitations

The authors report that Sep-Latent remains less accurate than explicit SFT-CoT on some complex logical tasks and suggest RL as a possible improvement. At inference, it transitions adaptively from latent reasoning to standard answer decoding.

Limitations

  • Sep-Latent does not yet surpass explicit SFT-CoT in accuracy, and textual chains may provide more stable supervision for some complex logical tasks.The authors report that correct answers can appear in the output distribution despite greedy-decoding failures, and suggest RL could better align the latent path.Source: Limitations · p. 9

A Inference Mode.

  • At inference, the model deactivates the explicit CoT path and Attention-Decay, then transitions from continuous latent reasoning to discrete answer decoding.Source: A Inference Mode · pp. 10–11
  • The latent phase terminates when the decoded token is the terminal separator, letting the model adjust reasoning depth to query complexity.Source: A Inference Mode · p. 11

B Datasets

The examples contrast explicit textual reasoning with implicit reasoning using latent-token placeholders. The worked duck problem illustrates the explicit calculation and final answer.

B.1 Examples

  • The duck example's explicit path calculates 16-3-4=9, then 9*2=18, and returns 18.Source: §B.1 · p. 11
  • The implicit example presents latent-token placeholders between the think markers and returns the same answer, 18.Source: §B.1 · p. 11
  • The example's latent_token is only a placeholder and does not participate in computation.Source: §B.1 · p. 11

B.2 Benchmark Distribution

The benchmark distribution describes an augmented GSM-Aug training set and out-of-domain tests spanning structural, numerical, symbolic, logical, commonsense, and implicit-strategy reasoning.

  • GSM-Aug expands the original 8.5k training problems to approximately 385k examples through paraphrasing, numerical resampling, and GPT-4 synthetic generation.Figure 7 indicates that most training problems involve 1–4 reasoning steps and that all data are used for training.Source: §B.2 · p. 11, Figure 7 · p. 11
  • SVAMP tests structural robustness through changes in phrasing and problem structure, while GSM-Hard tests arithmetic generalization using much larger numerical values.Source: §B.2 · p. 11
  • GSM-SYM tests stability under symbolic or linguistic perturbations, while LogiQA evaluates high-level deductive and inductive reasoning.Source: §B.2 · p. 11
  • CommonsenseQA assesses world-knowledge and commonsense reasoning, while StrategyQA tests inference of unstated multi-step strategies.Source: §B.2 · pp. 11–12

C implementation Details

The implementation uses a staged Attention-Decay curriculum and specifies the training setup. Its comparison set includes explicit and direct-answer supervision, latent-reasoning methods, and SIM-enhanced variants.

C implementation Details

  • Training uses Hugging Face Transformers, DeepSpeed ZeRO-3, H20 GPUs, three epochs, batch size 128, and an initial learning rate of 1e-4.The learning rate follows linear warmup and cosine annealing; the separator token is "\n".Source: C implementation Details · p. 12
  • γ starts at 1.0, cosine-decays to a target such as 0.1 during the first epoch, then remains fixed for two epochs.The schedule is intended to let the model learn logical mappings before compressing information into separator tokens.Source: C implementation Details · p. 12

D Difference from Baselines

Loading 2610.22924v1…