PaperlayerView exact revision on arXiv

Source-linked AI summary

DART: Distilling Autoregressive Reasoning to Silent Thought

Nan Jiang, Ziming Wu, De-Chuan Zhan, Fuming Lai, Shaobing Lian

arXiv:2506.11752v2cs.CL

TL;DR

Autoregressive Chain-of-Thought reasoning adds computational cost and latency that hinder use in latency-sensitive applications. DART distills CoT into non-autoregressive Silent Thought using paired training pathways and a Reasoning Evolvement Module. Across benchmarks it balances accuracy and latency; on GSM8K, it reports a 10.1% accuracy gain over non-autoregressive baselines with 1 ms latency overhead.

  • Problem

    Autoregressive CoT adds computational cost and latency, limiting its effectiveness in real-time applications.

    Source: §1 · p. 1

  • Method

    DART distills CoT into non-autoregressive Silent Thought through CoT and ST training pathways, with a Reasoning Evolvement Module aligning their hidden states.

    Source: §1 · pp. 1–2, §3.2 · p. 3

  • Results

    Across benchmarks, DART balances accuracy and latency and is robust; on GSM8K it reports a 10.1% accuracy gain over other non-autoregressive methods with 1 ms latency overhead.

    Source: §4.2 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    DART presents non-autoregressive Silent Thought as a potential alternative to CoT, with a reported balance between accuracy and latency.

    Source: §1 · p. 2, §5 · p. 5

    DART relies on CoT data, which may not always be available; generating synthetic CoT data with large LLMs incurs preprocessing costs.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Chain-of-Thought (CoT) reasoning has significantly advanced Large Language Models (LLMs) in solving complex tasks. However, its autoregressive paradigm leads to significant computational overhead, hindering its deployment in latency-sensitive applications. To address this, we propose \textbf{DART} (\textbf{D}istilling \textbf{A}utoregressive \textbf{R}easoning to Silent \textbf{T}hought), a self-distillation framework that enables LLMs to replace autoregressive CoT with non-autoregressive Silent Thought (ST). Specifically, DART introduces two training pathways: the CoT pathway for traditional reasoning and the ST pathway for generating answers directly from a few ST tokens. The ST pathway utilizes a lightweight Reasoning Evolvement Module (REM) to align its hidden states with the CoT pathway, enabling the ST tokens to evolve into informative embeddings. During inference, only the ST pathway is activated, leveraging evolving ST tokens to deliver the answer directly. Extensive experimental results demonstrate that DART offers significant performance gains compared with existing non-autoregressive baselines without extra inference latency, serving as a feasible alternative for efficient reasoning.

1 Introduction

DART replaces autoregressive Chain-of-Thought with non-autoregressive Silent Thought, using paired training pathways and a Reasoning Evolvement Module to align them. The approach targets CoT's latency cost and limitations in existing implicit-reasoning methods, and is evaluated across multiple reasoning benchmarks.

1 Introduction

  • CoT supports LLM reasoning across tasks, but its autoregressive steps add computational cost and latency that limit real-time use.The passage describes CoT as decomposing complex problems into intermediate reasoning steps.Source: §1 · p. 1
  • Existing implicit- and continuous-thought approaches either have unsatisfactory performance or retain autoregressive generation, limiting efficiency.Continuous-thought methods compress discrete tokens into compact representations, while implicit CoT reasons in hidden states.Source: §1 · p. 1
  • DART trains CoT and Silent Thought pathways, aligns their hidden states with a lightweight Reasoning Evolvement Module, and uses the ST pathway alone to answer at inference.The ST tokens evolve into increasingly informative embeddings as they pass through the REM-equipped pathway.Source: §1 · pp. 1–2
  • The paper presents non-autoregressive ST as a CoT alternative and reports experiments on multiple reasoning benchmarks with efficiency, accuracy, and interpretability evaluation.The stated contributions include exploring ST and introducing DART's REM-based dual-pathway framework.Source: §1 · p. 2

2 Related Work

Related work covers explicit CoT, continuous-thought approaches, and hidden-space reasoning. The cited continuous-thought approaches remain autoregressive, while iCoT is reported to lack scalability for larger models.

2 Related Work

  • CoT developed through supervised fine-tuning and reinforcement learning is effective, but its intermediate steps add computation and reduce throughput.The passage cites supervised fine-tuning (Yue et al., 2024; Yu et al., 2024) and reinforcement learning (Wang et al., 2024; Shao et al., 2024; DeepSeek-AI et al., 2025).Source: §2 · p. 2
  • Coconut uses curriculum learning to generate final-layer hidden states autoregressively, while CODI uses end-to-end distillation; both retain autoregressive generation.The passage identifies these hidden states as continuous thought and describes them as reducing the number of intermediate steps.Source: §2 · p. 2
  • iCoT embeds CoT reasoning in the model's hidden space but lacks scalability for larger models.The passage cites iCoT (Deng et al., 2023, 2024) and the scalability limitation (Shen et al., 2025).Source: §2 · p. 2

3 Method

DART trains CoT and Silent Thought pathways: one produces explicit reasoning and answers, while the other answers from ST tokens. A lightweight REM aligns ST-pathway hidden states with the CoT pathway to distill reasoning capability.

3.1 Dual-Pathway Architecture

  • DART trains a CoT pathway to generate reasoning and answers, alongside an ST pathway that generates answers conditioned on an ST sequence.Each ST token is a special <st> token, and the sequence contains 20 tokens in this paper.Source: §3.1 · p. 2

3.2 REM-based Self-Distillation

  • REM approximates the hidden-state shift induced by autoregressively generated intermediate tokens using a lightweight module at each decoder layer.The approximation uses both model parameters and question context.Source: §3.2 · p. 3
  • REM adds few parameters, captures contextual and domain-specific knowledge through interactions between the question and model parameters, and can be merged into decoder-only LLMs without extra inference-time parameters.The distillation loss guides the learning process.Source: §3.2 · p. 3
  • Aligning ST-pathway hidden states with CoT-pathway states encourages the ST pathway to distill reasoning capability and evolve initially meaningless <st> tokens into informative representations.The paper describes this token evolution as a blur-to-concrete thinking process.Source: §3.2 · p. 3

4 Experiments

Experiments compare DART with autoregressive and non-autoregressive baselines, then examine its component contributions, token-count sensitivity, and performance across tasks and models.

4.1 Experimental Settings

  • Experiments use Llama-3.2-1B fine-tuned on GSM8K-Aug with the final CoT step omitted, and use GSM-HARD, SVAMP, and MultiArith for out-of-distribution evaluation.Baselines include CoT, Coconut, CODI, No-CoT, iCoT, and PauseFT.Source: §4.1 · pp. 3–4

4.2 Main Results

  • DART achieves the best performance among non-autoregressive baselines on GSM8K, with a 10.1% accuracy gain and only 1 ms latency overhead, and consistently outperforms NAR methods on out-of-distribution datasets.Autoregressive methods achieve higher accuracy but have significantly lower inference efficiency due to stepwise generation.Source: §4.2 · p. 4, Table 1 · p. 4
  • DART has comparable total FLOPs to CoT and uses 44,040,192 trainable parameters, versus CODI’s 98,574,336; Coconut uses multi-stage full fine-tuning.Experiments were conducted on eight Nvidia A10 GPUs.Source: Table 2 · p. 4, §4.2 · p. 5
  • Removing Ldistill or ST tokens reduces performance, REM improves reasoning over no added module or vanilla LoRA, and using answer tokens for distillation hinders alignment.The authors attribute the ST-token ablation’s accuracy loss as likely due to losing CoT-derived positional priors.Source: Table 3 · p. 4, §4.2 · p. 5
  • As the number of ST tokens increases, GSM8K accuracy rises rapidly then stabilizes, while inference latency remains negligible.Figure 2 reports accuracy and inference time on GSM8K as C, the number of ST tokens, varies.Source: Figure 2 · p. 4, §4.2 · p. 5

4.3 Further Analysis

  • On ProsQA and CommonsenseQA-CoT, DART can outperform the CoT baseline while maintaining low inference latency.The benchmarks evaluate logic and commonsense reasoning, respectively.Source: §4.3 · p. 5, Table 4 · p. 5
  • Across Llama-3.2-3B, Qwen2.5-1.5B, and GPT2, DART consistently boosts reasoning capability without notable latency.Table 5 reports accuracy and inference time for these base models.Source: §4.3 · p. 5, Table 5 · p. 5
  • Translated ST tokens match ground-truth CoT words 69.9% of the time, increasing to 80.1% among correct-answer cases; they also exhibit a progressive reasoning process similar to CoT.Source: §4.3 · p. 5

5 Conclusion

DART distills CoT data into evolving Silent Thought tokens for non-autoregressive implicit reasoning, balancing accuracy and latency across benchmarks.

5 Conclusion

  • DART uses evolving Silent Thought tokens to perform implicit reasoning non-autoregressively.The framework distills knowledge from CoT data.Source: §5 · p. 5
  • The evolving ST tokens support a balance between accuracy and latency.Source: §5 · p. 5
  • Experiments confirm DART’s robustness across benchmarks and the effectiveness of its design.Source: §5 · p. 5

Limitations

DART’s dual-pathway design requires additional training resources, its current alignment signal is limited, and it depends on CoT data that may be unavailable.

Limitations

  • DART requires more computational resources for training than LoRA, with 2.86% of parameters trainable versus 1.79%.Source: Limitations · p. 6
  • DART aligns only the activation of the last word before the answer, potentially overlooking information in intermediate tokens.The authors note that more comprehensive supervision may improve performance.Source: Limitations · p. 6
  • DART relies on CoT data, which may be unavailable; generating synthetic CoT with large reasoning-capable LLMs adds preprocessing cost.Source: Limitations · p. 6

A Analysis of the Shift Value

The analysis studies how intermediate tokens shift the last separator token’s hidden state and motivates REM’s use of an L1 loss to approximate that shift.

A Analysis of the Shift Value

B Datasets

The appendix points readers to dataset statistics and examples of experimental data.

B Datasets

  • Statistics for the datasets used in the experiments are provided in Table 7.Source: B Datasets · p. 7
  • The appendix provides examples of data used in the experiments.Source: B Datasets · p. 7

GSM8K-Aug

The example asks for net salary after a 7% tax on a monthly salary of $800, and shows the calculation and answer.

  • The example asks for the net salary from a monthly salary of $800 taxed at 7%.Source: GSM8K-Aug · p. 7
  • The worked CoT calculates the tax as 56 from 800*7/100.Source: GSM8K-Aug · p. 7
  • Subtracting 56 from 800 gives the stated net salary answer of 744.Source: GSM8K-Aug · p. 7

GSM-HARD

The section includes GSM-HARD examples, a notation reference, and dataset statistics; it also gives a CommonsenseQA-CoT example with a natural-habitat answer.

GSM-HARD

  • GSM-HARD examples include arithmetic questions answered as 3431580.0, 51.0, and 79, plus a logic question answered “Jack is a boompus.”Source: GSM-HARD · pp. 7, 9
  • Table 6 is described as listing frequently used notations alongside their mathematical meanings.Source: Table 6 · p. 8
  • Table 7 reports dataset statistics and identifies GSM-HARD, SVAMP, and MultiArith as evaluation-only datasets.Source: Table 7 · p. 9

CommonsenseQA-CoT

  • The CommonsenseQA-CoT example selects “natural habitat” because the fox moves from the city into the forest.The other options are described as not matching the fox’s purpose or behavior.Source: CommonsenseQA-CoT · p. 9

C Implementation Details

The implementation uses specified REM and optimization settings, with distinct training schedules for the primary model, other LLMs, GPT2, and Coconut.

C Implementation Details

  • REM uses d = 128 and α = 32, while λ = 20 balances a distillation loss reported as much smaller than the other two losses.Training uses AdamW, cosine annealing after a 3% warm-up, and bfloat16 precision.Source: C Implementation Details · p. 9
  • Llama-3.2-1B is trained for 10 epochs from a learning rate of 8e-4, while Llama-3.2-3B and Qwen2.5-1.5B use 5e-4 with the same other configurations.Source: C Implementation Details · p. 9
  • GPT2 uses a 2e-3 initial learning rate for 40 epochs, and Coconut follows its official implementation with the same epoch count as these experiments.Source: C Implementation Details · p. 9
Loading 2506.11752v2…