PaperlayerView exact revision on arXiv

Source-linked AI summary

Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time

Wang Yang, Xiang Yue, Vipin Chaudhary, Xiaotian Han

arXiv:2504.12329v2cs.CLcs.AI

TL;DR

Small models offer efficiency but struggle with complex reasoning, while retraining is costly and existing inference-time scaling can be inconsistent. Speculative Thinking selectively delegates reflective reasoning segments to larger models during inference, improving accuracy while often shortening outputs. The approach also improves a non-reasoning model, though its effectiveness depends on choosing a sufficiently capable target model.

  • Problem

    Small models are efficient but underperform on complex reasoning, while post-training is costly and existing inference-time methods often provide limited or inconsistent gains.

  • Method

    Speculative Thinking uses a small model for most generation and selectively delegates structurally marked difficult reasoning segments to a stronger model.

  • Results

    +6.2% on MATH500 and a 15.7% reduction in average output length were reported for the 1.5B model assisted by a 32B model; Qwen-2.5-7B-Instruct improved from 74.0% to 81.8% on MATH500.

  • Takeaways & Limitations

    The framework combines small-model efficiency with large-model reasoning strength for inference-time augmentation of reasoning and non-reasoning models.

  • Takeaways & Limitations

    The method depends on structural cues whose stability across model sizes, families, and training setups remains uncertain, and its hyperparameters may require task-specific tuning.

Abstract

from arXiv · show

Recent advances leverage post-training to enhance model reasoning performance, which typically requires costly training pipelines and still suffers from inefficient, overly lengthy outputs. We introduce Speculative Thinking, a training-free framework that enables large reasoning models to guide smaller ones during inference at the reasoning level, distinct from speculative decoding, which operates at the token level. Our approach is based on two observations: (1) reasoning-supportive tokens such as "wait" frequently appear after structural delimiters like "\n\n", serving as signals for reflection or continuation; and (2) larger models exhibit stronger control over reflective behavior, reducing unnecessary backtracking while improving reasoning quality. By strategically delegating reflective steps to a more capable model, our method significantly boosts the reasoning accuracy of reasoning models while shortening their output. With the assistance of the 32B reasoning model, the 1.5B model's accuracy on MATH500 increases from 83.2% to 89.4%, marking a substantial improvement of 6.2%. Simultaneously, the average output length is reduced from 5439 tokens to 4583 tokens, representing a 15.7% decrease. Moreover, when applied to a non-reasoning model (Qwen-2.5-7B-Instruct), our framework boosts its accuracy from 74.0% to 81.8% on the same benchmark, achieving a relative improvement of 7.8%.

1 Introduction

Small models are efficient but struggle with complex reasoning, while retraining is costly and inference-time methods often provide limited gains. Speculative Thinking addresses this gap by selectively routing difficult reasoning segments to a stronger model during inference.

  • Smaller language models reduce computational and memory requirements but often underperform on complex reasoning tasks.
  • Post-training methods can improve reasoning but are costly, data-intensive, and difficult to scale.
  • Inference-time scaling avoids retraining but depends on existing model abilities and often yields limited or inconsistent improvements on complex tasks.
  • Speculative Thinking selectively delegates difficult reasoning segments from a small model to a stronger model at structurally marked reflection points.
  • +6.6% on AIME, +6.2% on MATH500, +8.1% on GPQA, and +5.0% on AMC23 were reported for a 1.5B model assisted by a 32B model.Qwen-2.5-7B-Instruct also gained +7.8% on MATH500 and +14.2% on GPQA.

2 Motivations

The paper identifies structural and behavioral signals of inefficient reasoning: delimiters often precede reflective tokens, while smaller models over-reflect and produce longer incorrect responses. These observations motivate delegating difficult reasoning segments to larger models.

  • 2.1 Analysis of LLM Reasoning Process: Reasoning-supportive tokens such as “wait,” “hmm,” and “alternatively” are associated with model self-reflection.
  • 2.1 Analysis of LLM Reasoning Process: Over 80% of “wait” occurrences are preceded by “\n\n,” suggesting that the delimiter acts as a thinking cue.
  • 2.1 Analysis of LLM Reasoning Process: Reflective segments following delimiters can affirm, revise, or state the preceding thought.
  • 2.2 Comparisons between Small and Large Reasoning Models: Smaller reasoning models show lower accuracy and longer outputs, with incorrect responses contributing substantially to excessive length.
  • 2.2 Comparisons between Small and Large Reasoning Models: Reflective phrases occur more frequently in incorrect responses, particularly for smaller models, indicating inefficient exploration through over-reflection and under-reasoning.
  • 2.3 How to Combine Small and Large Reasoning Model?: Larger models reflect more efficiently, producing fewer redundant revisions and shorter outputs overall.
  • 2.3 How to Combine Small and Large Reasoning Model?: Speculative Thinking delegates segments marked by paragraph breaks and reflective phrases to larger models, which handle them more concisely.

3 Method: Speculative Thinking

Speculative Thinking uses a small model for primary generation and selectively invokes a larger target model at reflection, verification, and excessive-reflection triggers. The framework is evaluated using accuracy, output length, and estimated speed.

  • 3 Method: Speculative Thinking: The framework assigns primary reasoning to a speculative small model and selective interventions to a larger target model.
  • 3 Method: Speculative Thinking: Affirmation/Reflection Takeover detects a delimiter, classifies the following sentence, and lets the target model generate the next n1 tokens for affirmation or reflection cases.
  • 3 Method: Speculative Thinking: Verification Takeover invokes the target model for n2 tokens when verification-related cues follow a delimiter.
  • 3 Method: Speculative Thinking: Excessive Reflection Takeover counts negative reflection sentences and prompts the model to exit the reflection loop after a threshold is exceeded.
  • 3 Method: Speculative Thinking: Table 2 evaluates accuracy, average output length, estimated speed, and the proportion of final-output tokens generated by the target model.

4 Experiments

The experiments evaluate accuracy, output length, and estimated inference speed across four benchmarks, testing selective assistance for reasoning and non-reasoning models, cross-family generalization, and comparison with speculative decoding.

  • Experimental Setup: The evaluation measures accuracy, average output length, and estimated inference speed on AIME 2022–2024, GPQA-Diamond, MATH500, and AMC23.Estimated speed is theoretically computed because runtime measurements are affected by backend GPU optimizations.
  • Large Reasoning Models Monitor Small Reasoning Models: Accuracy increases by 6.2% on MATH500, 8.1% on GPQA, 5.0% on AMC23, and 6.6% on AIME for the 1.5B model assisted by the 32B model.Average output length decreases by 15.7%, 3.9%, 16.9%, and 11.7% on the same datasets, respectively.
  • Large Reasoning Models Monitor Small Reasoning Models: The assisted 1.5B model consistently exceeds the standalone 32B model in estimated generation speed while using the larger model selectively.The target model modifies approximately 20% of the speculative model’s output.
  • Reasoning Models Monitor Non-Reasoning Models: Qwen-2.5-7B-Instruct improves from 74.0% to 81.8% accuracy on MATH500 with reasoning-model assistance, but output length increases.Assistance from a 1.5B reasoning model does not produce consistent improvements.
  • Speculative Thinking Generalize to Different Families of Models: Cross-family settings pair a small Qwen model with a Phi model and a small LLaMA3 model with a large Qwen model.Both settings improve performance over the smaller model alone and achieve speedups relative to the larger model.
  • Comparisons between Speculative Decoding and Speculative Thinking: Unlike speculative decoding, Speculative Thinking does not require similar token distributions and intervenes only when necessary rather than regenerating nearly 50% of tokens.The comparison uses a 7B speculative model and a 32B target model, with 20 generated tokens per speculative-decoding step.

5 Related Works

Related work addresses language-model reasoning through reinforcement learning and supervised fine-tuning, while also exploring speculative decoding and methods for reducing excessive reasoning output length.

  • LLM Reasoning: Reasoning-capability enhancement primarily uses reinforcement learning and supervised fine-tuning.The passage cites GRPO-based reasoning improvements and training on selected high-quality reasoning samples.
  • Speculative Decoding: Speculative decoding accelerates generation by having a smaller model propose tokens that a larger model verifies.Its relevance here is as the inspiration for a reasoning-level rather than token-level approach.
  • Efficient Reasoning: Efficient-reasoning research responds to unnecessarily verbose outputs from reasoning-enabled models.Kimi 1.5’s Long-to-Short method is identified as an early effort in this direction.

6 Conclusion

The conclusion presents Speculative Thinking as a training-free way to combine small-model efficiency with large-model reasoning strength through selective delegation during inference.

  • Conclusion: Speculative Thinking uses structurally meaningful points, especially reflection cues like "\n\n", to let larger reasoning models guide smaller ones without additional training.The framework is reported to improve accuracy, output conciseness, and efficiency across four math-reasoning datasets.
  • Conclusion: The framework improves both reasoning and non-reasoning models without additional data or training-computation cost.The conclusion characterizes collaborative inference across models with different capacities as a promising paradigm.

Limitations

The method’s main limitation is dependence on structural cues and manually selected intervention settings whose stability across models, families, training setups, and tasks remains uncertain.

  • Cue stability across model sizes, families, and training setups remains uncertain.
  • The method also depends on classification rules that assign segments to Affirmation, Reflection, or Statement categories based on keyword counts.
  • The framework uses multiple cue categories, including Reflection, Affirmation, and Verification keywords, to trigger different target-model interventions.
  • Intervention lengths are fixed at n1 = 20, n2 = 125, and n3 = 125 for different takeover scenarios.

D Hyperparameter Choices and Ablation Analyses

Speculative Thinking requires careful hyperparameter selection for different model pairs and tasks, with implementation details, empirical justifications, and ablations provided in the appendix.

  • Speculative Thinking requires careful hyperparameter selection for different model pairs and tasks.The appendix provides implementation details, empirical justifications, and ablation results for these choices.

D.1 Classification Keyword Selection

The framework selects semantic keywords from model generations, tests reduced cue sets, and examines threshold and initialization choices that affect performance and efficiency.

  • Semantic cues classify generated segments into Affirmation, Reflection, and Verification categories that guide speculative decisions.
  • Leading tokens after paragraph breaks are frequency-ranked and grouped into three keyword categories.
  • Reduced keyword sets hurt performance, highlighting the benefit of richer lexical cues.
  • A fixed negativity threshold c = 15 balances speed and accuracy, whereas lower thresholds harm speed and higher thresholds hurt accuracy.
  • The initial 100-token target-model generation provides necessary scaffolding for alignment when the speculative model is non-reasoning.

E Results of Non-reasoning model

Speculative Thinking improves non-reasoning models across most evaluated datasets, with especially large gains on AIME and GPQA, while the overall trade-off remains favorable in accuracy and speed.

  • Speculative Thinking improves the 1B-Instruct model’s accuracy across most datasets, with particularly large gains on AIME and GPQA.
  • The overall trade-off remains favorable in accuracy and speed, despite a slight MATH500 performance drop.
  • Table 12 reports accuracy, average output length, estimated speed, and target-model modification ratio for four datasets.

F Results of Deepseek-Distilled Qwen-2.5-7B

Speculative Thinking improves the 7B model’s accuracy across four datasets while reducing average output length. The framework is presented as suitable for deployments balancing performance, cost, latency, and scalability.

  • Results: Across all four datasets, Speculative Thinking consistently improves the 7B model’s accuracy while reducing its average output length.The evaluated datasets are AIME 2020–2024, MATH500, GPQA, and AMC23.
  • Results: The 32B model guides the 7B model toward more concise and accurate responses, with especially pronounced improvements on MATH500 and AMC23.The passage attributes these trends to reasoning-aware alignment.
  • Results: Larger models produce more accurate answers, longer outputs, and higher reflection-token frequencies, whereas smaller models generate shorter responses with fewer reasoning cues.These patterns support assigning reasoning-specialist roles to larger models and faster baseline roles to smaller models.
  • Deployment implications: Speculative Thinking targets scenarios where efficiency, cost, and scalability are prioritized and some accuracy trade-off is acceptable.Proposed contexts include educational QA, enterprise assistants, and edge-cloud hybrid systems.
Loading 2504.12329v2…