Source-linked AI summary
Self-Speculation for Faster Reasoning Models
Ravisri Valluri, Tung Nguyen, Aditya Grover
TL;DR
Long reasoning traces improve complex-task performance but create latency problems for interactive applications. SSR uses partial-CoT distributions to draft while full reasoning continues, adding suffix recovery and iterative drafting; it reports consistent latency reductions while preserving output quality. The paper also identifies scope boundaries where overlap is low or reasoning dominates generation.
Problem
Long reasoning traces improve complex-task performance but increase end-to-end latency, while existing acceleration methods do not use reasoning-workflow structure as a speculation signal.
Method
SSR derives a draft from a partial-CoT answer distribution and verifies it with the same model after full reasoning, using concurrent reasoning, suffix decoding, and iterative draft sampling.
Results
SSR achieves consistent latency reductions on structured and long-form generation tasks while preserving output quality.
Takeaways & Limitations
Reasoning structure can serve as a source of speculation for reducing response-generation latency without auxiliary models or target-model-specific modifications.
Takeaways & Limitations
Gains are limited when reasoning dominates generation time or when high-entropy outputs have low lexical overlap with drafts; concurrency also increases instantaneous batch size and compute bursts.
Abstract
from arXiv · showhide
Large language models (LLMs) are deployed for increasingly complex tasks involving planning and multi-step decision making, but high-quality performance on these tasks often requires generating long reasoning traces. This is a poor fit for latency-sensitive and interactive applications like voice assistants or coding agents, where generation latency can strongly affect user experience. Existing acceleration methods typically focus on token-level generation, without utilizing the structure of reasoning workflows. We introduce SSR: Self-Speculation for Reasoning Models, a training-free self-speculative decoding method that leverages the chain-of-thought (CoT) as a source of speculation. SSR uses the partial-CoT answer distribution as the drafter and the full-CoT distribution as the verifier, deriving both from the same model at different reasoning budgets. This builds on the observation that later partial-CoT responses often exhibit greater semantic and lexical overlap with the full-budget response. Due to this overlap, SSR can accept long draft prefixes at once, leading to large speedups on structured and long-form generation tasks. To further exploit draft-response overlap beyond the contiguous prefix accepted by standard speculative decoding, SSR also incorporates suffix decoding, using the draft to seed a suffix cache and recover useful spans beyond the accepted prefix, further reducing latency on tasks with high lexical overlap between the draft and the final response. We evaluate SSR on multiple structured and long-form generation tasks where it is most useful, and demonstrate a relative improvement of up to 24.1% on total generation latency for popular open-source models such as Qwen3.5 and Gemma-4.
1 Introduction
SSR uses partial chain-of-thought answer distributions to draft responses while full reasoning continues, then verifies and extends useful draft spans. It targets latency in long-reasoning applications through suffix decoding and iterative drafting, without training or model-specific modifications.
- Motivation: Reasoning models improve complex-task performance with long traces, but those traces increase latency in interactive applications.The motivation includes planning, coding, decision making, chatbots, voice assistants, and coding agents.
- Motivation: Standard speculative decoding verifies multiple tokens in parallel, but requires a draft that is cheap and sufficiently aligned with the target.Accepted tokens reduce sequential decoding steps when the draft is accurate.
- SSR: SSR uses a partial-CoT answer distribution as the draft and the completed-CoT distribution as verifier within the same model.Draft generation and continued reasoning proceed concurrently, hiding drafting overhead behind ongoing CoT generation.
- SSR: Suffix decoding recovers draft spans beyond the first rejected token, addressing long-form outputs whose responses share later spans despite early disagreements.Standard left-to-right verification otherwise discards content after the first mismatch.
- SSR: An iterative SSR variant samples drafts at multiple CoT points, bootstrapping later drafts from earlier ones when the optimal sampling point is unknown.This reduces multi-point sampling overhead and increases the chance of substantial draft–answer overlap.
- Contributions: SSR is training-free and requires no auxiliary parameters or target-model-specific modifications, with an implementation designed as a serving-stack addition.The reported evaluation covers HumanEval, ClassEval, and LongProc, with latency reductions of up to 24.1% while preserving output quality.
2 Related Work
Prior speculative-decoding research reduces drafting cost through smaller models, target-model components, approximations, or text reuse. Test-time reasoning work controls or monitors CoT budgets, but generally does not reuse intermediate outputs for subsequent generation.
- Speculative decoding: Speculative decoding reduces repeated target-model decode steps by proposing tokens with a draft model and verifying them in parallel.Exact formulations preserve the target sampling distribution, while later work also studies lossy variants and relaxed verification.
- Draft model construction: Smaller draft models are a baseline, while lightweight draft heads and feature predictors reduce memory and computation but require target-model-specific training or architectural changes.These approaches share target-model parameters or computation to reduce drafting overhead.
- Self-speculative decoding: Self-speculative methods eliminate a separate draft model through early exiting, sparsity, or quantization, with training requirements varying by approximation.Quantization- and sparsity-based variants can be training-free, whereas some early-exit methods require training.
- Model-free methods: Model-free methods propose text spans from prompt or generation history, offering negligible drafting overhead in domains with high lexical overlap.Suffix decoding indexes contexts and retrieves matching continuations without a draft distribution.
- Reasoning-time computation: Budget forcing and stability-based methods control reasoning length or stop redundant computation, while primarily treating intermediate outputs as predictions or endpoints.They generally do not reuse intermediate-budget outputs as computation for subsequent generation.
3 Preliminaries
The preliminaries define reasoning outputs as a chain-of-thought followed by a final answer, with partial reasoning budgets inducing conditional answer distributions. They then describe speculative verification, residual sampling, and suffix-cache proposals.
- Autoregressive reasoning language models: A reasoning model samples a chain-of-thought until an end-of-thinking token, then samples the final response conditioned on the query and reasoning trace.Budget forcing samples a response after a specified number of CoT tokens.
- Autoregressive reasoning language models: For a reasoning budget b, pθ(a | q, b) denotes the answer distribution conditioned on the first b reasoning tokens.The full output sequence concatenates the reasoning trace r and final answer a.
- Speculative decoding: Speculative decoding uses a draft distribution d to propose tokens and a target distribution p to verify them sequentially until the first rejection.The accepted tokens form the accepted prefix.
- Speculative decoding: Exact verification accepts draft tokens according to the target-to-draft probability relation and samples rejected tokens from the renormalized positive residual p−d.Acceptance and residual sampling preserve the target output distribution.
- Speculative decoding: High acceptance rates allow many draft tokens to be accepted in one verifier forward pass, reducing expensive sequential decoding steps.Approximate variants may resample from p upon rejection instead of maintaining the residual distribution.
- Suffix decoding: Suffix decoding retrieves continuations from a cache of prompt and previously generated suffixes when no draft distribution is available.Because proposed tokens lack draft probabilities, exact rejection sampling does not apply.
4 Method
SSR uses one reasoning model at two CoT budgets to draft and verify answers concurrently, then combines prefix verification with suffix reuse to accelerate continuation. Its multi-stage extension progressively reuses intermediate outputs, while serving-time gains depend on sufficient reasoning budget and overlap.
- Core self-speculation: SSR uses partial-CoT answer distribution pθ(a | q, bd) as the draft and full-CoT distribution pθ(a | q, bv) as the verifier, with bd < bv.Draft generation and continued reasoning proceed concurrently, hiding drafting latency behind ongoing CoT generation.
- Budget requirements: Concurrent drafting requires |r| > bd + m, and useful drafts generally require bd to be a substantial fraction of total CoT length.Small draft budgets can produce divergent drafts in high-entropy tasks because partial reasoning has not converged toward the final answer.
- Single-shot procedure: The procedure generates reasoning to bd, samples a draft concurrently, collects it at bv, verifies its longest matching prefix, and samples a continuation.The accepted prefix is retained before autoregressive continuation under the verifier distribution.
- Suffix decoding: Suffix decoding reuses draft spans beyond the first rejected token by building a suffix cache and greedily verifying proposed spans during continuation.This is most effective when early disagreements are minor but later draft and final responses share long spans, as in coding and planning.
- Serving-time implementation: Serving-time speedups hide draft latency rather than reduce total FLOPs, because parent and child requests overlap and can increase instantaneous batch size.Prefix caching avoids redundant computation over their shared prompt and partial-CoT prefix.
- Multi-stage extension: Multi-stage decoding iteratively verifies drafts under progressively larger budgets, reusing intermediate outputs and reducing sensitivity to advance selection of a single draft budget.Regular draft updates make the budget choice less brittle when total CoT length is unknown.
5 Experiments
SSR is evaluated end-to-end on structured and long-form generation tasks, comparing latency with matched naive autoregressive generation. Results show the largest benefits when answers are long or structured, while concurrent drafting and suffix reuse help reduce overhead and recover overlapping text.
- Experimental setup: SSR is evaluated end-to-end against naive autoregressive generation with the same model and sampling settings across coding and long-form benchmarks.The evaluation reports speedup, absolute latency, accepted prefix tokens, and suffix-accepted tokens.
- End-to-end results: 24.1% latency reduction is achieved by Gemma-4-E4B-it on ClassEval, while Qwen3.5-4B achieves 18.5%.LongProc 2K improves by 7.1%–9.1%, and HumanEval improves by 2.9%–14.6%.
- End-to-end results: HumanEval shows smaller gains because SSR accelerates answer generation while CoT generation dominates latency for its relatively short completions.For Qwen3.5-4B, HumanEval CoT generation takes 20.9 seconds versus 4.6 seconds for continuation.
- Latency breakdown: Concurrent drafting hides most drafting cost, with under 2% effective overhead for Qwen3.5-4B and virtually no degradation for Gemma-4-E4B-it.Drafting runs concurrently with CoT generation, while verification takes 0.07–0.24 seconds.
- Ablations: 24.1% latency reduction is obtained by suffix-only decoding on ClassEval, compared with 7.9% for prefix-only verification.Suffix decoding can recover exact lexical overlaps that prefix verification would discard after the first mismatch.
- Iterative SSR: 11.9% and 3.3% reductions in draft-child wall time are achieved by iterative SSR at settings (i=750, m=500) and (i=500, m=250), respectively.Later drafts are more likely to overlap with the final answer, allowing bootstrapped drafts to complete with fewer tokens.
6 Limitations and Future Work
SSR’s gains are scope-dependent: it helps most when answer generation is comparable to reasoning, while reasoning-dominated and high-entropy tasks limit reuse.
- SSR gains are limited when reasoning dominates generation time because the method does not accelerate chain-of-thought generation.Its effectiveness is highest when answer length is comparable to CoT length.
- High-entropy tasks can limit SSR because semantic similarity may not produce enough lexical overlap for prefix or suffix reuse.The authors identify accelerating CoT generation and earlier structural commitment as future directions.
7 Conclusion
The paper concludes that SSR uses reasoning structure to reduce generation latency without auxiliary models or target-specific modifications. Suffix decoding extends reuse beyond the accepted prefix, supporting latency reductions on structured and long-form tasks while preserving output quality.
- The same model supplies draft and verifier distributions at two reasoning budgets, while concurrent reasoning hides draft-generation overhead.
- Suffix decoding recovers useful draft content beyond the accepted prefix, contributing to latency reductions on structured and long-form generation tasks.The conclusion states that output quality is preserved.
- SSR’s approach treats reasoning structure as a resource for efficient generation without auxiliary models or target-model-specific modifications.
A Serving-Time Implementation
The serving implementation coordinates parent reasoning, draft children, verification, and continuation through ordinary vLLM requests and scheduler metadata.
- At the draft budget, the scheduler spawns a drafting child from the shared reasoning prefix while the parent continues reasoning.The child appends model-specific end-of-thinking tokens and generates within a configured draft-token budget.
- Multi-drafting mode creates draft children at fixed token increments and selects the latest usable draft after the parent finishes.Outstanding draft children are removed at that point.
- Verification compares prompt and draft log probabilities to accept a prefix, then continuation generates remaining tokens and may use the draft as a suffix cache.These stages are coordinated as ordinary vLLM requests using scheduler metadata.
B Inference Hyperparameters
The inference experiments use controlled single-request vLLM settings and model-specific thinking boundaries to define verification and continuation points.
- Latency experiments use batch size 1, one vLLM worker per GPU, and a separate non-speculative baseline generating the same total response-token count.Exploratory runs use eager mode, no warmup, and one measured run.
- Qwen-style models use </think> as the end-of-thinking delimiter, while Gemma-4-E4B-it uses <channel|>.
- When a model emits its thinking boundary before the cap, the CoT is truncated and verification begins from the resulting prefix.Continuation may terminate at EOS before reaching the cap.
C Multi-Stage Self-Speculative Decoding
Multi-stage SSR generates drafts at increasing reasoning budgets, verifies them against higher-budget reasoning, and reuses accepted material through prefix and suffix caching.
- Multi-Stage Self-Speculative Decoding: The procedure generates reasoning at successive budgets b1 < b2 < ··· < bT and starts draft generation from each partial reasoning state.Each stage continues reasoning to the next budget while collecting the preceding draft.
- Multi-Stage Self-Speculative Decoding: Each draft is verified under the higher-budget context, accepting its longest matching prefix.The accepted prefix is reused to bootstrap the next draft rather than generating it from scratch.
- Multi-Stage Self-Speculative Decoding: The final continuation is sampled from the highest-budget reasoning context and incorporates the previously accepted prefix.Suffix decoding uses the updated cache during continuation, after which the response is returned as a concatenation.
NeurIPS Paper Checklist
The NeurIPS checklist requires authors to report claims, limitations, reproducibility, experimental significance, and societal-impact considerations. It also specifies submission-format and documentation requirements for these disclosures.
- Claims and limitations: Authors must ensure that abstract and introduction claims accurately reflect the paper’s contributions, assumptions, limitations, and scope.The checklist asks authors to explain how broadly empirical results can be expected to generalize.
- Claims and limitations: The paper should discuss limitations, strong assumptions, robustness to assumption violations, scope boundaries, and factors influencing performance.The checklist also asks authors to address computational efficiency and scaling where relevant.
- Reproducibility: Experimental papers should provide enough core-setting detail to interpret results and support reproducibility through code, data, or another reasonable avenue.The checklist allows full details in code, appendices, or supplemental material, and recognizes that open access may not always be possible.
- Experiment statistical significance: Statistical significance reporting should include suitable error bars, confidence intervals, or significance tests for experiments supporting the main claims.The checklist requires explaining captured variability, calculation methods, and assumptions.
- Societal impact and risks: Authors should address societal impact, including potential misuse, fairness, privacy, security, and safeguards for high-risk released models.The checklist expects explanation when societal impact or associated risks are marked not applicable or absent.