PaperlayerView exact revision on arXiv

Source-linked AI summary

Carryover Drafting: Recycling Rejected States for Speculative Decoding

Jahyun Koo, Sunghyeon Woo, Jaeeun Kil, Jeongtae Lee, Sungjae Lee, Kyomin Jung, Minsub Kim

arXiv:2609.14717v1cs.LGcs.AI

TL;DR

Speculative decoding computes hidden states for rejected tokens, but conventional drafters discard them; using them must address inference overhead and training alignment. Carryover Drafting reuses rejected states as temporary KV context and trains with parallel draft–verify–draft transitions. Across two target models and eight datasets, it improves average acceptance length and end-to-end vLLM speedup over corresponding baselines.

  • Problem

    Conventional speculative drafters discard rejected-token hidden states computed during verification, while recycling them raises inference-overhead and training-alignment challenges.

    Source: §1 · p. 2

  • Method

    Carryover Drafting reuses rejected target hidden states as temporary KV context and uses parallel draft–verify–draft training to expose the drafter to inference-aligned states.

    Source: §1 · p. 2

  • Results

    6.5–14.7% higher average acceptance length and 7.9–14.4% higher end-to-end vLLM speedup over corresponding baselines across two target models and eight datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Carryover improves acceptance length and end-to-end speedup across different target models and drafting regimes.

    Source: §8 · p. 9

    Carryover is not evaluated with confidence scheduling; doing so would require matching training and inference scheduling policies and may require recalibrating confidence estimates and thresholds.

    Source: §6 · p. 8

Abstract

from arXiv · show

Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to accept several tokens. By construction, verification computes representations for both accepted and rejected tokens. Yet, conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We find that these discarded hidden states generated during target forward retain useful information about future tokens that can improve subsequent drafts. However, realizing this opportunity poses two distinct challenges. At inference, recycling overhead can increase drafting latency, diminishing the speedup gained from increased acceptance length. During training, standard parallel drafter training does not produce inference-aligned rejected states, while obtaining them through sequential rollouts would sacrifice parallelism across training positions. We introduce Carryover Drafting, which addresses both challenges. Carryover recycles rejected target hidden states as temporary KV context, allowing the drafter to selectively attend to them. It reuses the drafter's existing interface and adds only a single learned embedding to distinguish rejected states from committed context. The additional KV context is replaced each drafting round, keeping its length bounded by one proposal block. We introduce parallel draft--verify--draft training that exposes the drafter to inference-aligned rejected states while preserving parallelism across training positions. Experiments with DFlash and a DSpark-derived semi-autoregressive drafter across two target models show that this simple Carryover mechanism improves average acceptance length by 6.5--14.7% and end-to-end vLLM speedup by 7.9--14.4% over the corresponding baselines, with speedup gains reaching 28.8% on translation.

1 INTRODUCTION

Carryover Drafting reuses rejected target hidden states to improve later drafts, while parallel draft–verify–draft training supplies inference-aligned states without losing parallelism across training positions. Across two target models and eight datasets, it improves acceptance length and end-to-end speedup over corresponding baselines.

1 INTRODUCTION

  • Rejected-token hidden states are computed during verification but discarded, even though they may help improve subsequent drafts.Better drafts allow more tokens to be accepted per target pass, making draft quality central to acceleration.Source: §1 · p. 2
  • Carryover reuses rejected target hidden states as temporary KV context through existing projections, with one learned embedding distinguishing them from accepted context.Replacing carried-over states each round bounds additional KV context to one proposal block.Source: §1 · p. 2
  • Parallel draft–verify–draft training exposes the drafter to inference-aligned rejected states while keeping training parallel across positions.Each transition uses a frozen target to verify detached proposals, then trains a second draft against the recorded continuation at the shifted anchor.Source: §1 · p. 2
  • The training procedure retains recorded responses as supervision rather than generating new target continuations.Intermediate draft–verify stages provide shifted positions and rejected states for carryover training.Source: §1 · p. 2, Figure 2 · p. 3
  • 6.5–14.7% higher average acceptance length and 7.9–14.4% higher end-to-end vLLM speedup over corresponding baselines across two target models and eight datasets.Speedup gains reach 28.8% on translation.Source: §1 · p. 2

2 CARRYOVER DRAFTING

Carryover adds rejected target hidden states to the next draft’s KV context through the existing drafter interface. The section describes one-round reuse and a parallel training procedure that constructs inference-aligned carryover inputs from recorded responses.

2 CARRYOVER DRAFTING

  • Carryover reuses the baseline’s selected target-layer hidden states and drafter projections, while added drafting latency means acceptance gains must outweigh reuse costs.The design motivates reusing existing projections and replacing rejected context each round to keep added latency minimal.Source: §2 · p. 3

2.1 ONE-ROUND REUSE THROUGH THE EXISTING INTERFACE

  • Rejected states are distinguished from accepted context by one learned embedding before projection, with the drafter and existing projections fine-tuned.A dedicated projector gives slightly higher acceptance in an interface probe but adds another projection module to the drafting path.Source: §2.1 · p. 4
  • Each round carries over rejected target states as read-only drafter context, replaces the prior carried context, and keeps added context bounded by B rows per layer.Full acceptance leaves no rejected states to carry over; the carried states neither replace accepted-token hidden states nor enter the target KV cache.Source: §2.1 · pp. 3–4

2.2 PARALLEL TRAINING ON RECORDED TARGET RESPONSES

  • At each sampled anchor, detached proposals are verified in parallel by a frozen target, and rejected states feed a second draft at the shifted position.Gradients flow only through the second drafter pass; each anchor supplies one supervised transition on the same recorded response, preserving parallelism across anchors.Source: §2.2 · p. 4

3 EXPERIMENTAL SETUP

The experiments compare DFlash and a semi-autoregressive Markov drafter across two target models, using shared training data and paired evaluation settings. Table 1 reports acceptance and speedup, with average speedup measured separately from per-dataset speedups.

3 EXPERIMENTAL SETUP

  • DFlash and Markov are evaluated with Qwen3-4B and Gemma 4 12B; both drafters use five layers and propose 15 tokens, with confidence scheduling disabled.Markov adds the rank-256 first-order correction head to the parallel backbone.Source: §3 · p. 4
  • All variants use the same 500K prompts and target-regenerated responses, with baselines trained for 10 epochs and Carryover for three additional epochs from baseline checkpoints.The prompt mix is UltraChat-200K and Magpie-Llama-3.1-Pro-300K-Filtered.Source: §3 · p. 4
  • Table 1 reports acceptance and speedup over autoregressive decoding; its per-dataset speedups are offline-vLLM, while Avg. speedup is a separate online-vLLM measurement.Avg. acceptance is a micro average over 844 prompts.Source: Table 1 · p. 5
  • Evaluation uses 844 Red Hat speculator prompts at T = 0 and T = 1, with acceptance length including verification bonus tokens.Measurements use vLLM on one H200 GPU; baseline–Carryover pairs share settings.Source: §3 · p. 5

4 RESULTS

Carryover improves acceptance and end-to-end speedup across drafting regimes, with the largest relative acceptance gains on translation. Interface ablations favor recycling target hidden states through KV context, while concurrency tests show gains persist under heavier serving loads.

4.1 ACCEPTANCE ACROSS TASKS AND DRAFTING REGIMES

  • Translation has the largest relative acceptance gain in every target–drafter–temperature combination, reaching 28.6% for Qwen3-4B Markov at T = 1.Figure 3 compares acceptance length across datasets at T = 0; Appendix F reports T = 1.Source: §4.1 · p. 5
  • Carryover raises average acceptance length by 6.5–14.7% and end-to-end speedup by 7.9–14.4% over corresponding baselines.The acceptance comparison spans evaluated datasets; speedup includes projection and attention overhead for carried states.Source: §4.1 · p. 5, §4.2 · p. 6, Figure 3 · p. 6

4.2 END-TO-END SPEEDUP AND THROUGHPUT

  • As concurrency rises from 1 to 32, both methods deliver smaller speedups over autoregressive decoding, but Carryover retains relative gains across targets and drafting regimes.These experiments use fixed verification lengths.Source: §4.2 · p. 6

4.3 CARRYOVER INTERFACE ABLATION

  • Among the tested carryover designs, recycling target hidden states gives the largest gains, while token hints give inconsistent or modest improvements.Token hints enter through initial proposal embeddings, whereas carried hidden states enter as KV context accessed by existing attention layers, so the comparison changes both information and access path.Source: §4.3 · p. 6
  • The single carryover embedding retains nearly the same acceptance as a 19.7M-parameter projector, while reuse without an explicit distinction from accepted context remains competitive.The embedding is chosen to avoid an additional projection module.Source: §4.3 · p. 6

5 HOW DO REJECTED STATES CONTRIBUTE TO DRAFTING?

Section 5 tests rejected states at matched decoding anchors and examines which properties of rejected branches are associated with larger Carryover gains. It finds improved target-token recall when states are supplied, and stronger associations with re-convergence than with rejected-state count or post-rejection acceptance.

5.1 TARGET-TOKEN RECALL AT MATCHED ANCHORS

  • The matched-anchor test compares ON, OFF, and Baseline using shared committed tokens, accepted target states, and target KV cache; only ON receives recorded rejected states.Recall@1 and Recall@16 measure top-target-token inclusion among the top 1 and top 16 draft candidates, respectively.Source: §5.1 · pp. 6–7
  • Supplying rejected states raises ON over OFF by 1.28–5.60 Recall@1 points and 0.29–1.89 Recall@16 points at every displayed position.Gains begin at position 0, where variants share the accepted anchor, and persist across the block.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 WHY DO GAINS DIFFER ACROSS DATASETS?

  • Carryover gains associate more closely with re-convergence than with rejected-state count or post-rejection acceptance.Re-convergence counts verifier top-1 predictions from rejected states matching the eventual committed token, allowing a one-position tolerance; it is not used to select states or supervise training.Source: §5.2 · pp. 7–8
  • Markov produces more re-converged positions than DFlash on all eight datasets (2.76 versus 2.51 per rejected round) and has larger Carryover gains on seven of eight.Higher re-convergence offers one explanation for Markov’s larger gains and suggests stronger drafters may leave more useful rejected states.Source: §5.2 · p. 8

6 DISCUSSION

The discussion identifies deployment settings not evaluated for Carryover and describes possible extensions through tree decoding and target-verified rollouts.

6 DISCUSSION

  • Carryover is not evaluated with confidence scheduling because training must match the inference policy, and changed predictions may require recalibrating confidence estimates and thresholds.The scheduling policy changes which rejected states reach the next draft.Source: §6 · p. 8
  • Figure 5 examines local replay gain by rejected-branch statistics, re-converged positions per round, and normalized re-convergence against dataset-level acceptance gain.The figure reports final Qwen3-4B Markov results at T = 0.Source: Figure 5 · p. 9
  • Tree decoding remains future work, including matching tree-construction policies across training and inference and selecting or combining states from multiple rejected branches.The additional verification could provide more rejected states and information across branches.Source: §6 · pp. 8–9
  • Target-verified rollouts can provide rejected states for Carryover without an extra verification stage and could support retention beyond one round.The parallel training procedure described in this work does not cover retention beyond a single round.Source: §6 · p. 9

7 RELATED WORK

Carryover extends target-conditioned drafting by making rejected target hidden states available, complementing methods that refine the current block or reuse other speculative computation.

7 RELATED WORK

  • Domino, DSpark, xPress, DFlash 2, and D-PACE improve current-block generation through stronger dependencies, candidate refinement, or a changed training objective.Carryover complements these approaches by extending which target hidden states are available after verification.Source: §7 · p. 9
  • Carryover extends target-conditioned drafting by retaining an ordered sequence of rejected target hidden states through the drafter’s existing KV interface.Earlier methods use target hidden states for autoregressive drafting or as KV context for parallel block prediction.Source: §7 · p. 9
  • Prior recycling methods reuse token candidates, Jacobi n-grams, phrases, verifier-derived steering vectors, or drafter hidden states in token trees.Carryover instead reuses rejected target representations as KV context.Source: §7 · p. 9

8 CONCLUSION

Carryover reuses rejected target hidden states through an existing drafter interface, with parallel training to expose inference-aligned states. Across different target models and drafting regimes, it improves acceptance length and end-to-end speedup.

8 CONCLUSION

  • Carryover recycles rejected target hidden states through the drafter’s existing KV interface, using one learned embedding and one round of additional context.The additional context is introduced for the next draft.Source: §8 · p. 9
  • Parallel draft–verify–draft training exposes rejected states without sequential rollouts.The conclusion identifies this as the training approach accompanying Carryover.Source: §8 · p. 9
  • Across different target models and drafting regimes, Carryover improves acceptance length and end-to-end speedup.The conclusion reports consistent gains across these settings.Source: §8 · p. 9

A INFERENCE AND TRAINING ALGORITHMS

The algorithms modify inference by retaining rejected states as temporary drafter context and train on verification-generated states at shifted anchors. The training procedure preserves parallel processing across anchors and continues the corresponding baseline checkpoint.

A INFERENCE AND TRAINING ALGORITHMS

  • At inference, Carryover collects all proposal states, embeds the rejected portion, and applies the existing projections; training shifts supervision anchors after a parallel draft–verify step.The added embedding is the only illustrated change to the interface.Source: A INFERENCE AND TRAINING ALGORITHMS · p. 12

A.1 INFERENCE UPDATE

  • Accepted states extend committed drafter context, while rejected KV rows become read-only context for one next proposal and never enter the target’s committed KV cache.Rejected states retain their original token positions, with RoPE applied once when projected into keys.Source: §A.1 · p. 12
  • Carryover changes the drafter’s proposal distribution but leaves target probabilities and the usual acceptance and residual-correction rule unchanged.Rejected text is input only to the drafter; overwriting committed target KV would break that separation.Source: §A.1 · p. 12

A.2 PARALLEL TRAINING UPDATE

  • Training verifies detached proposals at sampled anchors, then trains a second draft on the same recorded response at the shifted correction-or-bonus position.Verification supplies rejected-state inputs and the shift, without generating a new target response or replacing continuation labels.Source: §A.2 · p. 13
  • Anchors proceed in parallel within each training stage, with branch-isolating causal masks; fully accepted proposals still supply supervision using empty carried-over context.The first drafter pass and target verification are detached, while the second drafter pass trains the drafter and embedding.Source: §A.2 · p. 13

B.1 TRAINING DATA AND SETTINGS

  • Carryover continues the corresponding Baseline-10 checkpoint for three epochs with fresh optimizer and scheduler states, while detached proposals and frozen-target prefill add computation.The target responses remain fixed, preserving parallelism across anchors.Source: §B.1 · p. 14

B.2 EVALUATION PROTOCOLS

The diagnostics link normalized re-convergence more closely to final dataset-level gain than broader acceptance after rejection, while showing that drafters attend to carried states. The evaluation protocol separates offline dataset measurements from online serving speedup.

B.2 EVALUATION PROTOCOLS

  • The protocol reports offline dataset rows separately from online serving speedup, and does not average the two measurement types together.Dataset rows use batch size 1; online speedup uses client concurrency 1 on the mixed 844-prompt workload, while the throughput sweep spans concurrency 1–32.Source: Table 5 · p. 15, §B.2 · p. 15
  • Matched-anchor recall compares all three variants on rounds where ON and Baseline match the target through the preceding position; the evaluated rounds vary by position.Recall is micro-averaged over those rounds, with error bars from a normal approximation.Source: §B.2 · p. 15

C ATTENTION TO CARRIED STATES

D RE-CONVERGENCE AND ACCEPTANCE AFTER REJECTION

  • Normalized re-convergence correlates with final dataset-level gain at r = 0.96 (0.90 without Translation), versus 0.52 (0.71) for all acceptance after rejection.Rejected-state count normalized by Baseline acceptance correlates at r = 0.74 across all datasets and 0.84 without Translation.Source: D RE-CONVERGENCE AND ACCEPTANCE AFTER… · p. 17, Figure 7 · p. 17

E RE-CONVERGENCE ACROSS DRAFTER TYPES

F ACCEPTANCE ACROSS TASKS AND TEMPERATURES

Carryover gains remain positive across the displayed stochastic settings, though their size varies. Figure 9 compares acceptance length at T = 1 across both targets and drafting regimes.

F ACCEPTANCE ACROSS TASKS AND TEMPERATURES

  • At T = 1, Figure 9 compares Baseline and Carryover acceptance length for both targets and drafting regimes.The caption directs readers to Table 1 for numerical results.Source: Figure 9 · p. 18
  • Carryover gains remain positive in every displayed stochastic setting, although their magnitude varies.Qwen3-4B DFlash on RAG gains 1.1%, while Qwen3-4B Markov on translation gains 28.6%.Source: F ACCEPTANCE ACROSS TASKS AND… · p. 18
  • Qwen3-4B DFlash on RAG shows a 1.1% stochastic-setting gain.The gain is positive but modest compared with the Qwen3-4B Markov translation example.Source: F ACCEPTANCE ACROSS TASKS AND… · p. 18
  • Qwen3-4B Markov on translation gains 28.6% in the displayed stochastic setting.The magnitude varies across settings despite positive gains throughout.Source: F ACCEPTANCE ACROSS TASKS AND… · p. 18
Loading 2609.14717v1…