PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual Correction of Parallel Draftsfor Speculative Decoding

Roy Eisenstadt, Harel Lidar, Tal Gonen, Omer Luxembourg, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman

arXiv:2610.12570v1cs.CL

TL;DR

Parallel drafters predict block positions without the realized preceding tokens, weakening intra-block dependencies and potentially reducing acceptance. DCOCO scores retained adjacent candidates in parallel and selects them prefix-first; it leads average throughput across evaluated configurations, with speedups of up to 6.6× over vanilla decoding.

  • Problem

    Parallel drafting predicts multiple positions without conditioning on realized earlier tokens, weakening intra-block dependencies and potentially reducing draft acceptance.

    Source: §1 · pp. 1–2

  • Method

    DCOCO uses one drafter pass to score contextual compatibility among retained adjacent-position candidates in parallel, then selects candidates prefix-first.

    Source: §1 · p. 2

  • Results

    DCOCO has the best average speedup in all twelve verifier-and-thinking-mode cells, by 8–19% over the strongest baseline in each.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    Prefix-first selection reflects speculative verification’s contiguous-prefix acceptance, where a later position helps only if every preceding position is accepted.

    Source: §1 · p. 2

Abstract

from arXiv · show

Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by the target model. Recent diffusion-based drafters make this paradigm particularly attractive: deep attention-based neural backbones provide strong predictions, while drafting multiple future positions in parallel avoids the sequential cost of autoregressive generation. This parallelism, however, comes at a cost. Tokens within a draft block are predicted without access to the realized identities of preceding draft tokens, removing intra-block dependencies that would naturally be available under autoregressive generation. We introduce DCOCO (DFlash Contextual Corrector), a lightweight mechanism that restores these dependencies while preserving parallel drafting. Rather than committing to each position's top prediction, DCOCO retains multiple candidate tokens and evaluates their contextual compatibility across adjacent draft positions in parallel, enabling contextually informed token selection at low additional cost. Across four verifier backbones and ten benchmarks, DCOCO achieves the highest average throughput in every evaluated configuration, spanning greedy and stochastic decoding with and without extended reasoning, with speedups of up to 6.6x over vanilla decoding. Its advantage persists under concurrent serving: DCOCO outperforms the strongest competing method by 6-23% across 4-32 concurrent requests, while other sequential correctors fall below uncorrected DFlash at these loads. These results demonstrate that recovering intra-block dependencies through parallel contextual scoring delivers consistent acceleration across decoding regimes and serving loads.

1 INTRODUCTION

Parallel diffusion drafting lowers draft latency by predicting positions jointly, but removes access to realized earlier tokens. DCOCO restores contextual selection from existing candidates while keeping neural scoring parallel, and reports throughput gains across decoding settings and serving loads.

1 INTRODUCTION

  • Parallel diffusion drafters reduce latency by predicting a block jointly, but positions lack realized earlier-token identities, weakening continuation consistency and potentially reducing acceptance.Source: §1 · pp. 1–2
  • Prior correctors restore intra-block dependencies through position-by-position modules, whose recurrence adds latency on every drafting cycle.This motivates restoring dependencies without reintroducing a sequential module.Source: §1 · p. 2
  • Because verification accepts only a contiguous prefix, useful correction should prioritize early positions rather than optimize an aggregate score over the block.Source: §1 · p. 2
  • DCOCO scores contextual compatibility among retained candidates from one DFlash pass, computing neural scores in parallel without an additional drafter pass.Its prefix-first greedy rule selects left to right, aligning decisions with verification, which stops at the first rejected token.Source: §1 · p. 2
  • DCOCO improves end-to-end throughput under greedy and stochastic decoding, reaching up to 18% over the strongest evaluated speculative baseline and 6.6× over vanilla autoregressive decoding.It remains ahead of DFlash as serving load grows, while sequential correctors fall below the uncorrected drafter.Source: §1 · p. 2

2 RELATED WORK

Prior work improves speculative drafting through autoregressive target-informed drafters and parallel block generation. DCOCO instead scores transitions between parallel-draft candidates, avoiding sequential learned correction.

2 RELATED WORK

  • Autoregressive drafters preserve dependencies: EAGLE, EAGLE-2, and EAGLE-3 exploit target features or confidence-aware drafting [Li et al., 2024a; Li et al., 2024b; Li et al., 2025b], while ReDrafter uses recurrent target-conditioned drafting [Cheng et al., 2024].Source: §2 · p. 3
  • Parallel drafters such as DFlash generate multiple future positions in one pass, but do not condition each position on earlier realized tokens.Source: §2 · p. 3
  • DSpark and Domino restore prefix-dependent context with sequential correction modules [Cheng et al., 2026; Huang et al., 2026].Source: §2 · pp. 3–4
  • DCOCO scores adjacent candidate interactions in one batched operation from a single DFlash pass, then performs only candidate selection sequentially.Figure 2 depicts parallel transition scoring followed by prefix-first selection from the first-slot draft argmax.Source: §2 · p. 4, Figure 2 · p. 4

3 PARALLEL DRAFTS ALREADY CONTAIN THE RIGHT TOKEN

DFlash’s candidate pool retains verifier-committed tokens more often than its top-ranked choice, especially deeper in the block. The analysis measures this top-k coverage using verifier outcomes during speculative decoding.

3 PARALLEL DRAFTS ALREADY CONTAIN THE RIGHT TOKEN

  • Top-k coverage is the fraction of drafted slots whose k highest-scoring candidates contain the verifier-committed token, measured on DFlash candidates at T = 0.Source: §3 · p. 4
  • At the last slot on Qwen3-8B, top-16 coverage stays above 0.60 while top-1 coverage falls below 0.16.Source: §3 · p. 4
  • Across every verifier, the top-16 −top-1 coverage gap grows from 0.12–0.14 at the first slot to 0.38–0.45 at the last.Source: §3 · p. 4

4 METHOD

DCOCO corrects a DFlash block by rescoring candidate transitions with contextual information and reselecting the proposed block, while preserving parallel drafting.

4.1 BLOCK DRAFTING AND PREFIX ACCEPTANCE

  • Verification accepts the longest correct proposal prefix, halts at the first rejection, and commits one additional token per round.Source: §4.1 · p. 5

4.2 A LEARNED CORRECTOR OVER THE BLOCK GRID

  • Each DFlash slot contributes its top-K candidates to a B × K grid, with every candidate connected to all candidates in the next slot.Source: §4.2 · p. 5
  • Candidate representations fuse token embeddings with slot hidden states, and the corrector uses information available from a single DFlash pass.Source: §4.2 · p. 5
  • Training jointly optimizes the drafter and corrector with frozen token embeddings and masked, exponentially downweighted cross-entropy over verifier-argmax candidate pairs.Source: §4.2 · p. 6

4.3 PREFIX-FIRST PATH SELECTION

  • Prefix-first greedy selection keeps the drafter’s top candidate at slot one, then chooses each successor conditionally, adding an O(BK) walk after O(BK2) batched scoring.It achieves acceptance comparable to geometrically discounted global selection at higher throughput.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments compare DCOCO with vanilla decoding, DFlash, DSpark, and Domino across four verifier backbones and ten benchmarks. DCOCO leads average speedup across the reported configurations, retains an advantage under concurrency, and greedy path selection provides the best throughput among the evaluated strategies.

5 EXPERIMENTS

  • Evaluation uses vLLM with four verifier backbones, ten benchmarks, greedy and stochastic decoding, and a common training corpus of approximately 800K prompts.The primary metric is end-to-end tokens per second on a single B200 GPU; the reported default concurrency is one unless otherwise stated.Source: §5 · pp. 6–7

5.1 MAIN RESULTS

  • On Qwen3-8B, DCOCO improves accepted length over Domino by 8.1% at T=0 and 7.5% at T=1, while request throughput rises by 14.3% and 13.6%.Accepted length measures proposal quality; request throughput includes prefill and serving overhead.Source: §5.1 · p. 7
  • Against the strongest correction baseline, DCOCO combines 7–8% longer accepted prefixes with 6% shorter cycles, yielding a 13–15% throughput gain.Against uncorrected DFlash, it pays 3–4% more per cycle for 18–27% longer accepted prefixes.Source: §5.1 · p. 7
  • Removing pair state reduces accepted length by 7.3%/5.6% at T=0/T=1, and removing candidate representations reduces it by 4.8%/3.7%.A frozen corrector improves DFlash by 4.4%; joint training adds approximately 15%.Source: §5.1 · p. 7
  • DCOCO achieves the best average speedup in all 12 verifier/thinking cells, by 8–19% over each cell’s strongest baseline, and the best average accepted length in 11 of 12.Table 2 summarizes unweighted averages over ten suites; Qwen3-Coder-30B-A3B has no thinking mode.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 SCALING WITH CONCURRENCY

  • At 4–32 concurrent streams, DCOCO has the highest throughput; at 32 streams its margins over the strongest alternative are 19% under greedy decoding and 12% under sampling.At the highest concurrency, Domino and DSpark become slower than DFlash as correction overhead outweighs acceptance gains, while DCOCO remains faster.Source: §5.2 · p. 8, Figure 4 · p. 9

5.3 PATH SELECTION STRATEGY

  • Greedy selection yields higher throughput than Geometric on every evaluated suite, averaging 2.8% improvement, while their average accepted lengths are 5.254 and 5.258.Geometric improves accepted length over Global-Sum on every suite; its accepted-length differences from Greedy have both signs.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

DCOCO uses candidate coverage and token identities to correct parallel block drafts, scoring adjacent transitions in parallel before selecting a continuation greedily. Across several backbones and ten benchmarks, the corrected system leads in accepted length and throughput at almost every measured point, with its advantage persisting or widening at higher serving concurrency.

  • DCOCO scores adjacent candidate transitions in parallel with a cheap neural scorer, then greedily selects a continuation from the drafter’s existing output.The selection proceeds front to back, mirroring verification, which stops at the first miss.Source: §6 · p. 10
  • Top-K lists offer better coverage than top-1 choices, while each drafted token’s identity is information unavailable to other slots.Source: §6 · p. 10
  • Across several backbones and ten benchmarks, the corrected system leads in accepted length and throughput at almost every measured point.Source: §6 · p. 10
  • DCOCO’s advantage persists or widens at higher serving concurrencies, where competing correctors fall below the uncorrected drafter.Source: §6 · p. 10

A TRAINING DETAILS

Training combines a transition-scoring correction loss with DFlash’s loss, using verifier labels from teacher-forced data. The appendix specifies the data, training schedule, architecture, and native draft-length convention.

  • The correction term is a weight-normalized cross-entropy over jointly softmax-normalized transition scores, added with unit weight to DFlash’s loss.Only transitions reachable under the candidate sets and alignment mask receive supervision; the batch uses one weighted mean, with the normalizer clamped below by 1.Source: A TRAINING DETAILS · p. 12
  • Training uses approximately 800K prompts from Nemotron Post-Training Dataset v2 and CodeAlpaca-20k, with each verifier generating its own continuations.Continuations use greedy decoding, thinking disabled, and at most 4096 generated tokens; all methods for a verifier share the resulting corpus.Source: A TRAINING DETAILS · p. 12
  • DCOCO starts from a six-epoch DFlash drafter and trains jointly for six more epochs; baseline comparisons use six epochs under each method’s published objective.Source: A TRAINING DETAILS · p. 12
  • Training uses AdamW with learning rate 6×10−4, no weight decay, cosine scheduling, and a nominal budget of 753,664 tokens per optimizer step; each method is trained once with seed 42.Validation metrics are diagnostic rather than used for checkpoint selection, and variability across training seeds is not measured.Source: A TRAINING DETAILS · p. 12
  • The training objective weights earlier transitions as a fixed proxy for prefix survival, distinct from the geometric discount used to change inference-time path selection.The objective orders the true successor above alternatives in its predecessor’s row while also suppressing competing rows.Source: A TRAINING DETAILS · p. 12
  • Verifier-argmax labels come from precomputed verifier outputs on fixed teacher-forced contexts, and supervision applies only when the label is in the slot’s top-K.These verifier-correct prefixes are the states in which a transition’s decision is consumed, because earlier slots must be accepted first.Source: A TRAINING DETAILS · p. 13
  • The corrector uses K = 16 candidates per slot and width 128 for candidate representations and pair states; on Qwen3-8B it has 2,229,505 parameters.The frozen token-embedding table is shared with the underlying model and is not an additional corrector parameter allocation.Source: A TRAINING DETAILS · p. 13
  • DCOCO’s nominal block size is 16, with B = 15 drafted positions excluding the anchor, scoring 3,584 adjacent candidate transitions and committing at most 16 tokens per round.Comparisons use native configurations rather than a strictly matched number of drafted tokens.Source: A TRAINING DETAILS · p. 13

B TRAINING BUDGET AND CONVERGENCE

The budget comparison tests whether DCOCO’s gains come from extra training, while component ablations assess the corrector’s input pathways and backbone co-adaptation. The reported results distinguish small gains from extended baseline training from larger changes associated with the correction design and joint training.

B TRAINING BUDGET AND CONVERGENCE

  • No more than 2% improvement follows a second six epochs for baselines, while the joint phase accounts for DCOCO’s improvement.The comparison extends DFlash, Domino, and DSpark to twelve epochs under their own objectives and recipes; the main tables compare converged systems, not different budgets.Source: Table 4 · p. 13, B TRAINING BUDGET AND CONVERGENCE · p. 13
  • Removing either input pathway hurts throughput and accepted length; removing the pair state reduces their averages by 6.3%/4.6% and 7.3%/5.6%, respectively, at T=0/T=1.The pair state carries the larger share, and the pathway contributions are consistent across domains, largest on math and code.Source: B TRAINING BUDGET AND CONVERGENCE · p. 13, Table 5 · p. 14, Table 6 · p. 15
  • With the drafter frozen, training the corrector raises average throughput from 783.1 to 817.7 tok/s at T=0 (+4.4%), but freezing reduces throughput by 15.9%/12.9% at T=0/T=1 versus joint training.Extending drafter training alone improves throughput by at most 2%, so the gains from joint training cannot be explained by additional drafter training alone.Source: B TRAINING BUDGET AND CONVERGENCE · p. 14

C EVALUATION DETAILS

The evaluation spans ten suites across mathematics, code, and general tasks, with greedy and stochastic settings and specified thinking conditions. The appendix also defines the serving setup, rate aggregation, and fixed benchmark subsets.

  • Ten benchmarks cover mathematics, code, and general tasks and chat, evaluated at T = 0 and T = 1 with specified token limits and thinking settings.Both thinking settings are evaluated for Qwen3-4B and Qwen3-8B; Qwen3-Coder-30B-A3B and Llama-3.1-8B are evaluated without a separate thinking mode.Source: C EVALUATION DETAILS · p. 16
  • All systems run on one NVIDIA B200 GPU with tensor parallelism 1, using vLLM 0.27.0 and PyTorch 2.13.0+cu130.Prefix caching is disabled, CUDA Graphs are enabled, and evaluation starts after server initialization and graph capture.Source: C EVALUATION DETAILS · p. 16
  • Reported rate is generated tokens divided by request latency including prefill; concurrency-1 rates are benchmark medians and ten-suite averages weight benchmarks equally.For T = 1, mean and standard deviation across seeds quantify evaluation variability for fixed checkpoints, not training-run variability.Source: C EVALUATION DETAILS · p. 16
  • Evaluation uses fixed subsets selected with seed 1234 and shared across methods, including 128 rows each for seven benchmarks and all 30 AIME-25 rows and 80 MT-Bench rows.SPEED-Bench uses 80 rows from each of 11 subsets, for 880 rows total, and is treated as one of ten suites.Source: C EVALUATION DETAILS · p. 16

D FULL PER-DATASET RESULTS

The section reports per-dataset throughput and accepted-length results across verifier and reasoning settings. It also notes that accepted length alone does not capture throughput, which includes correction overhead.

  • The section points readers to per-dataset values in tables formatted like Tables 1 and 7.Source: D FULL PER-DATASET RESULTS · p. 16
  • Accepted length explains throughput movement, while the corrector’s own overhead is visible only in throughput.Source: Table 7 · p. 17
  • Throughput tables cover Qwen3-8B and Qwen3-4B with thinking on, plus Llama-3.1-8B, Qwen3-Coder-30B-A3B, and Qwen3-4B with thinking off.They report output tokens per second per stream and speedup over no-spec decoding at concurrency 1.Source: Table 8 · p. 17, Table 10 · p. 18, Table 12 · p. 19
  • DCOCO attains the best mean accepted length in every cell for Qwen3-8B with thinking on.Source: Table 9 · p. 18
  • Domino has the best accepted length on some Llama-3.1-8B datasets, yet its average throughput is lower.Table 10 reports throughput and Table 11 reports mean accepted length.Source: Table 10 · p. 18, Table 11 · p. 19
  • DCOCO leads mean accepted length in 19 of 20 cells and in both averages for Qwen3-4B with thinking off.Source: Table 15 · p. 21

E COVERAGE BY VERIFIER

Across verifier backbones and concurrency levels, DCOCO leads throughput, while the coverage analysis shows that correct verifier tokens often remain in the candidate pool even when they are not the drafter’s first choice.

E COVERAGE BY VERIFIER

  • Across four verifiers, top-16 coverage falls from about 95% at the first slot to 57–61% at the last, while top-1 falls from about 81% to about one in five.DCOCO uses K=16; the table reports means over 15 drafted slots and shows the first and last slot per verifier.Source: E COVERAGE BY VERIFIER · p. 23, Table 18 · p. 23

F THROUGHPUT AT HIGHER CONCURRENCY

  • DCOCO is fastest across all four backbones, both temperatures, and concurrency levels, beating the strongest baseline by 6–23%; Domino falls below DFlash as load grows and DSpark stays at or below it.Source: F THROUGHPUT AT HIGHER CONCURRENCY · p. 24, Figure 5 · p. 24, Figure 6 · p. 24
  • As concurrency grows, per-cycle correction overhead weighs more heavily against the acceptance it buys; Domino’s widening overhead consumes its acceptance gain, while DCOCO’s throughput remains above DFlash.DSpark gains little or no acceptance, and DCOCO’s accepted-length and throughput curves remain close.Source: F THROUGHPUT AT HIGHER CONCURRENCY · p. 24

G STOCHASTIC VERIFICATION WITH DETERMINISTIC PROPOSALS

DCOCO selects draft proposals deterministically even when the verifier decodes stochastically. Speculative verification then preserves the verifier distribution, and DCOCO retains a throughput advantage at higher concurrency.

  • DCOCO uses a deterministic prefix-first proposal rule, including when the verifier uses stochastic decoding.Conditional on the current context and selected draft prefix, the proposal distribution is a point mass.Source: G STOCHASTIC VERIFICATION WITH… · p. 24
  • At T=1, DCOCO’s throughput margin over the strongest baseline is 6–12% for Llama-3.1-8B across concurrency levels.Source: Figure 7 · p. 25
  • For Qwen3-4B and Qwen3-Coder-30B-A3B, Domino’s acceptance gain is consumed by overhead as concurrency grows, while DCOCO remains faster than DFlash.Source: Figure 8 · p. 25, Figure 9 · p. 26
  • Under point-mass proposals, a draft token is accepted with probability qj(ŷj), and the residual sample at rejection preserves the verifier’s distribution.Verification continues through accepted tokens, commits a residual sample at the first rejection, and discards the remaining suffix.Source: G STOCHASTIC VERIFICATION WITH… · p. 25
Loading 2610.12570v1…