LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

1 INTRODUCTION

• Existing drafting architectures inherit prefix-dependent costs through growing KV caches, full-prefix traversals, or target features injected into draft layers.As context length grows, this cost can erode the efficiency advantage of speculative decoding.Source: §1 · p. 2
• Because the target verifies every proposal and corrects errors before commitment, the drafter can use a compressed, lossy prefix view.The paper frames drafting efficiency around accepted tokens relative to drafting overhead and calls prefix-independent per-round overhead fixed-cost drafting.Source: §1 · p. 2
• LONGSPARK extracts fixed-size multiscale views from the target’s verification pass and uses them for parallel block prediction, keeping other per-round operations O(1) in prefix length.The views comprise a boundary anchor, a recent-token window, and a fixed-size global context summary.Source: §1 · p. 2, Figure 2 · p. 3
• LONGSPARK achieves the highest end-to-end throughput across reasoning, coding, and dialogue benchmarks at multiple scales, with 1.88× to 2.13× speedups that grow with target size.It also improves end-to-end throughput while reducing drafter memory overhead by several orders of magnitude.Source: §1 · p. 2

2 LONGSPARK: FIXED-COST SPECULATIVE DRAFTING

• LongSpark’s fixed-cost design requires bounded-size views of the prefix and a proposal model that can produce competitive drafts from those views alone.The section introduces these as the two ingredients of fixed-cost drafting.Source: §2 · p. 3

2.1 PRELIMINARIES

• The target verifies a block of draft candidates in one forward pass, and accepting r candidates advances the prefix by r + 1 tokens while preserving the target distribution.The drafter proposes candidates conditioned on the committed prefix; verification tests them in order.Source: §2.1 · p. 3

2.2 FIXED-COST CONTEXT INTERFACE

• The context interface combines a boundary state, a recent KV window, and a global summary with a fixed number of entries per layer and head.The summary uses fixed learned queries and incorporates only newly retained KV rows after verification, at most B + 1 per update.Source: §2.2 · p. 4

2.3 PROPOSAL MODEL

• The proposal model processes draft positions in parallel, then applies a lightweight per-position Markov correction; each position attends over at most B + W + R entries per head.The parallel backbone uses the boundary state and reads the recent window and global summary from corresponding target layers.Source: §2.3 · pp. 4–5

2.4 TRAINING AND INFERENCE

• The target parameters remain frozen during joint training of the draft network and interface; at inference, accepted-token KV rows update the interface, while draft KV is discarded.With fixed model dimensions, draft computation and additional state are independent of prefix length, and full-prefix processing is confined to target verification.Source: §2.4 · p. 5

3.1 EXPERIMENTAL SETUP

• The evaluation spans Qwen3-4B, 8B, and 14B; reasoning, code, and dialogue benchmarks; and long-context workloads at 32K, 64K, and 128K.The setup compares EAGLE-3, DFlash, DSpark, and vanilla autoregressive decoding; speculative results use a target–drafter disaggregated framework.Source: §3.1 · p. 6

3.2 OVERALL PERFORMANCE

• 1.88×, 1.99×, and 2.13× speedups from 4B to 14B give LONGSPARK the highest average throughput at every scale, even when DSpark accepts more tokens.The evaluation uses target TP1, concurrency 32, and up to 2,048 generated tokens per request.Source: §3.2 · p. 7
• At concurrency 128, LONGSPARK leads DSpark across three benchmarks and target scales, with the throughput gap widening as load rises despite shorter accepted lengths.It starts marginally behind DSpark at low concurrency, then overtakes it as concurrency increases from 8 to 128.Source: §3.2 · p. 7, Figure 4 · p. 7

3.3 SCALING WITH CONTEXT LENGTH

• Across three long-context workloads, LONGSPARK has the highest mean throughput and lowest mean TPOT; on CodeSpan at 64K, TPS exceeds DSpark’s by 29.1% and TPOT is 23.1% lower.The Qwen3-8B evaluation covers contexts from 32K to 128K; Table 2 reports TPS and TPOT averaged over ten seeds.Source: §3.3 · p. 7, Table 2 · p. 7
• From 32K to 128K, LONGSPARK’s drafting time stays near 3.5 ms and context state at 6.3 MiB, while DSpark rises from 6.3 to 13.8 ms and 640 to 2560 MiB.At 128K, LONGSPARK’s drafter state is 406× smaller; target verification remains the only full-prefix operation, and the leaner proposal component’s savings accumulate across long generations.Source: §3.3 · p. 8

3.4 ABLATION STUDY

• Removing the recent KV window lowers average accepted length from 4.69 to 3.71; removing the global summary or boundary state lowers it to 4.35 or 4.28, respectively.The ablation also finds transient training-loss spikes without the global summary, while the full model’s loss descends smoothly; Figure 5 contrasts mean accepted length by domain with training loss.Source: §3.4 · p. 8, Figure 5 · p. 8
• LONGSPARK’s recent window supplies local grounding while its global summary provides a low-resolution view of the prefix; a sliding-window DSpark variant trails it in accepted length and throughput at every context length.Figure 5(a) removes context components in turn; the variants are trained for five epochs, with up to 2,048 generated tokens per request.Source: §3.4 · p. 8

Autoregressive Drafters

• Autoregressive drafters’ KV caches grow with the prefix; reusing target features or recurrent state strengthens proposals, while windowed LongSpec still reads the target’s full KV history and advances one position at a time.Examples include EAGLE and EAGLE-3 (Li et al., 2024; 2025; Hui et al., 2026), ReDrafter (Cheng et al., 2024), and LongSpec (Yang et al., 2026).Source: §4 · p. 9

Block-Parallel and Block-Diffusion Drafting

• Block-parallel methods predict multiple positions, but Medusa conditions on only the last hidden state, whereas DFlash and Orthrus still process or attend to the full confirmed prefix.The passage cites Xiao et al. (2024), An et al. (2026), Medusa (Cai et al., 2024; Wertheimer et al., 2024), DFlash (Chen et al., 2026; Li et al., 2026), and Orthrus (Nguyen et al., 2026).Source: §4 · p. 9

Subsequent work

• Later methods improve feature fusion, within-block coherence, candidate coverage, or proposal horizon, but retain a context interface tied to prefix length.The passage names DFlare (Zhang et al., 2026), DSpark, Domino, JetSpec, xPress, DDTree, AdaFlash, and a 128-token DSpark sliding window.Source: §4 · p. 9
• LONGSPARK replaces the growing context with fixed-size multiscale views covering the entire prefix, leaving target verification as the only full-prefix traversal per round.The paper characterizes it as, to its knowledge, the first block-diffusion drafter to make this replacement.Source: §4 · p. 9

5 CONCLUSION AND FUTURE DIRECTIONS

• The work establishes prefix-independent decoding cost with competitive accepted lengths by extracting useful context from target state at fixed cost.The conclusion identifies accepted tokens per unit of proposal overhead as the quantity drafting design should optimize.Source: §5 · pp. 9–10
• LONGSPARK attains the highest throughput across model scales even though the strongest prefix-scaling drafter reaches slightly longer accepted lengths; low proposal overhead outweighs marginal acceptance gains.The advantage grows with context length and serving concurrency, while ablations indicate that a compact, lossy prefix view can orient competitive proposals.Source: §5 · pp. 9–10
• Future work includes testing larger targets and longer contexts, alternative interfaces and drafter architectures, tree-structured or multi-drafter verification, and reinforcement-learning rollouts.These directions aim to assess scaling, improve proposal quality while retaining fixed cost, or increase accepted tokens per round.Source: §5 · p. 10

AI USE OF GENERATIVE MODELS

• Generative AI tools assisted with code development and debugging, experimental-result analysis, and manuscript, table, and figure preparation.The authors state that they reviewed and verified all AI-assisted work.Source: AI USE OF GENERATIVE MODELS · p. 10
• The authors take full responsibility for the final content, including its text, claims, and artefacts.They state that all AI-assisted work was reviewed and verified.Source: AI USE OF GENERATIVE MODELS · p. 10
• AI assistance covered both research workflow tasks and preparation of the paper’s reported materials.The listed materials include the manuscript, tables, and figures.Source: AI USE OF GENERATIVE MODELS · p. 10

A.1 DRAFTER FORWARD PASS

• Each proposal uses selected-layer boundary states, recent Target KV windows, and global summary pairs, ordered by target layer.The architecture uses five Qwen3-style blocks to propose seven tokens, with 16 summary entries and a 256-entry recent window per selected layer and head.Source: §A.1 · p. 13, §B.1 · p. 14
• Draft slots are processed in parallel with attention over the three context sources and bidirectional attention among local slots.The Target embedding, final normalization, and vocabulary head remain frozen; local block KV is discarded after drafting.Source: §A.1 · p. 13
• Input slot 0 supplies base logits for the first draft token, while returned distributions are used in target verification.Only the Markov correction and sampling proceed sequentially, and retained Target KV rows subsequently update the summary.Source: §A.1 · p. 13

A.2 INCREMENTAL GLOBAL CONTEXT SUMMARY

• The global summary combines statistics for the previously processed prefix with those of newly retained rows.An empty new set leaves the state unchanged; the merge recovers attention over the combined rows while storing one value vector and log-normalizer per query.Source: §A.2 · p. 14

B.1 DRAFTER ARCHITECTURE

• LONGSPARK uses five Qwen3-style decoder blocks and reuses the Target embedding and final LM head.Selected Target layers vary by model scale; the drafter combines a global summary, recent KV window, and small local workspace.Source: §B.1 · p. 14

• Training uses approximately 1.42M Open-PerfectBlend sessions, with assistant responses regenerated by the frozen Target model.Responses use non-thinking mode with temperature 0.7, top-p 0.8, and top-k 20.Source: §B.2 · p. 14
• Training limits sequences to 4096 tokens and samples up to 512 anchors per sequence to optimize distillation across response positions.The objective is applied to regenerated Target responses.Source: §B.2 · p. 14
• Each selected layer and attention head has an independent bank of learned summary queries, shared across examples and fixed during inference.The queries are jointly optimized with the drafter and support incremental summary updates.Source: §B.2 · p. 14
• Main drafters train for 10 epochs and ablations for 5, using AdamW, BF16 mixed precision, and a peak learning rate of 6 × 10^-4.All use a global batch size of 512, FP32 master weights, cosine decay with 4% linear warmup, and data-parallel degree 8 with 64 accumulation steps.Source: §B.2 · p. 14

C.1 LONG-CONTEXT EVALUATION DATASETS

• LongSpec continuation tasks use 32,768-token inputs drawn from code, arXiv, and book subsets of its released training corpus.The drafting-cost analysis uses a prefix from the code subset.Source: §C.1 · p. 15
• CodeSpan contains 32 code-continuation examples from distinct files across 17 projects, with complete inputs of 65,536 tokens.The examples use contiguous prefixes from single files, predominantly C/C++.Source: §C.1 · p. 15
• LongSWE-Bench inputs fit a 128K total-context budget while reserving up to 8,192 generated tokens.Evaluation inputs and ordering are shared across methods and random seeds.Source: §C.1 · p. 15

C.2 MEASUREMENT PROTOCOL

• Steady-state serving is measured for 120 seconds after a 30-second warm-up, with throughput counting all output tokens emitted in the interval.Accepted length is computed from emitted tokens after the first token per verification round, including bonus or residual tokens and accounting for termination.Source: §C.2 · p. 15
• TPOT divides post-first-token request time overlapping the measurement interval by subsequent output tokens emitted within it.Its numerator includes observed time for requests still active at interval end, retaining scheduling and concurrent prefill effects.Source: §C.2 · p. 15

D.1 GREEDY-DECODING RESULTS

• LONGSPARK consistently achieves the highest speedups across model scales and benchmarks under greedy decoding.The reported trends closely mirror sampling-based results at T = 1; the evaluation uses T = 0, target TP1, concurrency 32, and up to 2,048 generated tokens per request.Source: §D.1 · p. 15, Table 3 · p. 16

D.2 LONG-CONTEXT PERFORMANCE ACROSS MODEL SCALES

• Across Qwen3-4B, 8B, and 14B, LONGSPARK achieves the highest average TPS and lowest average TPOT on every long-context benchmark.Table 4 extends the long-context evaluation to the 4B and 14B targets.Source: §D.2 · p. 16, Table 4 · p. 16

D.3 LONG-CONTEXT COMPONENT ABLATIONS

• Across 32K–128K-token workloads, Table 5 compares accepted length and serving performance for the full drafter against variants that remove one component at a time.Other settings follow Table 2.Source: §D.3 · p. 17, Table 5 · p. 17

D.4 COMPARISON WITH SLIDING-WINDOW DRAFTING

• Across every workload, LONGSPARK exceeds DSpark-SWA in accepted length and throughput, using target state to keep the entire prefix in view at fixed cost.DSpark-SWA is restricted to a 256-token inference window, matching LONGSPARK’s recent-token window but excluding earlier tokens.Source: §D.4 · p. 17, Table 6 · p. 17

D.5 PERFORMANCE ACROSS CONCURRENCY LEVELS

• At C=128, LONGSPARK maintains a significant throughput lead over all baselines, while DSpark is competitive at C=8.The evaluation uses Qwen3-8B, target TP1, T=1, and up to 2,048 generated tokens per request.Source: §D.5 · p. 17, Table 7 · p. 18

D.6 FIXED-REQUEST END-TO-END PERFORMANCE

• LONGSPARK achieves the lowest mean completion time on all three workloads for fixed sets of 32 requests served at concurrency 16.Completion time includes prefill, queuing, decoding, and the final requests; responses were sampled independently, so times also reflect variation in generated response length.Source: §D.6 · p. 18, Table 8 · p. 18