PaperlayerView exact revision on arXiv

Source-linked AI summary

SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Jeonghoon Park, Seongwoon Jo, Jongwon Lee, Taesik Gong

arXiv:2610.05106v1cs.AI

TL;DR

Speculative decoding can become less advantageous during long generations as draft acceptance declines, and growing visible-Key counts can dilute attention mass on high-scoring Keys. SHARPDRAFT applies training-free cardinality-aware Query scaling with a fixed-slope approximation. Across three systems and benchmarks, it reports 2.59–3.19× geometric-mean end-to-end speedups over target-only autoregressive decoding.

  • Problem

    During long generations, draft acceptance can decline and attention mass on the highest-scoring Keys can fall as softmax normalizes over more visible Keys.

    Source: §1 · p. 2

  • Method

    SHARPDRAFT counteracts attention-mass dilution with training-free Query scaling based on visible-Key cardinality, using a closed-form fixed-slope approximation.

    Source: §1 · p. 2

  • Results

    2.59–3.19× geometric-mean end-to-end speedups over target-only AR are reported across AIME-26, GPQA-Diamond, and LongGenBench Diary for DFlash, PARD, and EAGLE 3.1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On evaluated DFlash workloads, SHARPDRAFT achieves higher end-to-end speedup than Frozen and online adaptation while matching Frozen's peak allocated memory.

    Source: §7 · p. 9

    Whether latency gains persist under stochastic decoding, batching, or concurrent serving remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentrated on the highest-scoring Keys can decrease. We introduce SharpDraft, a training-free method that counteracts this effect through cardinality-aware Query scaling, without the computational overhead of online adaptation. Under explicit assumptions, we derive an exact top-$k$ mass correction and deploy a closed-form fixed-slope approximation. Across AIME-26, GPQA-Diamond, and LongGenBench Diary, SharpDraft achieves $2.59$-$3.19\times$ geometric-mean end-to-end speedups over target-only autoregressive decoding when applied to DFlash, PARD, and EAGLE 3.1. With DFlash, it improves decoding speed and outperforms full-parameter and LoRA-based online adaptation in end-to-end speedup, while matching the unmodified drafter's reported peak allocated GPU memory.

1 INTRODUCTION

Speculative decoding can lose throughput as long generations progress, alongside declining attention mass on the highest-scoring Keys. SHARPDRAFT counters this with training-free cardinality-aware Query scaling, and the reported evaluations show end-to-end speedups across three systems and benchmarks.

1 INTRODUCTION

  • As generation proceeds, DFlash, PARD, and EAGLE 3.1 lose throughput advantage over AR, while average attention mass on their 32 highest-scoring Keys declines.Figure 1 contrasts decode throughput in tokens per second with top-32 attention mass on LongGenBench Diary.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The introduction asks whether growing context systematically changes draft attention and whether correcting that change can preserve draft acceptance.Speculative decoding proposes multiple tokens for parallel target verification, and its speedup partly depends on mean accepted length, or output tokens emitted per target verification.Source: §1 · pp. 1–2
  • SHARPDRAFT counteracts attention-mass dilution through training-free Query scaling based on visible-Key cardinality, without changing draft-model parameters.It derives an exact top-k mass correction under explicit assumptions, then uses a closed-form fixed-slope approximation requiring a scalar computation and Query rescaling.Source: §1 · p. 2
  • 2.59–3.19× geometric-mean end-to-end speedups over target-only AR are reported across AIME-26, GPQA-Diamond, and LongGenBench Diary for DFlash, PARD, and EAGLE 3.1.With DFlash, SHARPDRAFT also outperforms full-parameter and LoRA-based online adaptation in end-to-end speedup while matching the unmodified drafter's reported peak allocated GPU memory.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses long-generation acceptance declines through online draft adaptation and studies attention normalization as context grows. SHARPDRAFT instead targets attention-mass dilution in speculative decoding by scaling Queries with the current Key count.

2.1 SPECULATIVE DECODING AND ONLINE DRAFT ADAPTATION

  • Drafts trained on short sequences may generalize poorly to longer contexts, and reported acceptance declines during generation reduce decoding speedup.LongSpec introduces novel position indices for short-sequence draft training (Yang et al., 2026), while TTS reports declining acceptance for EAGLE-3 and DFlash (Kumar et al., 2026).Source: §2.1 · p. 2
  • Online adaptation methods use target feedback to update draft models during inference, while SHARPDRAFT addresses growing-Key attention dilution without updating draft weights.Prior approaches include OSD [Liu et al., 2024], OnlineSpec [Qian et al., 2026], and TTS [Kumar et al., 2026].Source: §2.1 · p. 2

2.2 ATTENTION NORMALIZATION AND LENGTH GENERALIZATION

  • Earlier attention-normalization work studies dispersion as attended-token counts grow; SHARPDRAFT applies a Query-scaling rule to declining draft acceptance in speculative decoding.Related approaches include entropy-based temperature adjustment (Veliˇckovi´c et al., 2025) and logarithmic Query scaling in Scalable-Softmax (Nakanishi, 2025).Source: §2.2 · p. 3

3 CARDINALITY-AWARE QUERY SCALING

SHARPDRAFT scales draft Queries according to visible-Key cardinality, addressing the reduction in top-k attention mass as competing Keys grow. Its fixed-slope rule approximates reference-mass correction and is integrated without rebuilding cached context Keys and Values.

3 CARDINALITY-AWARE QUERY SCALING

  • SHARPDRAFT uses a closed-form fixed-slope approximation to scale draft Queries according to visible-Key count.The primary evaluations use a fixed slope rather than constructing an empirical reference curve.Source: §3 · p. 3, §3.2 · p. 4

3.1 MOTIVATION: ATTENTION DILUTION UNDER GROWING KEY COUNTS

  • Adding lower-scoring Keys enlarges the softmax denominator and reduces an existing top-k group's mass, while changing content, positions, and top-k membership prevents attributing the observed trend to cardinality alone.At fixed contrast and k, doubling the complement size halves attention odds; positive scaling preserves score ordering and, under strict top-k separation, increases contrast.Source: §3.1 · p. 3

3.2 CARDINALITY-AWARE QUERY SCALING

  • The reference correction matches a reference top-k mass by offsetting the log-odds loss from more competitors with increased reference contrast.The deployed rule uses a first-order Taylor approximation with fixed slope s and caps the scale; below the reference count, the scale is 1. Mass error depends on slope choice and approximation error, and the analysis does not guarantee draft acceptance or latency.Source: §3.2 · pp. 3–4

3.3 INFERENCE INTEGRATION

  • DFlash integration rescales Queries after Q-normalization and leaves target verification and persistent context Key/Value construction unchanged.The scale applies to scores against context and current-block Keys, so changing it does not require rebuilding cached context Keys and Values. The count includes visible persistent-context and current-draft-block Keys, excluding masked and padded positions.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments evaluate SHARPDRAFT’s end-to-end speedup, memory use, and draft acceptance across models and generation lengths. They report the strongest end-to-end speedup across all nine model–benchmark pairs, improved MAL for EAGLE 3.1, and higher late-position MAL than Frozen in multi-turn generation.

4 EXPERIMENTS

  • The evaluation measures end-to-end speedup, peak allocated memory, online update cost, and MAL for DFlash drafts paired with three target models.The primary comparison includes Frozen, Full Adaptation, LoRA, and SHARPDRAFT, with target-only autoregressive decoding as the latency reference.Source: §4 · p. 4

4.1 END-TO-END EFFICIENCY

  • SHARPDRAFT achieves the highest E2E speedup on all nine model–benchmark pairs; overall speedups are 2.593×, 2.675×, and 2.967× for Qwen3-4B, Qwen3-8B, and Llama-3.1-8B, respectively.Full Adaptation takes longer than Frozen on AIME-26 and GPQA-Diamond for Qwen3-8B and Llama-3.1-8B, and all tested LoRA ranks trail SHARPDRAFT on every pair.Source: §4.1 · p. 5, Table 1 · p. 5
  • SHARPDRAFT matches Frozen’s reported peak memory, while Full Adaptation adds 7.78–14.53 GiB across the three models.LoRA reduces this overhead but still incurs additional memory and update costs; SHARPDRAFT requires no optimizer updates.Source: §4.1 · p. 5

4.2 GENERALIZATION ACROSS DRAFT MODEL FAMILIES

  • For EAGLE 3.1, SHARPDRAFT raises MAL from 3.152 to 3.302 on AIME-26, 3.659 to 3.904 on GPQA-D, and 5.668 to 5.968 on Diary.Whether MAL gains reduce latency also depends on drafting and verification costs.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 DRAFT ACCEPTANCE IN MULTI-TURN GENERATION

  • SHARPDRAFT maintains higher late-position MAL than Frozen across all four models in multi-turn generation.Full Adaptation and LoRA can achieve higher MAL, particularly for Qwen3-4B, at the cost of online training; the curves measure drafting efficiency rather than reasoning quality.Source: §4.3 · p. 6, Figure 3 · p. 6

5 ANALYSIS AND ABLATION STUDIES

The analysis compares dynamic scaling with fixed Query scales, evaluates fixed-slope scaling against layerwise calibration, and tests how the reference count affects verifier calls. Dynamic scaling leads across output-length groups, while the tested fixed-slope rule avoids the extra runtime cost of layerwise calibration; the preferred reference count varies by model.

5.1 DYNAMIC VERSUS FIXED SCALING

  • 4.063 mean MAL for SHARPDRAFT exceeds the best fixed scale’s 4.023 and Frozen’s 3.182, with the highest mean in every output-length group.Dynamic scaling improves request MAL on 26 of 30 problems; the best fixed scale shifts from e0.2 for short outputs to e0.4 for long outputs.Source: §5.1 · p. 7, Table 3 · p. 7
  • Figure 4’s matched-prefix comparison shows fixed scales raise top-32 attention mass, but mass still declines with context; SHARPDRAFT stays near its short-context level through 16K.SHARPDRAFT has the highest mean cumulative MAL from 4K onward; later checkpoints include only problems that reach them.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 FIXED-SLOPE APPROXIMATION

  • Layerwise Taylor reduces mean absolute top-32 mass error from 6.163 to 2.388 percentage points and raises whole-request MAL from 4.545 to 4.667.Its 2.7% MAL gain is concentrated mostly in capped or repetitive outputs, while the comparison does not isolate dilution’s causal effect.Source: §5.2 · p. 8, Figure 5 · p. 8
  • On matched requests, Layerwise Taylor raises mean E2E time from 25.87 to 31.28 seconds, a 20.9% increase over SHARPDRAFT.For this native-vLLM implementation, calibration cost outweighs the acceptance gain.Source: §5.2 · p. 8
  • The default slope s = 5 gives the highest observed whole-request MAL at 4.545; s = 4 and s = 6 remain within 1.6%.The sweep holds N0, k, and βmax fixed; the pooled-reference diagnostic yields effective slopes from 4.995 to 5.112.Source: §5.2 · p. 8

5.3 SENSITIVITY TO THE REFERENCE CARDINALITY

  • Raising N0 from 2K to 4K increases AIME-26 calls per token by 3.2%, 9.8%, and 2.0% for Qwen3-4B, Qwen3-8B, and Llama-3.1-8B, while GLM needs 1.2% more calls at 2K.For Qwen3-8B, the pattern is similar across AIME-26, GPQA-Diamond, and LGB(Diary); 8K needs approximately 13–14% more calls per token than the default.Source: §5.3 · pp. 8–9, Figure 6 · p. 9

6 DISCUSSION AND LIMITATIONS

Query scaling sharpens draft attention but cannot correct every source of disagreement with the target. The theoretical guarantee and empirical scope are bounded by assumptions, evaluation settings, and differences between target–draft pairs.

6 DISCUSSION AND LIMITATIONS

  • Positive Query scaling preserves score order for fixed Queries and Keys, so it can reinforce an incorrect preference rather than correct content mismatch.Learned updates may still help when mismatch concerns content rather than concentration.Source: §6 · p. 9
  • The mass-preservation guarantee assumes current cardinality is at least the reference, current and reference contrasts agree throughout the application range, and upper clipping is inactive.The deployed fixed-slope Taylor approximation also depends on current/reference contrast offsets, so its slope and reference count should be validated for new draft models.Source: §6 · p. 9
  • End-to-end evidence is strongest for DFlash under greedy decoding; latency gains under stochastic decoding, batching, or concurrent serving remain untested.The multi-turn study measures acceptance, not latency, through 100K generated tokens and retains repetitive continuations; answer quality should be assessed alongside latency.Source: §6 · p. 9

7 CONCLUSION

SHARPDRAFT scales draft Queries with visible-Key cardinality without training the draft model. On evaluated DFlash workloads, it improves end-to-end speedup over Frozen and online adaptation while matching Frozen’s peak allocated memory.

7 CONCLUSION

  • On evaluated DFlash workloads, SHARPDRAFT yields higher end-to-end speedup than Frozen and online adaptation while matching Frozen’s peak allocated memory.The ablations show a modest but consistent benefit from varying scale with context, while more precise mass restoration adds little further acceptance.Source: §7 · p. 9
  • SHARPDRAFT scales draft Queries according to visible-Key cardinality without training the draft model.The fixed-slope approximation is derived by separating attention-score contrast from the number of competing Keys.Source: §7 · p. 9
  • Varying scale with context provides a modest but consistent benefit, while more precise mass restoration adds little further acceptance.Source: §7 · p. 9

A THEORETICAL ANALYSIS

The analysis separates key-count effects from score contrast, establishes when a reference inverse exists, and states conditions for preserving reference top-k mass.

A.1 EXACT PARTITION IDENTITY

  • At fixed Ft(1), increasing Ct decreases uncalibrated top-k mass, without requiring selected Keys to be task-relevant.Source: §A.1 · p. 12

A.2 REFERENCE CONTRAST AND MONOTONICITY

  • Under strict top-k separation, the average is strictly monotone for a finite calibration set, establishing an inverse on the selected interval.Source: §A.2 · p. 13

A.3 REFERENCE INVERSE AND CONDITIONAL MASS PRESERVATION

  • The reference inverse preserves ρref when Ct ≥ Cref, upper clipping is inactive, and current and reference contrasts agree; contrast error ε bounds log-odds error by ε.The logistic function’s 1/4-Lipschitz property converts that log-odds bound to a probability bound.Source: §A.3 · p. 13

A.4 TRAINING-DISTRIBUTION REFERENCE AND APPROXIMATION SCOPE

This section defines training-distribution and experimental references, explains the fixed-slope approximation’s scope, and describes its implementation and cost.

A.4 TRAINING-DISTRIBUTION REFERENCE AND APPROXIMATION SCOPE

  • The effective total-Key reference generally differs from exp(E[log Nx]), and total-length ratios approximate the exact competitor-count ratio only when N is large relative to k.Source: §A.4 · p. 13
  • The training-distribution reference represents mean log-odds, while N0 without that distribution is an experimental proxy rather than a measurement of training-time attention.Source: §A.4 · p. 13
  • The fixed-slope rule replaces Δref with s; its error depends on slope choice, Taylor remainder, and current/reference contrast mismatch.It inherits the conditional guarantee when the assumptions hold and the reference contrast is linear with slope s.Source: §A.4 · pp. 13–14

B.1 QUERY-SCALING IMPLEMENTATION

  • DFlash uses k=32, s=5, βmax=e^0.5, and N0=2048 for primary requests and Qwen/Llama sessions or 4096 for GLM; these are experimental proxies.Source: §B.1 · p. 14
  • The implementation computes the gain and product in FP32, casts the product to BF16 before addition, and leaves normalization weights unchanged.Rescaling is proportional to query-tensor size; its arithmetic and temporary allocations are included in end-to-end time and peak memory.Source: §B.1 · p. 14

B.2 END-TO-END EVALUATION

This block specifies evaluation protocols and how latency, acceptance, and attention analyses are aggregated across benchmark and ablation settings.

B.2 END-TO-END EVALUATION

  • Table 1 uses matched DFlash draft models, shared prompts and verification policy, and target-only AR with the same target checkpoint and precision as baseline.The evaluation uses one RTX 6000 Ada GPU per run and specified AIME-26, GPQA-Diamond, and Diary cohorts.Source: §B.2 · p. 14
  • E2E includes prefill, drafting, verification, calibration or learning, and synchronization; speedup compares total E2E time with target-only AR.Figure 2’s training-time highlights neither locate execution intervals nor provide an additive critical-path decomposition; drafting, verification, and calibration are not timed separately.Source: §B.2 · pp. 14–15

B.3 TRANSFER ACROSS DRAFT MODEL FAMILIES

  • The H200 transfer evaluation specifies Qwen3-8B cohorts, released EAGLE 3.1 checkpoint configuration, and proposal counts and N0 settings for PARD and EAGLE 3.1.Both use k=32, s=5, and a log-gain cap of 0.5; LoRA updates are scheduled every five verification rounds.Source: §B.3 · p. 15

B.4 LONG-GENERATION MOTIVATION

  • Figure 1(a) plots smoothed absolute decode throughput in tokens per second, while panel (b) shows local means of sampled top-32 attention mass and variation bands.Panel (b)’s cross-method mass levels are descriptive because draft architectures have different query scopes and visible-Key sets.Source: §B.4 · p. 15

B.5 MULTI-TURN ACCEPTANCE

  • The multi-turn MATH-500 evaluation retains caches and adaptation state across questions, so MAL measures acceptance on observed continuations rather than answer quality.It uses a 223-question subset and does not reuse questions within a session.Source: §B.5 · pp. 15–16
  • Qwen/Llama and GLM use different context limits, output budgets, and draft-block settings; context exhaustion can stop a session early.Source: §B.5 · p. 16

C.1 FIXED VS. DYNAMIC SCALING

  • Table 3 pairs Frozen, five fixed scales, and SHARPDRAFT on identical outputs; its global and group-specific best fixed scales are selected on this cohort, not held-out data.Length groups contain 9, 8, and 13 problems, respectively.Source: §C.1 · p. 16

C.2 ATTENTION ON MATCHED PREFIXES

  • Figure 4a reconstructs top-32 attention mass across matched prefixes, averaging over layers, heads, proposal positions, probes, and eligible problems.Only problems with all four probes in every condition contribute; the figure displays means without its computed bootstrap intervals.Source: §C.2 · pp. 16–17

C.4 FIXED-SLOPE SENSITIVITY AND REFERENCE VALIDATION

The reference-curve diagnostic finds that a fixed slope of five closely approximates the correction on the pooled curve. The slope sweep also tracks when upper-scale clipping affects smaller slope choices.

C.4 FIXED-SLOPE SENSITIVITY AND REFERENCE VALIDATION

  • 23.9% of post-reference rounds hit the upper scale cap at s = 3, compared with 2.8% at s = 4 and zero at s ∈{5, 6, 8}.The AIME-26 sweep used 30 problems, one run per slope and problem, with N0 = 2048, k = 32, and βmax = e^0.5 fixed.Source: §C.4 · p. 17
  • The reference diagnostic numerically inverts measured Fref to infer β*(N), then computes the effective reference slope as a secant slope.It pools 19,200 Query rows and uses 256-Key increments through 40K; all solutions remain within the measured scale range.Source: §C.4 · p. 17
  • 4.995–5.112 is the inferred slope range, and s = 5 closely approximates the correction on the pooled reference curve.This is not evidence of live-state slopes or MAL optimality; current-state contrast can differ from the reference.Source: §C.4 · p. 18, Figure 7 · p. 18

C.5 LAYERWISE TAYLOR DIAGNOSTICS

The layerwise diagnostics compare Taylor re-centering and timing with the deployed fixed-slope rule, then specify a reference-cardinality sensitivity sweep across models and datasets. The reported timing favors SHARPDRAFT over Layerwise Taylor in the measured comparison.

C.5 LAYERWISE TAYLOR DIAGNOSTICS

  • Mean absolute contrast error falls from 0.0433 at center one to 0.0023 with re-centering on the same Taylor states and applied scales.The comparison isolates re-centering rather than the fixed slope; these diagnostics cover centers visited during generation, not an independent sweep over fixed centers.Source: §C.5 · p. 18
  • 181.08 seconds for SHARPDRAFT versus 218.95 seconds for Layerwise Taylor gives per-request means of 25.87 and 31.28 seconds, respectively.Taylor was 20.9% longer across seven output-matched requests on the same physical GPU; timing includes prefill, decoding, calibration, and streaming delivery.Source: §C.5 · pp. 18–19
  • A local derivative near five does not account for the current/reference offset or justify a universal effective slope.Median layer-mean derivatives on Taylor trajectories were 4.84, 4.39, 5.26, 4.72, and 4.64, while effective denominators on Frozen states had medians 3.68, 6.25, 11.30, 9.83, and 7.69.Source: §C.5 · p. 19

C.6 REFERENCE-CARDINALITY SENSITIVITY

  • The reference-cardinality sensitivity runs compare Frozen with N0 ∈{1024, 2048, 4096, 8192} across six model–dataset groups.The setup reuses AIME-26 runs and adds GPQA73 and Diary requests; Diary requests reach the output cap in every arm.Source: §C.6 · p. 19
Loading 2610.05106v1…