PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification

Zhendong Tan, Xingjun Zhang, Chaoyi Hu, Junjie Peng, Kun Xia

arXiv:2512.02337v2cs.LG

TL;DR

As context length grows, verification becomes a dominant bottleneck in speculative decoding for long-context generation. SpecPV uses partial KV verification with periodic full checks; it reports up to 6× decoding speedup over standard autoregressive decoding with minor accuracy degradation.

  • Problem

    In long-context speculative decoding, verification time grows from around 60% to nearly 80%, raising whether full verification is necessary.

    Source: §1 · p. 2

  • Method

    SpecPV performs verification with partial KV states and periodically uses full verification to rectify accumulated deviations.

    Source: §1 · p. 2

  • Results

    Up to 6× decoding speedup over standard autoregressive decoding, with minor accuracy degradation.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that SpecPV contributes to more efficient and practical speculative decoding for long-context LLM applications.

    Source: §5 · p. 8

    Partial verification may miss fine-grained details not reflected by standard metrics, and acceleration may vary across languages or application scenarios.

    Source: Limitations · p. 8, §4.4 · p. 8

Abstract

from arXiv · show

Growing demands from tasks like code generation, deep reasoning, and long-document understanding have made long-context generation a crucial capability for large language models (LLMs). Speculative decoding is one of the most direct and effective approaches for accelerating generation. It follows a draft-verify paradigm, where a lightweight draft model proposes several candidate tokens and the target model verifies them. However, we find that as the context length grows, verification becomes the dominant bottleneck. To further accelerate speculative decoding in long-context generation, we introduce SpecPV, a self-speculative decoding approach that performs fast verification using partial key-value states (KV) and periodically applies full verification to eliminate accumulated errors. We validate SpecPV across multiple long-context benchmarks and models, including LLaMA-3.1-8B-Instruct and Qwen3-series. Experimental results show that SpecPV achieves up to 6x decoding speedup over standard autoregressive decoding with minor degradation.

1 Introduction

Long-context applications and longer reasoning outputs increase the demand for efficient generation, while verification becomes the dominant speculative-decoding bottleneck as context grows. SpecPV uses partial KV verification with periodic full checks, achieving up to 6× decoding speedup with negligible accuracy loss.

1 Introduction

  • Document analysis, summarization, and code generation require longer inputs, while explicit reasoning can produce verbose outputs that further challenge efficient long-context generation.The paper frames long-context generation as an increasingly important LLM capability.Source: §1 · p. 1
  • Speculative decoding reduces repeated autoregressive forward passes by having a lightweight draft model propose multiple tokens for target-model verification.The target verifies the candidates in a single forward pass, and multiple tokens can be accepted at once.Source: §1 · p. 1
  • Verification time rises from around 60% to nearly 80% as context length increases, making full verification a long-context bottleneck.Figure 1 shows drafting and verification time for EAGLE-3 on LLaMA-3.1-8B-Instruct.Source: Figure 1 · p. 1, §1 · p. 2
  • Up to 6× decoding speedup over standard autoregressive decoding, with negligible accuracy loss, and around 2× speedup over full verification at 60K context length.The reported evaluation uses the EAGLE-3 draft.Source: §1 · p. 2
  • SpecPV uses partial KV states for some verification steps and periodically performs full verification to rectify deviations and keep outputs consistent with the original.The target model's layer features from accepted tokens are reused to guide subsequent drafts.Source: §1 · p. 2

2 Related Work

Related work addresses both speculative-decoding design and long-context efficiency. SpecPV builds on self-speculative drafting while targeting verification, a stage that prior long-context approaches generally leave to full KV.

2 Related Work

  • Speculative decoding accelerates autoregressive models, and Leviathan et al. (2023) introduced Speculative Sampling as theoretically equivalent to standard sampling.Candidate organization and draft-model design are identified as two key challenges.Source: §2 · pp. 2–3
  • SpecInfer expands candidate space with tree-structured tokens, followed by dynamic tree construction explored by Li et al. (2024a) and Liu et al. (2024b).Classical approaches use a small speculative model to generate a single candidate sequence.Source: §2 · p. 3
  • Self-speculative methods reuse target-model features: some use layer skipping or early exiting, while EAGLE, Medusa, and MTP use top-layer features as draft inputs.EAGLE-3 (Li et al., 2025b) uses a training-time test technique and predicts tokens directly rather than predicting features.Source: §2 · p. 3
  • Long-context efficiency work includes KV-cache quantization, sparse attention, and linear or hybrid linear attention architectures.These approaches target memory use, attention computation, or alternatives to full attention.Source: §2 · p. 3
  • Prior speculative long-context methods focus mainly on drafting, whereas SpecPV adopts an EAGLE-3–style draft and targets verification with partial KV.Triforce uses streaming-cache drafting, MagicDec uses sparse-KV drafting, and TokenSwift combines partial KV with a Medusa-style draft.Source: §2 · p. 3

3 SpecPV

SpecPV targets verification, which becomes a bottleneck as context grows, by using partial KV verification within a self-speculative decoding framework. Periodic full verification corrects accumulated errors; the section also describes cache selection, refresh triggers, and offloading.

3.1 Self-Speculative Decoding Framework

  • Self-speculative drafting reuses target-model layer features already produced during verification, so the draft phase needs no additional target-model forward pass.The draft module is typically lightweight and may consist of a single decoder layer.Source: §3.1 · pp. 3–4

3.2 Partial Verification

  • The partial KV cache retains initial sink blocks, query-dependent retrieval tokens, a fixed-size local window, and a buffer for partially verified and candidate tokens.Key-state summaries are cached to accelerate retrieval, and the four components are handled as one continuous KV segment during attention.Source: §3.2 · p. 4
  • Verification scores blocks against multiple candidate-token query states, then aggregates per-query scores into one block-importance score using a reduction such as max, mean, or last.Block key-state summaries use element-wise maximum and minimum key states.Source: §3.2 · p. 4

3.3 Rectified with Full Verification

  • Periodic full verification corrects accumulated partial-cache errors from inaccurate states of partially verified tokens and changes in retrieved context.Before verification, partially verified tokens are prepended to new candidates; full KV is used and the partial cache is refreshed.Source: §3.3 · pp. 4–5
  • Partial verification begins after a full verification initializes its cache, and exceeding the partial-cache buffer limit triggers full verification and a cache refresh.The buffer size controls how frequently full verification is triggered.Source: §3.3 · p. 5
  • With memory constrained, SpecPV offloads only the full KV cache and keeps the smaller partial and draft caches on-device, reducing full-cache access and PCIe transfer overhead.The next layer is prefetched when needed.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate SpecPV’s efficiency and output quality across model families, context lengths, and tasks. They report greater efficiency benefits at longer contexts and under KV-cache offloading, while generation quality remains close to full verification in most evaluated settings.

4.1 Setup

  • The evaluation uses LLaMA-3.1-8B-Instruct and Qwen3 models, with EAGLE-3 drafts adapted using 6,400 PG-19 samples at 32K sequence length.Efficiency is tested on PG-19 continuation; quality tasks cover LongBench v1 and v2 summarization and question answering.Source: §4.1 · p. 5
  • Baselines include TriForce, TokenSwift, and EAGLE3-YARN; evaluation reports throughput speedup, draft accept length, exact-match QA accuracy, ROUGE-L, and BLEURT.Draft accept length is the average number of tokens accepted per verification step.Source: §4.1 · pp. 5–6

4.2 Generation Efficiency

  • SpecPV’s efficiency benefits grow with context length; it achieves significant speedup with KV-cache offloading and consistent acceleration across Qwen3 models and sizes.The reported comparisons include LLaMA3.1-8B-Instruct and the Qwen3 series, with partial KV budgets varying across experiments.Source: Table 1 · p. 6, Figure 4 · p. 6, §4.2 · p. 7
  • The partial KV budget affects draft accept length, so the authors recommend selecting it flexibly by context length to balance efficiency and accuracy.Source: §4.2 · p. 7

4.3 Generation Quality

  • Summarization performance is comparable to full decoding with minor degradation, while QA quality remains highly consistent; at 4096 tokens, most datasets are comparable to full verification.The QA evaluation excludes LongBench v2 samples exceeding 64K context length.Source: Table 2 · p. 6, Figure 5 · p. 7, §4.3 · p. 8

4.4 Case Study

  • In a Qwen3-8B, 50K-token book-summary case, SpecPV preserves key information and crucial facts, but full verification includes more fine-grained details.The authors conjecture that some fine-grained details may be lost with subset KV cache, a limitation not well reflected by standard accuracy metrics.Source: §4.4 · p. 8, Figure 6 · p. 8

5 Conclusion

SpecPV accelerates long-context speculative decoding by using partial KV cache for fast verification and periodically applying full verification to correct accumulated errors. The paper reports up to 6× speedup over standard autoregressive decoding with minor accuracy degradation.

5 Conclusion

  • SpecPV achieves up to 6× decoding speedup over standard autoregressive decoding with minor accuracy degradation.Source: §5 · p. 8

Limitations

SpecPV uses controlled partial-KV approximation, which can miss fine-grained details not captured by standard metrics. Its acceleration depends on the draft model and may vary across languages or applications.

Limitations

  • Partial-KV verification may miss fine-grained details that standard metrics do not always reflect, despite preserving key semantic and factual content.The paper describes this as a controlled approximation relative to lossless speculative decoding.Source: Limitations · p. 8
  • Acceleration depends on the draft model—here, EAGLE3-style self-speculative decoding—and may vary across languages or application scenarios.Source: Limitations · p. 8
  • Combining speculative decoding with recent pre-training attention architectures is identified as a direction for future work.Source: Limitations · p. 9

A SpecPV Algorithm Description

The section introduces SpecPV pseudocode for clarity.

A SpecPV Algorithm Description

B YARN-Based Extension of the EAGLE-3 Module

The EAGLE-3 draft module needs positional-embedding extension for long contexts; the section describes YARN-based fine-tuning and reports that training loss decreases with more training-time test steps.

B YARN-Based Extension of the EAGLE-3 Module

C Ablation Study

The ablations examine retrieval-score reduction strategies and partial-cache refresh intervals, finding that mean reduction performs best among the tested strategies and longer intervals reduce similarity to full verification.

C Ablation Study

  • The appendix evaluates retrieval-score reduction strategies and buffer-size effects as separate ablation factors.Source: C Ablation Study · p. 11
  • Mean reduction yields the highest similarity and slightly longer draft accept length; reduction type otherwise has only a minor effect on performance.The comparison covers Qwen3-8B and LLaMA-3.1-8B-Instruct on GovReport under a 2K KV budget; Table 5 reports the reduction results.Source: C Ablation Study · pp. 11–12, Table 5 · p. 12
  • Across models on QMSum, similarity to full verification gradually decreases as the partial-KV refresh interval increases.The interval is controlled by buffer size; the authors typically set it to one verification step's tokens plus 20.Source: C Ablation Study · p. 12

D Generation Settings

Generation settings differ by model and task, while SpecPV’s block, sink-token, window, and buffer sizes are fixed. On QMSum, larger refresh intervals trade greater speedup for greater deviation from full verification.

D Generation Settings

  • LLaMA 3.1-8B-Instruct uses temperature 0, while Qwen3 uses non-thinking mode at temperature 0 for summarization and continuation, and thinking mode for question answering.Qwen3 thinking mode uses temperature 0.6, TopP 0.95, and TopK 20.Source: D Generation Settings · p. 12
  • On QMSum, larger refresh intervals yield higher speedup but greater deviation from full verification; Figure 8 plots ROUGE-L and speedup.ROUGE-L is shown with solid lines and speedup with dashed lines.Source: Figure 8 · p. 12
  • Table 6 reports summarization performance across temperatures and partial-KV budgets, with RL denoting ROUGE-L and BLT denoting BLEURT.Source: Table 6 · p. 12
  • SpecPV uses block size 16, 32 sink tokens, a 128-token local window, and buffer size 20.Source: D Generation Settings · p. 12

E Ablation on Temperature

Under non-deterministic decoding, SpecPV’s summarization quality remains broadly comparable to full verification across tested temperatures, while higher temperature is associated with shorter draft acceptance. The experiments use two temperatures and report context-length results alongside speedup and accept length.

E Ablation on Temperature

  • Experiments use T = 0.7 and T = 1.0 with topk = 0 and topp = 0.9; Table 7 reports context-length and temperature results using α for speedup and τ for average draft accept length.Source: E Ablation on Temperature · pp. 12–13, Table 7 · p. 13
  • With non-deterministic decoding, SpecPV maintains ROUGE-L and BLEURT scores comparable to full verification; a KV budget of 512 produces a more noticeable ROUGE-L drop, while semantic quality remains largely preserved.Source: E Ablation on Temperature · p. 13
  • At T = 1.0, average accept length becomes smaller; higher temperatures smooth token distributions, increase draft-target discrepancy, and cause more mismatches, while speedup trends remain consistent.Source: E Ablation on Temperature · p. 13

Related papers

Loading 2512.02337v2…