SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

1 Introduction

• Vid-LLM inputs can reach tens of thousands of video tokens, creating quadratic prefilling attention overhead and memory-bound decoding.The introduction gives an example of more than 1 million tokens for a two-minute video at 60 FPS in LLaVA-OneVision.Source: §1 · p. 1
• Existing video-token pruning reduces memory and compute but can lose information needed for spatial and temporal understanding, while offering limited decoding speedup.The cited methods generally prune during prefilling and still repeatedly access full model parameters during decoding.Source: §1 · pp. 1–2
• Speculative decoding is challenging for Vid-LLMs because draft-model KV caches grow with video length and existing long-context methods do not exploit video-token redundancy and attention patterns.These challenges motivate pruning video tokens for the draft model to reduce its KV cache and improve speculation efficiency.Source: §1 · p. 2
• SPECVLM uses verifier attention to retain highly informative tokens and spatially uniformly reduce low-attention tokens before draft-model prefilling.The framework is training-free and uses staged video token pruning.Source: §1 · p. 2
• The framework builds on the finding that draft-model speculation has low sensitivity to random video-token pruning at low pruning ratios.This observation motivates extending pruning to higher ratios through staged token selection.Source: §1 · p. 2
• SPECVLM prunes over 90% of video tokens for the draft model while retaining nearly 90% of speculation accuracy, with up to 2.68× and 2.11× lossless decoding speedup for LLaVA-OneVision and Qwen2.5-VL, respectively.Source: §1 · p. 2

2.1 Naive Speculative Decoding for Vid-LLMs

• In normal batches, SD speedup primarily depends on average accept length and the draft-to-target latency ratio because verification-to-target latency is close to 1.The speedup expression also includes the verification-to-target latency ratio.Source: §2.1 · p. 2
• For long video inputs, the draft model's latency bottleneck shifts from model parameters to its accumulated KV cache, so reducing draft KV size can cut latency.Figure 2(a) presents a draft-latency breakdown for LLaVA-OneVision-7B.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 Speculation Sensitivity for Token Pruning

• ≤50% pruning does not significantly compromise average accept length, whereas >50% random pruning causes a considerable drop and 100% pruning significantly lowers speculation accuracy and overall speedup.At certain pruning ratios, random pruning even improves average accept length; the authors attribute this to redundancy in long-video inputs.Source: §2.2 · p. 3
• Random pruning is stochastic and context-agnostic, with no deterministic performance lower bound and a nonzero chance of discarding tokens for critical semantic elements.The authors argue that low-ratio pruning also fails to sufficiently reduce KV-cache costs as video length grows.Source: §2.2 · p. 3
• SPECVLM's overview shows verifier-guided Top-P retention of informative tokens followed by spatially uniform reduction of low-attention tokens.The verifier guides pruning during prefilling; the draft model then speculates during decoding.Source: Figure 3 · p. 4

3.1 Attention-Guided Token Rating from Target Model

• SPECVLM ranks video tokens using language-to-video attention from the target model, whose language tokens attend to a few video tokens with high concentration.Target attention is computed regardless of speculation, so using it adds minimal latency overhead.Source: §3.1 · p. 4

3.2 Two-Stage Token Pruning for Draft Model

• A long-tailed attention distribution places over 50% of total attention on 10% of video tokens, while the remaining 90% have uniformly low scores.The low-score tail motivates treating the tokens in two groups for pruning.Source: §3.2 · p. 5
• Stage I retains tokens until their cumulative attention exceeds a threshold; Stage II uniformly samples the remaining low-attention tokens at fixed spatial intervals.The retained sets are combined before draft-model prefilling, reducing its KV cache to 1 − r of its original size.Source: §3.2 · p. 5
• Tables 1 and 2 report average accepted length τ, decoding speed, and speedup at r = 90% across four LLaVA-OneVision benchmarks and three Qwen2.5-VL benchmarks.Table 2 excludes MVLU because its sampling protocol is incompatible.Source: Table 1 · p. 6, Table 2 · p. 6
• Tree attention lets the target verify draft-tree candidates in parallel, enabling more tokens to be accepted per forward pass and improving decoding speed.SPECVLM adopts the static tree structure from EAGLE.Source: §3.2 · pp. 5–6

4 Experiments

• The evaluation covers VideoDetailCaption, MVBench, MVLU, and LongVideoBench, measuring decoding speed, wall-time speedup, and average accept length.The authors do not evaluate output quality, stating that their method is lossless.Source: §4 · p. 6

4.1 Main Result

• Up to 2.68× speedup for LLaVA-OneVision under Std.-SD and 2.11× for Qwen2.5-VL under Std.-SD; Self-SD reaches 1.33× and 1.59×, respectively.With 90% token reduction on VideoDetailCaption, average accept length for LLaVA-OneVision-7B falls by only 5%, 40% of the degradation observed with SD-Rand.Source: §4.1 · pp. 6–7

4.2 Scaling Law for Pruning Ratio

• As pruning ratios increase, SPECVLM maintains higher speculation accuracy than random pruning, and their gap widens at higher ratios.Figure 6 plots average accepted length across baselines as pruning ratios scale up.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Ablation Study

• Under a fixed KV budget, SD-Attention outperforms window-based pruning, while spatially uniform removal of low-attention tokens improves average accept length in most cases.On MVBench with LLaVA-OneVision / Self-SD 7B-7B, the Top-P retention variant is an outlier, attributed to repetitive action sequences and spatial redundancy.Source: Figure 7 · p. 7, Figure 8 · p. 7, §4.3 · p. 8
• Relying only on attention-based retention can lose structural information in uniformly low-attention areas and reduce speculation accuracy.The authors compare the full method with a variant lacking Stage II spatially uniform reduction.Source: Figure 8 · p. 7, §4.3 · p. 8

4.4 Exploration of Diverse Pruning Criteria

• Temporal-redundancy methods underperform spatially uniform pruning and even random pruning within speculative decoding.The compared approaches include regular full-frame dropping, FastVID-style frame pruning, and DyCoke-style token merging.Source: Figure 7 · p. 7, §4.4 · p. 8

4.5 Impact of Decoding Steps

• Across 50 VideoDetailCaption instances at 90% pruning, average accepted length is 3.39 for the first 10 decoding steps versus 3.48 overall.The paper also reports a higher average accepted length at the first decoding step, which it attributes as likely to fixed opening sentence structures.Source: §4.5 · p. 8, Table 3 · p. 8

5.1 Speculative Decoding

• Speculative decoding uses draft-and-verify generation; approaches include separate or self-speculative drafts, efficiency-focused drafting, and tree-based speculation.Autoregressive draft models can improve speculation accuracy but require their own KV caches, adding memory overhead for long video inputs.Source: §5.1 · p. 9
• Long-context speculative decoding methods address KV-cache overhead through hierarchical speculation, throughput-latency trade-offs, or constant-memory draft models.The paper states that Appendix D explains why these methods are not suitable for Vid-LLMs.Source: §5.1 · p. 9

5.2 Visual Token Reduction

• Visual-token reduction work prunes or merges tokens using importance and redundancy, while SPECVLM prunes post-encoder and is orthogonal to Tang et al. (2025)'s temporal key-frame extraction.The reviewed video approaches include cross-frame merging, static/dynamic token selection, density-based pruning, and compression based on frame uniqueness.Source: §5.2 · p. 9

6 Conclusion

• SPECVLM is presented as the first training-free speculative-decoding framework tailored to accelerate video LLMs.It uses verifier-guided attention to remove redundant video tokens and reduce the draft model’s KV cache without compromising generation quality.Source: §6 · p. 9
• The framework achieves up to 2.68× speedup on LLaVA-OneVision-72B and 2.11× on Qwen2.5-VL-32B across multiple video-understanding tasks.Source: §6 · p. 9
• The authors propose latency-efficient Vid-LLM reasoning from a token-sparsity perspective as a direction for further research.Source: §6 · p. 9

8 Ethical Considerations

• All experiments use open-source datasets and models, with no sensitive data, human subjects, or commercial use reported.Source: §8 · p. 10

• The method is primarily applicable to resource-constrained long-video scenarios where memory bandwidth is the dominant bottleneck.Source: §9 · p. 10
• Draft-model choice requires care for optimal speedup, even though its inference overhead is relatively small or avoidable with Self-SD.Source: §9 · p. 10
• Using an existing Vid-LLM draft without fine-tuning constrains maximum acceleration; smaller, faster drafts remain a prospective direction.Source: §9 · p. 10

• Experiments cover video captioning and description on VideoDetailCaption, MVBench, MVLU, and LongVideoBench, with 50 randomly sampled instances per benchmark.Source: A Experimental Details · p. 13
• LLaVA-OneVision inputs use 64 frames for the 72B model and 128 for the 7B model, producing 196 × 64 and 196 × 128 video tokens, respectively.Source: A Experimental Details · p. 13
• Models generate 256 tokens with greedy decoding on eight NVIDIA A100 GPUs, and the default video-token pruning ratio is 90%.Source: A Experimental Details · p. 13
• The model-level λr is selected through one offline evaluation on 2–4 calibration instances per task; Tables 1 and 2 use 0.4 and 0.5, respectively.Source: A Experimental Details · p. 13

B Impact of Tree Attention

• Tree drafting increases average accept length and, with tree attention, speculation accuracy; the tree expands higher-probability candidates more deeply and widely.Source: B Impact of Tree Attention · p. 13, Table 4 · p. 13, Figure 9 · p. 13

C Breakdown of Computation

• Video-token pruning reduces draft-model prefill length, making added prefilling time negligible relative to target-model inference while reducing target decoding time.Source: C Breakdown of Computation · p. 13, Table 5 · p. 13

D Why is Long-context SD Technologies Not Suitable for Vid-LLMs?

• Long-context SD methods using StreamingLLM or sliding-window caches lack visual-modality awareness, so naive use can leave the draft attending to too little uncompressed video to capture overall semantics.Source: D Why is Long-context SD Technologies… · p. 14

E Discussion on Smaller Draft Model

• Pruning can reduce small drafts’ input-length burden: single-layer LLaVA-OneVision-0.5B latency rises significantly with input length, and token pruning still provides a performance benefit.Source: E Discussion on Smaller Draft Model · p. 14, Table 6 · p. 14

• The draft model speculates using video tokens pruned by 90%, while the target model verifies against the original video tokens.Source: F Case Study · p. 14, Figure 10 · p. 15, Figure 11 · p. 15
• Despite pruning, the draft retains essential visual cues and maintains a coherent understanding of the overall video structure.Source: F Case Study · p. 14
• The examples visualize SPECVLM on VideoDetailCaption with LLaVA-OneVision-72B / 7B and Qwen2.5-VL-32B / 7B.Source: Figure 10 · p. 15, Figure 11 · p. 15