SpecVLM: Fast Speculative Decoding in Vision-Language Models

1 INTRODUCTION

• Speculative decoding reduces autoregressive steps, while question-adaptive visual compression reduces visual tokens and draft prefill compute, memory use, and KV-cache traffic.The combined approach is presented as preserving the target model’s output distribution.Source: §1 · pp. 1–2
• In LLaVA-1.6-7B, LLM prefill is the dominant latency bottleneck after the vision encoder and projector, with accumulated decoding latency another critical bottleneck.Figure 1(a) presents the latency breakdown for batch size 1 and temperature 0 on an NVIDIA RTX 4090.Source: §1 · p. 1, Figure 1 · p. 2
• VLM speculative decoding must address heavy visual prefill and large KV caches, while visual-token counts grow with resolution and video length.The paper identifies limited system-level exploration of how to select or combine compression methods within speculative decoding.Source: §1 · p. 2
• EagleVLM delivers 1.5–2× speedups over standard autoregressive inference while maintaining output quality.It is an EAGLE-2-inspired VLM speculative-decoding baseline.Source: §1 · p. 2
• Up to 2.5–2.9× end-to-end acceleration is reported within 5 epochs, without sacrificing output fidelity.The result is for SpecVLM, which combines the elastic compressor with online-logit training.Source: §1 · p. 3
• Online-logit distillation trains the draft without large offline distillation datasets and reveals a training-time scaling effect.The introduction reports that longer online training increases average accepted length and speculative efficiency.Source: §1 · p. 3
• SpecVLM is evaluated on the LLaVA benchmark suite and MMMU sets, with performance reported across model sizes and task difficulties.The introduction describes the evaluation as consistent and robust.Source: §1 · p. 3

2.1 SPECULATIVE DECODING

• Most speculative-decoding approaches assume text-only models, overlooking VLM visual-prefill costs, large KV caches, and sequence growth with resolution or video.Offline teacher-logit corpora can also be cumbersome at multimodal scale.Source: §2.1 · p. 3

2.2 VISUAL TOKEN COMPRESSION FOR VLMS

• Visual-token compression includes pruning, pooling, convolution, and resampling; parameter-free approaches are cheaper but less expressive than learned methods.Learned resamplers and convolutional compressors add parameters and compute or latency.Source: §2.2 · p. 3
• Prior compression work often studies compression in isolation; SpecVLM integrates a question-adaptive elastic compressor into speculative decoding while maintaining lossless decoding relative to the target.The integration combines fewer visual tokens with fewer target forward passes.Source: §2.2 · p. 4

3.1 PRELIMINARIES: BOTTLENECK ANALYSIS

• Speculative decoding proceeds in rounds of draft proposals and target verification, with speedup most sensitive to draft latency and average accepted length.For VLMs with long or high-resolution inputs, the draft’s expanding KV cache can dominate latency, making visual-token compression important.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 SPECVLM

• EagleVLM achieves 1.5–2× speedups over standard autoregressive inference while maintaining output quality.It pairs a lightweight draft with the full target model, which verifies proposed tokens in parallel.Source: §3.3 · p. 4

3.4 ELASTIC VISION COMPRESSOR

• The draft uses the target’s penultimate-layer features, while an elastic compressor selects among pruning, pooling, convolution, and resampler strategies for each question.The compressor includes question-conditioned expert weighting, concatenation across compression scales, and dynamic selection of one operator per input.Source: §3.3 · pp. 4–5, §3.4 · pp. 5–6

3.5 ONLINE-LOGIT DISTILLATION

• Online distillation aligns draft and target logits with cross-entropy and their features with Smooth L1, using teacher outputs generated during training.The target provides token-level logits and penultimate-layer features; the joint supervision improves alignment and increases average accepted tokens per round.Source: §3.5 · p. 6

4.1 EXPERIMENT SETUP

• Evaluation covers six LLaVA benchmarks and six MMMU-V1 disciplines, using five target models with both greedy and stochastic decoding.The LLaVA suite uses the first 60 image-containing question-answer pairs per benchmark; MMMU uses 30 validation pairs per subject.Source: §4.1 · p. 6
• Experiments use batch size 1 and report wall-clock speedup (τ) and average accepted tokens (σ); output fidelity is not separately evaluated because decoding preserves the target distribution.Training freezes the target VLM and updates only the draft decoder layer and vision compressor.Source: §4.1 · pp. 6–7

4.2 RESULTS

• For LLaVA-1.6-13B, SpecVLM reaches 2.38× on LLaVA-Wild and up to 2.70× on MMBench; on MMMU-V1, it reaches 2.14× on Art and up to 2.63× on Math.The MMMU results maintain high average accepted length.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8
• SpecVLM consistently improves speedup and average accepted length over EagleVLM across model sizes, benchmarks, and MMMU subject domains.The SpecVLM–EagleVLM gap widens with model size, and improvements are stable under both deterministic and stochastic decoding.Source: §4.2 · p. 8

5.1 COMPONENT ANALYSIS

• Dynamic selection in full SpecVLM achieves 2.03× average speedup and 3.69 average accepted length, the best overall performance among the tested configurations.The comparison includes the EagleVLM baseline, weighted expert fusion, multi-granularity concatenation, and dynamic selection.Source: Table 3 · p. 8, §5.1 · p. 8
• Pruning and pooling remain robust at 10–20× compression, while convolution performance drops at 10× or more and a 2-query resampler performs best.The reported strategy uses 10–20× pooling or pruning for simple or text-based tasks, and 3× convolution or a 2-query resampler for detailed visual inputs.Source: Table 4 · p. 8, §5.1 · p. 8

5.2 TRAINING-TIME SCALING

• Across variants and model sizes, accepted length and speedup rise with each training epoch; the 5th epoch is selected as the cost-performance trade-off.The draft starts to converge by epoch 5 and reaches optimal performance by epoch 8, but further training substantially raises computational cost.Source: Figure 5 · p. 9, Figure 6 · p. 9, §5.2 · p. 9

6 DISCUSSION AND LIMITATIONS

• Compression scales are manually configured; instance-wise adjustment remains a proposed direction.The current setup does not dynamically adjust compression ratios for each input.Source: §6 · p. 9
• The study focuses on training-time scaling, leaving larger training corpora and deeper drafts as unexplored axes.The authors suggest these dimensions could reveal broader scaling laws for speculative decoding in VLMs.Source: §6 · p. 9
• Draft and optimization hyperparameters are not exhaustively tuned, so reported performance may be suboptimal.Examples include draft depth, tree Top-K, number of nodes, learning rate, and batch size.Source: §6 · p. 9

7 CONCLUSION

• SpecVLM combines an EAGLE-2-style baseline with an elastic visual compressor and online-logit distillation for draft-model training.The online protocol avoids reliance on large offline datasets.Source: §7 · p. 9
• Experiments on LLaVA and MMMU show consistent speedup and efficiency gains across model sizes and task complexities while preserving output quality.Source: §7 · p. 9
• Extending online draft training improves average accepted length and inference speedup independently of model capacity or dataset size.Source: §7 · p. 10

• Table 5 reports SpecVLM and EagleVLM speedup (τ) and average accepted length (σ) on LLaVA benchmarks (epoch=1), with results obtained on an NVIDIA A100-80G.Source: Table 5 · p. 13, §A.1 · p. 13
• Across LLaVA benchmarks and model sizes, SpecVLM consistently exceeds EagleVLM in speedup (τ) and average accepted length (σ) under T=0 and T=1 decoding.Table 5 reports these metrics at epoch 1 on A100-80G.Source: Table 5 · p. 13, §A.1 · p. 13
• On MMMU-V1 across academic subjects, SpecVLM consistently achieves higher speedups and longer accepted lengths than EagleVLM across all subjects and model variants, with especially notable gains in Math and Physics.Source: §A.1 · p. 13

• The reported training-time trend applies across all models: both τ and σ rise with epochs, and longer online training enhances speculative efficiency while compounding the elastic compressor’s benefits.The authors state sufficient training is important to realize SpecVLM’s gains across sizes and regimes.Source: §A.2 · p. 13
• As epochs increase, SpecVLM maintains an advantage over EagleVLM in speedup (τ) and average accepted length (σ), with the gap widening over time.Table 7 reports training-time scaling on LLaVA-Bench-In-the-Wild at T=0 on MI250-64G.Source: §A.2 · p. 13, Table 7 · p. 14
• For Open-LLaVA-1.6-7B on LLaVA-Bench-In-the-Wild, both speedup (τ) and average accepted length (σ) increase with training epochs.Figure 7 reports results at T=0 on MI250-64G.Source: Figure 7 · p. 14