PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, Yonatan Geifman

arXiv:2604.09557v2cs.DCcs.AI

TL;DR

Speculative Decoding performance depends on input data and serving conditions, but existing benchmarks provide limited diversity and realism. SPEED-Bench addresses this with diverse qualitative and throughput splits plus unified production-grade measurement, showing that SD performance is highly data- and regime-dependent.

  • Problem

    Existing SD benchmarks provide limited semantic diversity, throughput-oriented evaluation, and production-realistic implementations despite data-dependent performance.

  • Method

    SPEED-Bench combines semantically diverse qualitative data, realistic throughput workloads, and a unified framework compatible with production-grade and research-oriented engines.

  • Results

    SD performance is deeply data-dependent and sensitive to the serving regime across diverse text domains, batch sizes, and input sequence lengths.

  • Takeaways & Limitations

    SPEED-Bench supports more rigorous assessment of SD robustness and efficiency across diverse domains and realistic deployment settings.

  • Takeaways & Limitations

    Synthetic random-token benchmarks can misrepresent SD throughput and MoE latency because they produce unrealistic predictability and expert routing.

Abstract

from arXiv · show

Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.

1. Introduction

SPEED-Bench addresses fragmented and unrepresentative speculative decoding evaluation with diverse data, realistic throughput workloads, and a unified production-grade measurement framework. It enables analysis across domains, input sequence lengths, and concurrencies while exposing effects masked by traditional benchmarks.

  • Motivation: Autoregressive decoding is memory-bound at low concurrency because parameter movement from HBM to on-chip caches dominates computation time.KV-Caching reduces recomputation costs but does not remove this bottleneck.
  • Evaluation gaps: Existing evaluations use inconsistent data domains, limited prompt volumes, short input sequences, and predominantly BatchSize(BS) = 1 settings.Larger batches shift serving toward compute-bound inference, which can diminish speculative decoding speedups.
  • SPEED-Bench: SPEED-Bench provides a Qualitative split for semantic coverage, a Throughput split for realistic serving scenarios, and a unified measurement framework.The benchmark evaluates varying text domains, ISLs, and concurrencies using production-grade inference engines.
  • Benchmark design: The benchmark supports qualitative draft-accuracy analysis and throughput measurement across varying input sequence lengths and concurrencies without massive-scale testing.Its framework is integrated with production-grade engines to measure real-world speedups.
  • Empirical analysis: The authors use SPEED-Bench to analyze properties and side effects that traditional benchmarking methodologies often mask.The supplied passage identifies this empirical analysis as a demonstration of the benchmark’s utility.

2. Related Work

Prior speculative decoding work balances drafter accuracy against latency, evolving from standalone draft models to lightweight heads integrated into target models. Existing evaluations commonly rely on narrow or low-variance datasets and task-specific subsets, limiting benchmark diversity.

  • Speculative Decoding: Speculative decoding initially used smaller standalone models as drafters, while Medusa and EAGLE integrated lightweight drafting heads into the target model.These approaches aim to reduce the memory and compute overhead of running a separate drafting model.
  • Speculative Decoding: SD efficiency depends on balancing drafter accuracy and latency.The related work frames this tradeoff as central to speculative decoding performance.
  • Benchmarking Methodology: Existing SD evaluations often use MT-Bench or domain-specific coding and math subsets with limited sample diversity and narrow task coverage.Examples include MT-Bench evaluations limited to 10 samples per category, simple Python-only HumanEval tasks, and Alpaca evaluations on the training set.

3. Background

Speculative decoding reduces the sequential cost of autoregressive inference by using a lightweight draft model to predict future tokens for target-model verification. Its benefits are especially relevant in memory-bound, low-concurrency settings and can extend to mixture-of-experts models, while evaluation uses acceptance and throughput metrics.

  • Speculative Decoding: Autoregressive generation of a sequence of length T requires T sequential forward passes and is memory-bound at low concurrencies on modern GPUs.Latency is dominated by loading model parameters from HBM to on-chip caches, leaving compute units underutilized.
  • Speculative Decoding: Speculative decoding uses a lightweight draft model Md to predict γ future tokens, which the target model then verifies.
  • Speculative Decoding: Mixture-of-experts models are well-suited for speculative decoding because each token activates only a sparse subset of experts, allowing benefits even at high batch sizes.Verification loads additional experts for draft tokens, but activating all experts across a massive number of tokens is required to become fully compute-bound.
  • Metrics: Conditional Acceptance Rate measures the probability of accepting draft token xi given an accepted draft prefix, while Acceptance Length measures expected tokens generated per verification step.Acceptance Length includes the “free” verification token.
  • Metrics: Throughput (Output TPS) measures tokens generated per second across concurrent requests, whereas User TPS measures tokens per second for one request and proxies end-user latency.

4. Overview of SPEED-Bench

SPEED-Bench combines a multi-purpose dataset with a unified measurement framework, dividing evaluation into Qualitative and Throughput splits tailored to different constraints. Its framework standardizes comparisons across production and research engines by ensuring identical input sequences.

  • SPEED-Bench provides a multi-purpose dataset and unified measurement framework organized into two constraint-specific splits.The splits target distinct evaluation needs.
  • Qualitative Split: The Qualitative Split selects samples from 18 public sources to maximize semantic diversity for efficient, high-fidelity speculation-quality measurement.It evaluates acceptance rates and acceptance lengths across fine-grained semantic categories.
  • Throughput Split: The Throughput Split groups samples into three difficulty categories across fixed ISL buckets from 1k–32k to construct stable throughput-latency Pareto curves.This design supports system-speedup evaluation under varying batch-size and input-length regimes.
  • Measurement Framework: The measurement framework supports SGLang, vLLM, TensorRT-LLM, and SpecBench while externally processing text so every engine receives identical sequences.This isolates speculation-algorithm effects from system implementation differences.

5. SPEED-Bench Qualitative Split

SPEED-Bench’s Qualitative Split uses a compact, semantically diverse sample of LLM inputs to measure speculator efficacy across domains and entropy while reducing evaluation cost. It combines embedding-based diversity selection with greedy local-swap refinement, achieving substantially lower semantic similarity than existing selection strategies.

  • Data Composition: The split aggregates 18 public datasets into 11 categories, selecting 80 samples per category for 880 total samples.Its guiding principle is semantic heterogeneity across data sources and samples.
  • Selection Algorithm: Samples are mapped to unit-normalized dense vectors with OpenAI’s text-embedding-3-large, enabling cosine-similarity-based diversity measurement.For a target subset size k, selection minimizes total pairwise similarity among candidates.
  • Selection Algorithm: A greedy selection algorithm with Local Swap Refinement approximately minimizes the diversity objective despite its NP-hard exact formulation.The method initializes a random subset, iteratively appends candidates, then swaps selected and unselected samples when the objective strictly decreases until convergence.
  • Results: Average semantic similarity decreases 40% versus SpecBench and 83% in Multilingual, while outperforming random selection on identical data.Figure 2 reports lower average similarity than both random selection and SpecBench across all categories; lower similarity indicates greater diversity.

6. SPEED-Bench Throughput Split

The Throughput Split evaluates speculative decoding under realistic, high-concurrency serving conditions, spanning latency Pareto curves across diverse scenarios. It uses controlled real-data workloads to avoid synthetic-benchmarking distortions and supports domain-specific speedup estimates under fixed serving constraints.

  • Throughput Split: The Throughput Split targets system-level efficiency under high concurrency (BS > 1) and long ISLs, where systems shift from memory-bound to compute-bound regimes.This shift changes the cost-benefit ratio of speculative decoding verification.
  • Throughput Split: SPEED-Bench provides workloads for constructing throughput-latency Pareto curves across diverse serving scenarios.
  • The Pitfalls of Synthetic Benchmarking: Random token batches can misrepresent speculative decoding because predictability-dependent inputs trigger trivial responses and other distortions in autoregressive measurements.Random noise also fails to trigger realistic mixture-of-experts routing, potentially collapsing routing to a subset of experts and skewing step-latency measurements.
  • Data Composition: The split controls input sequence lengths with fixed 1k, 2k, 8k, 16k, and 32k buckets drawn from 8 publicly available datasets.Prompts are truncated or padded with a neutral suffix, and lengths use the o200k base tokenizer for deterministic prefill load.
  • Estimating Domain-Specific Speedups: The Throughput Split estimates fine-grained domain-specific speedups under specified batch and ISL constraints without requiring the high data volume needed for stable measurements.Per-step latency is primarily governed by system constraints and serving parameters, including memory bandwidth, batch size, and ISL.

7. Measurement Framework

SPEED-Bench uses a unified thin-client framework to standardize input handling across inference engines and supports realistic concurrent serving evaluation. It records streaming-response timing and integrates with major production frameworks to measure latency and throughput under engine optimizations.

  • Framework Design: A unified lightweight thin client standardizes raw-text processing and draft-target model evaluation across inference engines.It mitigates inconsistencies from BOS-token insertion and chat-template application that can alter drafted sequences.
  • Metrics and Concurrency: Asynchronous request dispatch and streaming-response analysis support concurrency experiments and fine-grained performance measurement.The framework uses Python’s asyncio event loop and computes ARs from newly generated tokens and successful multi-token speculation chunks.
  • Metrics and Concurrency: Timestamped streaming responses yield TTFT, step latency, total request latency, User TPS, and overall output TPS.TTFT is reported for completeness even though speculative decoding primarily optimizes the decode phase.
  • Ecosystem Integration: Native integration with SGLang, vLLM, and TensorRT-LLM enables evaluation alongside CUDA Graphs, continuous batching, and kernel fusion.SPEED-Bench complements rather than replaces research toolkits such as SpecBench, which targets native PyTorch/HuggingFace evaluation.

8. Experiments and Observations

SPEED-Bench evaluates speculative decoding across diverse semantic domains and realistic serving regimes using production-grade engines and modern open models. Its experiments reveal domain-, batch-, engine-, and sequence-length-dependent behavior that conventional or synthetic benchmarks can obscure.

  • Experimental Setup: Experiments evaluate N-Grams, Vanilla SD, EAGLE3, and native MTP heads on large open models using draft chains in production-grade frameworks.The evaluation targets Llama 3.3 70B, GPT-OSS 120B, Qwen3 235B, Qwen3-Next, and DeepSeek R1, primarily on a single NVIDIA B200 GPU.
  • Qualitative and Accuracy Results: Across the Qualitative Split at BS = 32 and DL = 3, low-entropy Coding and Math domains produce larger ALs than high-entropy Roleplay.Throughput-Split results similarly show lower ALs for High Entropy prompts across context lengths, with deviations linked to training-data distributions.
  • Qualitative and Accuracy Results: Qwen3-Next’s pretrained MTP head maintains higher ALs than post-trained EAGLE3, while Vanilla SD sustains accuracy better at longer speculation horizons.These trends are reported from AL scaling across different DLs and indicate differing robustness as speculation length increases.
  • Vocabulary Pruning: Vocabulary pruning removes approximately 22% of target tokens in Multilingual inputs, and domain responses show high variance under this optimization.EAGLE3 typically prunes to 32k tokens; although effective for standard inputs, the heuristic degrades performance on long-tail user inputs.
  • Throughput Evaluation: 23%: synthetic random-input benchmarking overestimates SD throughput by an average of 23% compared with SPEED-Bench.The discrepancy is attributed to skewed acceptance rates, while random inputs also differ from realistic inputs in the baseline autoregressive setting because of expert imbalance.
  • Throughput Evaluation: The optimal DL depends on concurrency: longer drafts are preferred at lower, memory-bound batch sizes, whereas DL = 1 is preferred as verification becomes compute-bound.TensorRT-LLM achieves higher peak throughput through a unified CUDA graph, while vLLM incurs slight host communication overheads but supports more flexible dynamic drafting.

9. Conclusion

SPEED-Bench establishes a unified evaluation ecosystem for speculative decoding research and production-grade deployment through semantically diverse qualitative data and throughput-oriented serving evaluation. Its results show that SD performance depends strongly on data diversity and serving regime, including batch size and ISL-dependent draft lengths.

  • Evaluation ecosystem: SPEED-Bench establishes a unified evaluation ecosystem for SD research and production-grade deployment.It provides a semantically diverse Qualitative Split and a Throughput Split focused on large batches and fixed ISLs.
  • Evaluation ecosystem: The splits quantify robustness across text domains, identify batch-size and ISL-dependent optimal DLs, and measure speedups across varied serving settings.These capabilities support analysis of critical system properties in realistic evaluation regimes.
  • Empirical findings: Empirical results demonstrate that SD performance is deeply data-dependent and sensitive to the specific serving regime.This motivates evaluating methods across diverse text domains and varied serving settings rather than relying on a single workload.

A. Detailed Data Comparison with SpecBench … E.2. Topic Latching

The appendices detail SPEED-Bench’s broader data design, construction and selection procedures, showing improved semantic diversity and benchmark stability while exposing misleading behaviors from synthetic random-token inputs.

  • A. Detailed Data Comparison with SpecBench: SPEED-Bench’s Qualitative Split spans 11 categories, consolidates overlapping SpecBench categories, replaces Translation with Multilingual, and adds longer multi-turn samples plus task metadata.The metadata includes difficulty and sub-categories for fine-grained analysis of speculative decoding algorithms.
  • A. Detailed Data Comparison with SpecBench; B. Extended Details on Dataset Collection: The Throughput Split measures system and user TPS under large-batch fixed-ISL regimes from 1k-32k, a split unavailable in SpecBench.Throughput prompts use truncation or the neutral suffix ”please answer now” to enforce fixed ISLs.
  • B. Extended Details on Dataset Collection: Dataset construction filters code repositories to permissively licensed sources and verifies that CoSER roleplay books are public domain.The stated filters target open-source licensing compliance and exclude copyrighted roleplay works.
  • C. Alternative Selection Algorithm for the Qualitative Split: The appendix compares uniform random sampling with a convex quadratic-programming approximation of the diversity objective, using independently generated benchmarks from identical source pools.The comparison preserves the same number of samples per category and evaluates subsets with Llama 3.3 70B and EAGLE3.
  • C.1. Comparison Against Uniform Sampling: Optimized selection significantly reduces category-level AL instability and improves consistency of category rankings relative to uniform sampling.These evaluations focus on benchmark realizations whose category measurements and easy-versus-hard conclusions should remain stable.
  • C.2. Quadratic Programming Approximation: Greedy Selection with Swap Refinement achieves similar diversity scores to the QP approach while being faster and more scalable to large candidate pools.The QP formulation minimizes w⊤Gw subject to P wi = k over weights w ∈[0, 1]N.
  • D. Visualizing Semantic Diversity: Similarity visualizations show repetitive prompt clusters in SpecBench but dispersed low-similarity patterns in SPEED-Bench, indicating broader semantic coverage.The comparison covers Translation/Multilingual and Math, with darker green denoting redundancy and lighter yellow denoting diversity.
  • E. Pitfalls of Synthetic Benchmarking; E.1. Trivial Responses; E.2. Topic Latching: Synthetic random-token prompts can trigger generic repetitive responses or topic latching, making draft models appear unusually predictable.Example Output (Base: GPT-OSS 120b, Drafter: EAGLE3, Draft Length:3, Average AL: 3.44) illustrates a trivial response, while Average AL: 1.877 accompanies topic latching.

F. Tree-Based Verification Experiments

SPEED-Bench supports tree-based speculative verification in production serving environments, demonstrated with EAGLE3 and a Qwen3 235B target model in SGLang. Increasing either branching factor or draft length improves average AL by expanding the candidate verification space.

  • Tree-Based Verification Experiments: SPEED-Bench integrates directly with production-grade serving engines while externalizing data processing, tokenization, and metric collection.This lightweight wrapper is method-agnostic and can evaluate SD algorithms supported by the underlying backend.
  • Tree-Based Verification Experiments: SPEED-Bench supports tree-based speculative verification, demonstrated using EAGLE3 with a Qwen3 235B target model in SGLang.The framework fully supports tree-based strategies beyond its primary focus on draft-chain methods.
  • Tree-Based Verification Experiments: Increasing either the branching factor (top-k) or draft length (DL) improves average AL in tree-based speculative decoding.The experiments vary both parameters, with larger values expanding the candidate verification space available to the target model.

G. Expert Imbalance in Synthetic Benchmarking … N. Long-Context Inaccuracy in Existing EAGLE3 Models

SPEED-Bench exposes artifacts in synthetic benchmarking and supports domain-specific speedup estimation by separating serving step times from domain-dependent accepted lengths. Across validation and analyses, it reveals entropy-, context-, engine-, vocabulary-, and configuration-dependent behavior affecting speculative decoding evaluation.

  • G. Expert Imbalance in Synthetic Benchmarking: Random tokens imbalance MoE routing, disproportionately activating some experts and leaving 20-30% unavailable in certain layers, unlike relatively uniform SPEED-Bench inputs.This occurs for GPT-OSS 120B during 8k prefill at batch size 32 and confirms that synthetic noise fails to reproduce semantic routing behavior.
  • H. Estimating Domain-Specific Speedups using the Throughput Split; H.1. Protocol: Throughput Split measurements estimate domain-specific speedups by combining serving-dependent step times, tar and tsd, with domain-dependent average accepted length, AL.Practitioners measure tar and tsd on a matching ISL bucket and batch size, measure AL on representative target-domain prompts, then calculate projected speedup.
  • H.2. The Necessity of Realistic Data: Real semantic data is necessary because random-token inputs yield inaccurate SD step times and can also distort autoregressive timings on MoE models.The Throughput Split avoids these artifacts, making tar and tsd more reliable for proxy-based estimation.
  • H.3. Validation of Proxy Measurements for Estimating Speedups: Projected speedups closely match directly measured end-to-end speedups across 1k, 2k, and 8k ISL buckets for both EAGLE3 and Vanilla SD.The validation combines Mixed-category step latencies with accepted lengths from High Entropy and Low Entropy domains.
  • I. Detailed Experimental Setup: Experiments use public target and draft checkpoints, three production-oriented engines, and predominantly greedy decoding, with configurations and versions documented in the setup.The engines are TensorRT-LLM, vLLM, and SGLang; GPT-OSS 120B uses medium reasoning effort except in Section 8.2.
  • J. Measuring ALs on the Throughput Split: Low Entropy prompts produce the highest ALs, High Entropy prompts the lowest, and Mixed Entropy prompts intermediate values, with Vanilla SD and Native MTP remaining stable as ISL grows.GPT-OSS 120B with EAGLE3 instead degrades on Low Entropy inputs at longer contexts, crossing below the Mixed Entropy curve, plausibly due to training-distribution bias.
  • K. Vocabulary Pruning Analysis; L. Extended SpecBench vs SPEED Results: Vocabulary pruning causes only a marginal reduction in output-token coverage, while an extended comparison evaluates average AL across SpecBench and SPEED-Bench.The pruning analysis uses greedy completions from the Qualitative Split and the EAGLE3 training corpus to establish an achievable-accuracy upper bound.
  • M. Inference Engine Comparison; N. Long-Context Inaccuracy in Existing EAGLE3 Models: TensorRT-LLM achieves higher throughput than vLLM in the compared CUDA Graphs configuration, largely because it supports a one-model runtime paradigm.The comparison uses GPT-OSS 120B with an EAGLE3 drafter on the 2k Throughput Split across batch sizes 2 to 256; existing EAGLE3 checkpoints also show long-context decay when RoPE settings or training context lengths are mismatched.
Loading 2604.09557v2…