PaperlayerView exact revision on arXiv

Source-linked AI summary

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard

arXiv:2610.38968v1cs.ETcs.AI

TL;DR

Speculative decoding’s best draft length varies with load, but vLLM’s hand-written tables are not measured or corrected. TableTune profiles step times on vLLM’s serving path and updates tables from live acceptance; its tuned tables approach the best fixed-length throughput across drafter settings.

  • Problem

    The best draft length depends on load, but vLLM’s hand-written batch-size tables are not checked or corrected.

    Source: §1 · p. 1

  • Method

    TableTune measures step times on vLLM’s serving path, computes a table, and uses a plugin to update acceptance estimates and rebuild the table from live traffic.

    Source: §1 · p. 1, §4.2 · p. 4

  • Results

    At every load, tuned tables reach at least 94% of the best fixed-length throughput in four of five drafter settings and at least 88% in the fifth.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tables must be measured for each GPU, engine, and model and kept current as traffic changes; TableTune measures them in minutes and repairs them while serving.

    Source: §9 · p. 10

    Profiles can miss serving costs they cannot observe, and evaluation excludes prompts beyond 4,096 tokens, larger mixture-of-experts models, multi-node, and disaggregated serving.

    Source: §7 · p. 10

Abstract

from arXiv · show

Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table. We present TableTune. Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving. We measure TableTune only against other ways to speculate. On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth) without the operator having to know which length that is. Against reimplementations of TurboSpec- and Nightjar-style control on the same engine, its worst-case loss is never higher than theirs, and with the EAGLE-3 drafter it is 3% against 9% and 12%. On a mixture-of-experts model, vLLM's documented table loses up to 18% and the tuned table at most 3%. Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss. In llm-d's optimized-baseline deployment as shipped, two Qwen3-32B replicas on four GPUs, the tuned table is up to 32% faster than a fixed draft length of 3 and up to 25% faster than vLLM's documented table at high load, and the controller raises the documented table's throughput by up to 29%.

1 INTRODUCTION

TableTune replaces hand-written batch-size draft-length tables with measured tables and a controller that updates them from live traffic. Its tuned tables approach the best fixed-length throughput across loads, while controller and llm-d results show gains in other settings.

1 INTRODUCTION

  • Because speculation can slow busy-GPU serving and vLLM’s hand-written tables are not checked or corrected, TableTune measures and rebuilds tables without changing vLLM’s mechanism.The tuner profiles the serving path; the controller is a vLLM plugin that learns from live traffic.Source: §1 · pp. 1–2
  • At every load, tuned tables deliver at least 94% of the best fixed-length throughput in four of five drafter settings and at least 88% in the fifth.The fifth setting is DFlash, where any single fixed length falls to 81% or less at some load.Source: §1 · p. 2
  • With EAGLE-3, TableTune’s worst-case loss is at most 3.3%, versus up to 9.3% for TurboSpec-style control and 11.8% for Nightjar-style bandits.These are reimplementations compared on the same engine and draft lengths.Source: §1 · p. 2
  • Starting from mistuned tables, the controller raises throughput by up to 58% and recovers 96% of a stale table’s 10.8% loss after a DFlash workload switch on H100.For DFlash started from another drafter’s table, the reported increase is up to 23%.Source: §1 · p. 2
  • In llm-d’s shipped optimized-baseline deployment, the tuned table is up to 32% faster than fixed length 3 and 25% faster than vLLM’s documented table at high load.The controller raises the documented table’s throughput by up to 29% in the Qwen3-32B deployment.Source: §1 · p. 2
  • Table-based speculation requires full CUDA graphs per draft length, an off switch, and non-stalling length changes; vLLM’s default V1 runner loses 30–43% on H100 with any table.The reported loss is regardless of host cores.Source: §1 · p. 2

2 BACKGROUND

Speculative decoding trades extra draft work for multiple verified output tokens per target-model pass. Whether it improves throughput depends on batch-dependent step time and accepted guesses, while the serving engine’s graph behavior can affect whether a table helps.

2 BACKGROUND

  • Batch size is the number of requests in a step, and continuous batching makes it follow traffic; the table uses live batch size rather than concurrency.In closed-loop runs, live batch size averages about half to two thirds of concurrency.Source: §2 · p. 2
  • A drafter proposes tokens that the target verifies in one pass, accepting guesses up to the first disagreement and then appending one token of its own.Every kept token is verified, so output matches what the target model would produce alone.Source: §2 · p. 2
  • Goodput is kept tokens per second: the tuner predicts it from expected tokens per step and step time, and measures it as output tokens per second.Speculation raises both the expected-token numerator and the step-time denominator, so its payoff depends on which grows faster.Source: §2 · p. 2
  • Figure 1 reports throughput as a percentage of the best fixed draft length at each load, while vLLM’s table selects a length by batch-size range.vLLM’s default V1 runner replaces full decode graphs with piecewise graphs for table-based drafting; its V2 runner keeps full graphs per draft length.Source: §2 · pp. 2–3, Figure 1 · p. 3

3 MOTIVATION

The best draft length changes with load, and tuned tables also vary across GPUs. This motivates measuring step times separately from acceptance, which changes with traffic.

3 MOTIVATION

  • On H100 with Qwen3-4B and EAGLE-3, length 3 is best at low load while length 1 is best at 128 requests; fixed length 1 loses 15% at one request and length 3 loses 12% at 128.The best length changes with load, so no single fixed length wins across the tested range.Source: §3 · p. 3
  • The tuned tables differ across T4, A10G, and H100, and the H100’s plain step time rises from 4.4 to 6.1 ms between batch sizes 1 and 128.On the T4, plain step time rises from 35.7 to 92.3 ms over that range.Source: §3 · p. 3
  • TableTune separates stable step-time measurements, which depend on hardware, engine, and batch size, from traffic-dependent acceptance, which it tracks online.Acceptance varies across workloads, including EAGLE-3’s tokens per step at draft length 3.Source: §3 · p. 3

4 DESIGN

The tuner profiles vLLM’s serving path and selects draft lengths using measured step times and acceptance. The controller updates acceptance estimates from live traffic and rebuilds the table using those profiles.

4.1 The tuner

  • The tuner times each draft length through vLLM’s real serving path for each batch size, using the configured drafter and graph mode.It estimates step time from the difference between requests for 65 and one output token, divided by 64, and uses the median of three repeats.Source: §4.1 · p. 3
  • Acceptance profiling runs the longest draft length on 160 sample prompts and uses vLLM’s published counters to estimate acceptance at each position.A single run supplies estimates for shorter lengths because each is a prefix of the longest draft.Source: §4.1 · pp. 3–4
  • For each batch size, the tuner selects the length with highest predicted goodput, suppressing near-tie edges and merging ranges narrower than 25% of their first batch size.For n-gram drafting, it adjusts acceptance by proposal rate but charges the full step time.Source: §4.1 · p. 4

4.2 The controller

  • The controller is a vLLM plugin that learns acceptance from live traffic and turns it into a table using the tuner’s step times.Figure 2 depicts the tuner writing a vLLM-format table and the serving controller rebuilding it from acceptance counters.Source: Figure 2 · p. 3, §4.2 · p. 4
  • From vLLM’s per-request draft and acceptance counts, the controller estimates the probability that guesses through each position are all accepted.It compares the number of requests accepting position i with the number drafted through position i.Source: §4.2 · p. 4
  • For each batch size, acceptance determines the value of another guess while its added step time is the cost; the controller updates acceptance, not step times.Higher acceptance keeps longer drafts useful to larger batches, while lower acceptance moves table edges toward smaller batches and shorter drafts.Source: §4.2 · p. 4
  • The controller blends live counts with tuner priors, fades old counts, and can use exploration, prior decay, and hysteresis to respond to traffic changes.By default it uses only lengths in its starting table, whose profiles and captured graph sizes are available.Source: §4.2 · p. 4

5 IMPLEMENTATION

TableTune is a compact Python implementation with a one-command tuner and a controller deployable through vLLM configuration. It runs on both vLLM V1 and V2 runners and is available with its tables and evaluation materials.

5 IMPLEMENTATION

  • TableTune is about 1,880 lines of Python and includes 104 test functions.Source: §5 · p. 4
  • The tuner runs with one command: dsd-tuner run.Source: §5 · p. 4
  • The controller is an installed plugin enabled by environment variables, and deployment into serving stacks such as llm-d requires configuration alone, with no engine changes.Source: §5 · p. 4
  • The tuner runs unchanged on vLLM V1 and V2, and the project provides code, per-GPU tables, benchmark scripts, raw results, and analysis scripts.Source: §5 · p. 4

6 EVALUATION

Across the tested setups, measured tables generally stay close to the best fixed draft length, while the controller repairs mistuned tables and adapts after workload changes. End-to-end gains depend on the engine: the V2 runner’s full CUDA graphs avoid the costs and stalls observed with some tables on V1.

6.1 Setup

  • The evaluation spans V1 and V2 runners, H100, T4, and A10G GPUs, multiple drafter and target models, and code, chat, long-prompt, jumping-load, and open-loop workloads.Results are generally means of three runs, and gaps below about 2% are treated as ties.Source: §6.1 · pp. 4–5

6.2 Q1: measured tables against fixed lengths

  • On H100 V2, tuned tables stay within 3.0–5.7% of the best fixed length for four settings and within 11.6% for DFlash, whose best single fixed length loses 19.1%.On the mixture-of-experts target, the tuned table stays within 2.9% of the best fixed length, while vLLM’s documented table loses 17.5% at 96 requests and 12.2% at 128.Source: §6.2 · p. 5, Table 1 · p. 6

6.3 Q2: against prior adaptive control

  • With EAGLE-3, tuned tables’ worst loss is 3.3–3.7%, versus 9.3% for TurboSpec-style and 11.8% for Nightjar-style control.TurboSpec-style misses a jump in measured step time; Nightjar-style continues exploring separately at each batch size.Source: §6.3 · p. 6, Figure 3 · p. 7

6.4 Q3: the controller

  • From mistuned tables, the controller raises throughput by up to 58% on H100 V2 and reaches 96–103% of the tuned table’s throughput across tested GPUs.After a DFlash code-to-math workload switch, it recovers 96% of the stale table’s loss within the first minute.Source: §6.4 · p. 7, Figure 4 · p. 8, Table 2 · p. 8

6.5 Q4: end to end in llm-d

  • At 256–1024 open requests, the tuned table is 20–32% faster than fixed length 3 and 7–25% faster than vLLM’s documented table in llm-d’s Qwen3-32B deployment.The controller started from the documented table raises throughput by 9–29% at those loads; the figure compares output tokens/s against fixed length 3.Source: §6.5 · pp. 8–9, Figure 5 · p. 9

6.6 Q5: what the engine must provide

  • On V1, tables switch decoding to piecewise CUDA graphs, which can impose substantial launch overhead; V2’s full graphs per draft length remove the reported loss.Some length-2-to-1 tables stalled the V2 runner under load, while 0/1/3 tables and tables without that step did not.Source: §6.6 · p. 9

6.7 Q6: cost and reliability of measuring

  • Profiling takes 8 minutes on H100 and is repeated per GPU, engine, model, and drafter; step-time modeling can miss important behavior, and DFlash profiles can understate live step times.The authors report that a merge-tolerance option improves DFlash by 6.5% at 64 requests, while DFlash still trails fixed length 3 by 10% at 128 requests.Source: §6.7 · pp. 9–10

7 LIMITATIONS

The reported results depend on engine support for full CUDA graphs per draft length and stall-free table switching. The evaluation is limited to specified model, hardware, prompt, and traffic settings, and profiles may miss serving costs or become invalid.

Engines

  • The tuned-table results use one vLLM nightly whose V2 runner stalls when a table switches from length 2 to 1; affected deployments use a potentially worse 0/1/3 table.This includes both llm-d deployments.Source: §7 · p. 10

Accuracy of the table

  • Table quality is bounded by profile accuracy: DFlash leaves a 10% gap at 128 requests, while an invalid draft-model profile and unseen T4 KV-cache pressure also limit predictions.The controller does not correct cost errors, and the T4 high-load results are affected by KV-cache pressure and preemption.Source: §7 · p. 10

Scope

  • The study covers dense 4B, 8B, and 32B targets and one mixture-of-experts target on single-node H100s at tensor parallelism of at most 2; several larger or different serving settings remain untested.Missing settings include prompts beyond 4,096 tokens, mixed and recorded traffic, larger mixture-of-experts models, multi-node serving, and disaggregated serving.Source: §7 · p. 10

8 RELATED WORK

Related work adapts speculation at different levels, including per-request or per-token methods and load-aware choices. TableTune focuses on vLLM’s production draft-length table, measuring its serving-path costs and keeping it current with a plugin.

Drafters and per-request adaptation

Load-aware speculation in serving

  • TableTune measures a grid for vLLM’s production table and keeps it current with a plugin, unlike TurboSpec’s fitted step-time model and per-step choices or Nightjar’s online bandit.Nightjar also switches speculation truly off and moves the drafter to CPU memory.Source: §8 · p. 10
  • Other load-aware approaches batch speculation, select verified draft tokens across a batch, size token trees for hardware, or adapt speculation to latency targets.The passage names BASS (Qian et al., 2024), TETRIS (Wu et al., 2025), Sequoia, SpecServe/AdaSpec (Huang et al., 2025b), and AdaServe (Li et al., 2026).Source: §8 · p. 10

9 CONCLUSION

TableTune measures a draft-length table for each GPU, engine, and model, then uses a controller to repair it as traffic changes. Its scope and results are bounded by table transferability and engine requirements.

9 CONCLUSION

  • On an H100 with full CUDA graphs, TableTune’s tables stay within 3–6% of the best fixed length for most drafters, and the controller recovers up to 58% from mistuned tables.In llm-d’s optimized-baseline guide as shipped, the tuned table is up to 32% faster than fixed length and 25% faster than vLLM’s documented table.Source: §9 · p. 10
  • A dense-model table loses up to 18% on a mixture-of-experts model; table-based speculation also needs full graphs per draft length, stall-free switches, and a real off switch.These are engine and transferability boundaries for the reported approach.Source: §9 · p. 10

AI USAGE DISCLOSURE

The authors report using Claude for preliminary literature summarization, related-work outlining, graph-rendering boilerplate, and submission-format checks. They say they reviewed generated scripts, conducted the evaluations, and drafted the final text themselves.

AI USAGE DISCLOSURE

  • Claude assisted with preliminary literature summarization and the initial related-work outline.The authors state that they independently drafted the final text.Source: AI USAGE DISCLOSURE · p. 13
  • Claude generated boilerplate Python scripts for graph rendering, which the authors say they manually scrutinized.Source: AI USAGE DISCLOSURE · p. 13
  • Claude Code was used for submission-formatting checks, while the authors report conducting the evaluations themselves.Source: AI USAGE DISCLOSURE · p. 13
Loading 2610.38968v1…