PaperlayerView exact revision on arXiv

Source-linked AI summary

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein

arXiv:2610.40287v1cs.DCcs.AIcs.LGcs.PF

TL;DR

The paper examines how speculative decoding’s energy cost varies under facility-power and latency constraints, where batching shares weight reads but does not make an individual user faster. It measures speculative and matched non-speculative serving configurations, finding that speculation’s energy value depends on batch and drafter. Its conclusion is that speculation is most valuable when latency rules out larger batches, while energy-bound services should raise batch size first and then check whether a drafter still saves energy.

  • Problem

    The energy cost of speculation is difficult to pin down, while serving must meet both facility-power and user-latency objectives.

    Source: §1 · pp. 1–2

  • Method

    The paper measures five drafting schemes across 11 models and compares speculative runs with matched non-speculative baselines under production serving conditions.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Speculation is most valuable for energy when latency rules out a larger batch, and least valuable when batch size can still grow.

    Source: §12 · p. 17

  • Takeaways & Limitations

    Where energy per token is binding, raise batch size first and then check whether a drafter still reduces energy.

    Source: §12 · p. 17

    Measurements use mean time between tokens, GPU-board energy counters, short prompts, and batches up to 64.

    Source: §10 · p. 16

Abstract

from arXiv · show

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

1 Introduction

The paper asks how speculative decoding’s energy cost compares with batching under facility-power and latency constraints. It measures speculation against matched non-speculative baselines and finds that speculation can meet tight time-between-token objectives that batching alone misses.

1 Introduction

  • Facility power makes energy per token consequential because tokens per megawatt is its inverse.The introduction notes that industry comparisons already include systems with and without speculative decoding.Source: §1 · p. 1
  • The operator’s desired configuration must meet both energy and latency objectives; the time between tokens is set by decode alone.The first-token objective has other levers, including prefix caching and separate prefill machines.Source: §1 · pp. 1–2
  • Batching amortizes weight reads and lowers energy per token, but increases individual users’ time between tokens by 9–34% on dense models and 2.5–4.0× on Mixture-of-Experts models from batch 1 to 32.The paper motivates speculation as a way to produce multiple accepted tokens per weight read while addressing user latency.Source: §1 · p. 2
  • 1.3 to 4.3× shorter mean time between tokens at single stream; speculation brings every model missing a 25 ms or 10 ms objective under it.Without speculation, gemma-4-31B-it misses 25 ms per token at every tested batch, and 4 of 11 models miss 10 ms.Source: §1 · p. 2
  • The study pairs five published drafting schemes across 11 models and four attention designs with same-model, same-batch non-speculative baselines.It also uses hardware counters and draft-removal measurements to examine where verification crosses the balance point and energy savings end.Source: §1 · p. 2

2 Background: Decode, Speculation, and Draft Cost

Decode repeatedly reads model weights, so its energy and inter-token time track memory traffic. Speculation reuses a weight read for verification, but its benefit depends on accepted tokens relative to the cost of drafting and verifying all proposed tokens.

2 Background: Decode, Speculation, and Draft Cost

  • Decode operates more than an order of magnitude below the GPU balance point: about 10 operations per byte versus 316 on H100 and 163 on H200.The passage links decode energy to bytes moved and time spent moving them.Source: §2 · p. 2
  • Only accepted tokens divide the weight read; rejected drafts still incur drafting and verification work.At serving batch, verification work on rejected tokens displaces useful work for other requests.Source: §2 · p. 3
  • N-gram lookup uses no network, draft heads predict one token per sequential pass, and block drafters propose a whole block in one pass.Block sizes described are 16 tokens for z-lab DFlash and 7 for DeepSpec DFlash and DSpark.Source: §2 · p. 3
  • Every scheme verifies the last accepted token along with drafted tokens, so the verified count is one more than the draft proposal for heads and equals block size for block drafters.DFlash-16 verifies 16 tokens per request and step.Source: §2 · p. 3

3 Measurement

The measurement compares speculative decoding with matched non-speculative runs under production serving conditions. It also measures hardware behavior and accounts for CUDA-graph capture, which can materially change reported energy savings.

3 Measurement

  • The study spans 11 models across four attention families, with batch sweeps on H100 or H200 GPUs and 512 generated tokens per request.Runs use batches drawn from 96 prompts per task; selected models also have conversation, long-document, English-news, and Romansh-news measurements.Source: §3 · p. 3
  • Every speculative run is compared with the same model, batch, prompt pool, generation length, GPU, and engine settings without speculation.Energy saving is the fraction by which speculative energy per token falls below that matched baseline.Source: §3 · p. 4
  • The GPU energy counter is read at 10 Hz and covers the GPU board, not host processor, memory, network, or cooling.This scope favors n-gram drafting because its matching runs on the host.Source: §3 · p. 4
  • Table 1 identifies each drafter’s verified-token count, training provenance, GPU, and Mixture-of-Experts status.The number of verified tokens differs across schemes.Source: §3 · p. 4, Table 1 · p. 5
  • CUDA-graph capture changes reported savings: the credit is under 10 percentage points on dense models and 19–39 points at single stream on Mixture-of-Experts models.Capture can change which scheme saves the most energy, so the paper reports savings, batch sweeps, latency, and energy splits with capture enabled.Source: §3 · p. 5

4 What Speculation Saves, and Where the Saving Ends

Speculation can reduce energy per token, but its savings depend on batch size, model, drafter, and served text. As batching already amortizes weight reads, savings often shrink with larger batches; acceptance also varies across workloads and training regimes.

4 What Speculation Saves, and Where the Saving Ends

  • At single stream, every model has a scheme that saves energy, with the largest per-model saving having a median of 54% of decode energy.N-gram drafting is the exception on Mixture-of-Experts models, where it costs energy.Source: §4 · p. 5, Table 2 · p. 6
  • Savings decay as batch grows because batching has already amortized the weight read; at long contexts, speculation’s speedup can instead grow with batch.The reported decay holds for the short contexts served in this study.Source: §4 · p. 5
  • At batch 64, the best-saving scheme still saves 15 to 52% on every model that could run there, although some schemes cost energy.N-gram drafting costs energy on every reported model except Qwen3-Coder-30B-A3B-Instruct.Source: §4 · p. 6, Table 2 · p. 6

4.1 Does the Served Text Change the Answer?

  • Every scheme on every model accepts fewer tokens per verification on conversation than on code, and the largest per-model saving at batch 64 is a median 11 percentage points lower on conversation.The DFlash drafter trained for Qwen3-Coder-30B-A3B-Instruct accepts 2.0 to 3.0 tokens per verification on conversation versus 5.9 on code; at batch 64, it saves 51% of decode energy on code but costs energy on conversation.Source: §4.1 · p. 8
  • The energy-saving scheme can change with served text because every drafted token is verified, even when it is not accepted.On gemma-4-12B-it and gemma-4-31B-it, DFlash verifies 16 tokens per step and the assistant 6; on conversation, both accept about the same number, but the assistant saves most at every batch.Source: §4.1 · p. 8
  • For drafters serving Romansh, accepted length depends on when they were trained: co-trained heads retain 81 to 84% of English accepted length, while post-hoc drafters retain 70% on two models.On those two Mixture-of-Experts models, every drafter costs energy in Romansh; on gemma-4-12B-it, the vendor-trained assistant retains only 66 to 69%, like the third-party DFlash drafter.Source: §4.1 · pp. 8–9

5 How Much Batching Does It Replace?

Batching can reduce energy per token, but it does not provide the per-user speed that speculation does. How much speculation saves depends on the batch, and latency-constrained comparisons favor speculation on most tested models.

5 How Much Batching Does It Replace?

  • A speculative run at batch 1 used as little energy per token as a baseline batch of 2 to 8, depending on the model; at speculative batch 32, five models still beat baseline batch 64.The baseline batch sizes tested were powers of two from 1 to 64.Source: §5 · p. 9
  • As baseline batch size grows, energy per token falls by more than an order of magnitude; speculation raises single-stream per-user speed by 1.3 to 4.3×.On dense models, the two levers are close to perpendicular; on Mixture-of-Experts models, added requests slow every user.Source: §5 · p. 9
  • For services without a latency constraint, batching is the cheaper energy lever; speculation supplies per-user speed when latency is constrained and can recover speed lost to batching on Mixture-of-Experts models.The energy benefit speculation adds decays with batch.Source: §5 · p. 10
  • 18–69% less energy per token with speculation on 10 of 11 models was achieved at a 25 ms target; gemma-4-31B-it met it only with speculation.The comparison selects the lowest-energy configuration meeting the target within tested batch sizes; timing is a generation mean, not a per-token percentile.Source: §5 · p. 10
  • On the tested H200 configuration, a 1500 MHz lock lowered baseline energy per token by 4 to 11% but reduced throughput 3 to 7%; with speculation at batch 16 or more, it saved no energy.The experiment used Qwen3-8B on the code pool; the 1200 MHz lock lowered energy for every configuration but also throughput.Source: §5 · p. 10

6 Which Drafter, and Does It Change With Batch?

Which drafter saves the most energy changes with batch: block drafters often lead at single stream, while draft heads can lead at larger batches. The preferred draft length also falls with batch, as every proposed token must be verified.

6 Which Drafter, and Does It Change With Batch?

  • At single stream, block drafters led on 7 of 9 models with a block drafter, but draft heads saved the most at the largest batch on all five models that had both.The head took over between batch 8 and 32 on Qwen3.6-27B, gemma-4-12B-it, and gemma-4-31B-it; DFlash led again on gemma-4-31B-it at batch 16.Source: §6 · pp. 10–11
  • Block drafters verify 16 tokens per request and step, versus 4 to 6 for the draft heads; at larger batches, that extra verification compute can outweigh the block drafter’s added accepted tokens.The draft itself does not grow with batch, but rejected tokens add compute when serving batches are large.Source: §6 · p. 11
  • After adding recurrent-state advancement to the n-gram path, hybrid-model n-gram text departed from baseline no earlier than MTP text did.Without that advancement, output became incoherent within the first few words.Source: §6 · p. 11
  • On Qwen3.6-27B, n-gram drafting saved 12% at batch 1, broke even at batch 8, and cost energy from batch 16; on Qwen3.6-35B-A3B it cost energy at every batch.Source: §6 · p. 11
  • The DeepSpec DFlash drafter, which proposes 6 tokens, saved more energy at every measured batch than the z-lab version proposing 15 tokens on Qwen3-4B and Qwen3-8B.The z-lab version requires verification of 9 additional tokens per request and step.Source: §6 · p. 11
  • On Qwen3.6-27B, DFlash-16 failed to start at batch 16 on H100; recurrent-state rollback memory grows with drafted tokens times batch.At that batch the rollback buffer alone was about 10 times the drafter’s weights; DFlash-16 also failed at the largest batches on both hybrids on H200.Source: §6 · p. 12

6.1 How Many Tokens Should a Draft Hold?

  • On gemma-4-31B-it, 4 drafted tokens saved the most from batch 4 onward, while every tested draft length cost energy at serving batch; at batch 64, 4 tokens was least costly on all three swept models.The sweep tested 1 to 32 proposed tokens on one dense model and two Mixture-of-Experts models; the latter two already lost energy at single stream.Source: §6.1 · p. 12

7 Which Metric Predicts the Energy Ranking?

Acceptance predicts energy savings when batch is fixed, but does not by itself determine which drafter uses less energy across batches. Verification work helps explain the gap: every proposed token is checked, whether accepted or not.

7 Which Metric Predicts the Energy Ranking?

  • Block drafters accepted more tokens than draft heads in every matched comparison, yet heads used less energy at the largest batch on all five models with both.Acceptance alone misses the extra verification work performed by block drafters.Source: §7 · p. 12
  • At every fixed batch, energy saving rose with accepted tokens per verification, but the acceptance needed to break even rose with batch.Across 287 of 332 same-model, same-baseline scheme pairs, the higher-acceptance scheme used less energy; most exceptions verified more tokens per request.Source: §7 · p. 12
  • Figure 5a compares energy saving with accepted tokens per verification across models, schemes, and batches; Figure 5b splits decode energy between drafting and verification at batches 1 and 64.Source: §7 · p. 12, Figure 5 · p. 13

8 Where the Energy Goes

On dense models, verification dominates decode energy, while drafting is the smaller cost; the draft share stays close to constant or falls as batch size increases. The measurement split is not applicable to MoE models, and host phase markers can misattribute energy.

8 Where the Energy Goes

  • Verification accounts for most decode energy on every dense model and scheme, both at single stream and batch 64.The split is measured with capture enabled by repeating the run without draft passes while keeping verification steps and contexts the same.Source: §8 · p. 13
  • A block drafter takes at most 23% of decode energy, so even halving its cost would lower total decode energy by at most 12%.No drafter takes more than 45%; EAGLE3 on Qwen3 has the largest share because it runs seven passes per draft.Source: §8 · p. 13
  • The drafter’s energy share stays close to constant or falls from single stream to batch 64, as verification grows at least as fast as drafting.The share is small for every drafter on gemma-4-31B-it.Source: §8 · pp. 13–14
  • Host phase markers require capture to be disabled and misstate the Gemma split; at batch 64 they assign the gemma-4-12B-it assistant a larger energy share than the removal-based measure.With capture enabled, the authors use draft removal to measure the split.Source: §8 · p. 14
  • On MoE models, replacing drafted tokens with random or repeated tokens changes verify-only energy by up to 52%, so the paper does not report a draft/verification split for them.Verification cost depends on which tokens are drafted because each token routes to selected experts.Source: §8 · p. 14

9 Does Speculation Reach the Compute-Bound Regime?

Batching, rather than the number of verified tokens alone, moves decode toward the compute-bound regime. When a step crosses the GPU balance point, speculative energy savings end and can turn into losses.

9 Does Speculation Reach the Compute-Bound Regime?

  • At single stream, every scheme on every model stays within a few percent of its baseline arithmetic intensity; batch combined with verification moves the operating point substantially.The same decode step has close to the same arithmetic intensity on both GPUs.Source: §9 · p. 14
  • With 260-token contexts, DFlash-16 on Qwen3-4B reaches the H100 balance point at batch 32 and the H200 point at batch 16.DSpark-7 crosses the H200 point at batch 32 but stays below H100’s; longer contexts lower intensity at a given batch.Source: §9 · p. 15
  • Below the balance point, the n-gram drafter and gemma-4-12B-it assistant run below the memory roof from batch 1 because attention over drafted tokens uses slow kernels.DSpark runs close to the memory roof.Source: §9 · p. 15
  • DFlash-16’s energy saving turns negative around the batch where its step crosses the GPU balance point: between batches 16 and 32 on H100 and at 16 on H200.DSpark-7 still saves energy at batch 64 on H100, where it remains below the balance point, but costs energy on H200, where it passes it.Source: §9 · p. 15
  • Past the balance point, verified-token arithmetic is no longer hidden by weight reads, while DFlash-16 accepts fewer than 3 of its 16 proposed tokens per request.A GPU with more memory bandwidth per unit of arithmetic reaches this regime at a smaller batch; the balance point indicates where savings change sign, not their size.Source: §9 · p. 15

10 Measurement Pitfalls

Plausible energy measurements can still reflect a workload or measurement condition different from the one intended. The paper identifies ways to detect these mismatches and bounds the scope of its measurements.

10 Measurement Pitfalls

  • A requested batch can differ from the concurrency actually run, and mismatched templates, prefill in the profiler window, host phase markers, dropped result columns, or unnamed GPUs can distort comparisons.For example, Qwen3.6-27B at requested batch 16 ran 12 requests on H100 but all 16 on H200, changing the measured MTP-head saving.Source: §10 · p. 15
  • Runs extending beyond a model’s answer can inflate acceptance because drafting repeated text after the answer ends is cheap.With end-of-sequence honored, code prompts had median answers longer than 700 tokens, keeping 512-token runs within the answer.Source: §10 · pp. 15–16
  • A normal-looking result row can conceal broken generated text: n-gram produces broken text on state-space hybrids without the engine fix.Taken at face value, those runs would make n-gram appear to match or beat the co-trained head on Qwen3.6-27B up to batch 8.Source: §10 · p. 16
  • Measurements use mean time between tokens, GPU-board energy, short prompts, and batches capped at 64; the roofline analysis covers Qwen3-4B.The counter excludes host processor, host memory, network, and cooling; the evaluation also varies drafted-token counts only for n-gram.Source: §10 · p. 16

11 Related Work

Prior work studies speculative decoding through accepted length, speed, throughput, or energy, but leaves gaps in cross-scheme energy comparisons and draft/verification decomposition. This paper extends measurements across attention families, schemes, and model types.

Energy measurement

  • Prior energy work measured decode across attention mechanisms without speculation [9] or surveyed speculative strategies on older dense models, but none separated draft from verification energy.That split determines whether a drafter pays for itself.Source: §11 · p. 16

Positioning

  • Earlier studies measured speculative energy on two dense grouped-query models with one fixed-length drafter [15], while this work spans four attention families and five schemes and decomposes energy.Time and throughput comparisons exist, but do not report joules [25] [1].Source: §11 · p. 16
  • The paper measures speculative energy on state-space hybrid models, while prior work on MoE models established larger gains from speculation and linked them to launch-bound decode.This work reports that launch-bound decode makes MoE speculative energy especially sensitive to graph capture.Source: §11 · p. 16

Language and task

  • Prior language studies found separate small drafters less effective in low-resource languages and distillation to generalize poorly [30] [35], but did not compare production drafters on one model or measure energy.Broad-data pretraining was reported to help most on out-of-domain and multilingual text [5].Source: §11 · p. 16

12 Conclusion

Speculation can be needed to meet tight inter-token latency targets, but its energy value depends on whether batching can still grow. Its savings also depend on the GPU and serving batch, so acceptance alone does not reveal which drafter will save the most energy.

12 Conclusion

  • Under a tight latency objective, no batch size may meet the target without speculation, making speculation a latency choice rather than an energy choice.Its energy value is greatest when the latency target rules out a larger batch, and least when batch size can still grow.Source: §12 · p. 17
  • When energy per token binds, raise batch size first, then check whether speculation still reduces energy.For services prioritizing inter-token time, tune the drafter and drafted-token count to the batch.Source: §12 · p. 17
  • The same drafter at the same batch can save energy on one GPU and cost energy on another, while single-stream and serving-batch energy rankings can differ.Verification past the GPU-specific balance point loses savings; acceptance changes little with batch and does not reveal the shift.Source: §12 · p. 17

Related papers

Loading 2610.40287v1…