PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli

arXiv:2610.38981v1cs.LGcs.AI

TL;DR

Prior work predicts limited low-batch speculative-decoding gains for sparse MoEs using an activated-expert-bytes view of verification cost. This paper evaluates modern sparse MoEs and models routing overlap; many retain high speedup near batch size 1, and the framework predicts unique expert counts within 4% error.

  • Problem

    Prior work models sparse-MoE verification cost through activated-expert bytes and predicts limited speculative-decoding gains at low batch sizes.

    Source: §1 · p. 1

  • Method

    A maximum-entropy Iterative Proportional Fitting framework uses routing-overlap statistics to predict unique expert counts and generate synthetic routing.

    Source: §1 · p. 2

  • Results

    Across 10 sparse MoE models, many remain efficient at low batch sizes and peak near batch size 1; the framework predicts unique expert counts within 4% error.

    Source: §1 · p. 2, §10 · p. 10

  • Takeaways & Limitations

    The framework enables train-free screening of target and draft architecture tradeoffs, including sparser targets and MoE-versus-dense drafts, before full training.

    Source: §1 · p. 2

    Synthetic routing may leave residual curve gaps because pairwise overlap constraints and position-independent per-expert marginals do not capture higher-order routing statistics.

    Source: §8 · p. 9

Abstract

from arXiv · show

Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.

1 Introduction

The paper revisits whether speculative decoding remains effective at low batch sizes for modern sparse MoEs, where activated-expert bytes alone predict limited gains. Across 10 models, it finds low-batch efficiency and identifies routing overlap and GPU execution dynamics as additional factors in verification cost.

1 Introduction

  • Prior work treats speculative-decoding verification cost largely through activated-expert bytes, predicting limited low-batch gains for sparse MoEs.Speculative decoding proposes K tokens and verifies them with the next-token continuation in one target-model pass.Source: §1 · p. 1
  • Across 10 sparse MoE models, many remain unusually efficient at low batch sizes and peak near batch size 1, contrary to the predicted concave speedup curve.The models span 30B–1T parameters and 8–512 experts.Source: §1 · p. 2
  • Adjacent positions in fine-grained MoEs share 20–45% of activated experts versus 4–12% under independence, reducing unique experts by 21–34%.This overlap accounts for most overlap beyond the uniform baseline in high-sparsity models.Source: §1 · p. 2
  • Verification cost also reflects GPU execution dynamics: on H100, Qwen3-30B FP8 MoE kernels use more peak HBM bandwidth than attention, and L2 serves repeated co-routed expert reads.At batch size 1, MoE layers are 33% of forward-pass time despite 67% of weights; their L2 hit rate rises from 5.72% in autoregressive decoding to 27.79% under verification.Source: §1 · p. 2
  • A maximum-entropy IPF framework predicts unique expert counts from empirical pairwise routing overlap within 4% error and reproduces observed speedup curves under synthetic routing.It also enables train-free screening of target and draft architecture tradeoffs before full training.Source: §1 · p. 2

2 Related Work

Prior work covers speculative decoding proposal strategies and expert locality, while MoESD models sparse-MoE verification through activated-expert bytes. This paper focuses on how temporal locality across draft tokens changes verification cost.

2 Related Work

  • Prior work links expert co-activation to semantic and positional similarity, and ExpertFlow exploits that locality for memory-constrained deployment [40] [3].Source: §2 · p. 3
  • Modern speculative-decoding methods differ mainly in proposal construction and verification, using draft-model or seq2seq variants [31] [5] [53], retrieval- or tree-based proposals [21] [49] [50], extra decoding heads and multi-token prediction [4] [2] [19] [11] [55], and higher-capacity or parallel drafting variants [34] [35] [36] [18] [25] [22].Source: §2 · p. 3
  • MoESD models verification through activated-expert bytes and predicts concave speedup curves with limited low-batch gains; this work examines modern sparse MoEs and temporal routing overlap [23].The low-batch regime is relevant to longer-context, long-tailed RL rollout, and agentic workloads.Source: §2 · p. 3

3 Background and Experimental Setup

Speculative decoding drafts K tokens and verifies them with the next-token continuation in a single target-model pass. The experiments compare target-side verification efficiency across models and configurations.

3 Background and Experimental Setup

  • Speculative decoding helps time per output token only when savings from fewer target-model calls exceed verification overhead, requiring high acceptance and cheap verification.Source: §3 · p. 3
  • Target efficiency compares a single-token autoregressive step with verification over K+1 positions, isolating target-side verification cost from acceptance length.Higher target efficiency means the verification pass is closer in cost to a standard autoregressive step.Source: §3 · p. 3
  • The 10 models span sparsity ratios of 0.020–0.25 and per-expert FP8 weights of 4.7–176 MB; experiments use K=3, vLLM v1, and MT-Bench.Most experiments use NVIDIA H100 SXM GPUs; Kimi-K2.5 and DeepSeek-V3 use B200 GPUs.Source: §3 · p. 3
  • Figure 1 plots TPOT speedup against batch size for four representative MoE models.Source: Figure 1 · p. 4

4 Speedup Curve Regimes

Sparse MoE speedup curves vary with model and configuration: some are concave, while others peak at or near batch size 1. Target efficiency at batch size 1 generally tracks this curve shape, with a noted exception.

4 Speedup Curve Regimes

  • Less sparse models with larger experts show concave speedup curves, while sparser models with smaller experts often peak at batch size 1.Qwen2-57B and Mixtral-8x7B exemplify the former; Qwen3-235B and Kimi-K2.5 exemplify the latter.Source: §4 · p. 4
  • Higher target efficiency at batch size 1 generally accompanies non-concave curves, while lower efficiency leaves more room for a classic concave rise.Qwen3.5-35B is an exception: it is non-concave despite lower target efficiency than GLM-4.7-Flash.Source: §4 · p. 4
  • Table 1 compares activated-expert-bytes-predicted and measured target efficiency at batch size 1 using per-GPU weights.Source: Table 1 · p. 5

5 Limits of the Activated-Expert-Bytes View

The activated-expert-bytes model estimates verification cost from dense and expert weight bytes, but it systematically underestimates target efficiency, even with empirical unique-expert counts.

5 Limits of the Activated-Expert-Bytes View

  • The estimate assumes GPU time is proportional to dense and activated expert weight bytes, using the empirical unique expert count during SD verification.The estimate is Target efficiency ≈ (C_dense + k · C_expert) / (C_dense + unique_SD · C_expert).Source: §5 · p. 5
  • Target-efficiency gaps range from +0.06 for Mixtral with large experts to +0.37 for Qwen3-235B with small experts.The model underestimates target efficiency even when it uses empirical unique-expert counts from actual MT-Bench routing.Source: §5 · p. 5
  • Weight bytes alone do not explain GPU execution time at batch size 1.The estimate omits model behavior and hardware interactions that affect execution time.Source: §5 · p. 5

6 Temporal Overlap in MoE Expert Routing

Empirical routing overlap across verification positions reduces the number of unique experts activated compared with independent-routing estimates. A maximum-entropy IPF framework predicts unique-expert counts from overlap statistics with small error.

6 Temporal Overlap in MoE Expert Routing

  • In six of eight models, temporal overlap accounts for 69–92% of total overlap; empirical step-1 overlap exceeds both uniform and independence baselines.Across eight models, mean empirical overlap falls from 35.1% at step 1 to 26.4% at step 3, remaining 3.3×, 2.7×, and 2.5× the 10.5% mean independence baseline, respectively.Source: §6 · p. 5

6.1 Applicability to draft tokens.

  • Rejected draft tokens share a similar or slightly higher fraction of top-k experts with the preceding token, making accepted-token overlap a conservative proxy for aggregate verification cost.For Qwen3-235B at step 1, first-rejection overlap is 0.488 versus 0.447 for accepted tokens.Source: §6.1 · pp. 5–6

6.2 Sub-Linear Verification Cost

  • For Qwen3-30B, verifying four tokens loads 2.4×k unique experts per layer versus 3.6×k under uniform random routing; temporal overlap reduces counts a further 21–34% beyond independence in six high-sparsity models.Expert popularity accounts for at most 8% savings, typically 2–5%; overlap patterns also vary little across 13 Spec-Bench categories and seven Aya multilingual splits for Qwen3-30B.Source: §6.2 · p. 6

6.3 Predicting Unique Experts and Validating the Framework

  • The maximum-entropy IPF framework predicts unique-expert counts from pairwise overlap statistics; skewed IPF uses empirical per-layer expert probabilities and predicts counts within 4% for every measured model.The maximum skewed-IPF error is +3.7%; uniform IPF, which assumes uniform expert probabilities, is within 5%.Source: §6.3 · pp. 6–7

7 Interaction of GPU execution of sparse MoE on Speculative Decoding

At batch size 1 on H100, profiling shows that expert size shapes how MoE weight bytes translate into runtime. Small experts exhibit bandwidth and L2 effects that make the activated-expert-bytes estimate less accurate than for large experts.

7 Interaction of GPU execution of sparse MoE on Speculative Decoding

  • At batch size 1 on H100, profiling identifies bandwidth-utilization and L2-cache differences between attention and MoE kernels for Qwen3-30B, while these differences are largely absent for Mixtral.The profiling uses Nsight Systems for kernel timing and Nsight Compute for hardware counters.Source: §7 · p. 7, Table 5 · p. 7
  • For Qwen3-30B, increasing tensor parallelism from 1 to 2 raises target efficiency from 0.630 to 0.697 at BF16, while FP8 at TP1 raises it to 0.818; FP8 at TP2 reaches 0.860.Smaller per-GPU expert shards and quantization allow more of the co-routed working set to remain in L2; Mixtral experts are too large to fit in L2.Source: §7 · p. 8
  • For Qwen3-30B, MoE GEMMs have roughly 2× the bandwidth of attention GEMMs, and MoE runtime is 33% versus 67% of weights; for Mixtral, MoE and attention bandwidths are more similar.Verifying K+1 tokens barely increases attention cost (S=1.02), as KV-cache reads are reused across consecutive draft tokens.Source: Table 5 · p. 7, §7 · p. 8
  • Smaller experts leave bandwidth underutilized and fit co-routed repeated reads in L2, so execution time is below the weight-byte estimate; large experts exceed L2 capacity and keep execution time closer to proportional to weight bytes.Qwen3-235B (FP8, TP4) reproduces the small-expert mechanisms on H100, with a +0.38 weight-model gap.Source: §7 · p. 8

8 End-to-end validation via synthetic routing.

Synthetic-routing experiments test whether routing-overlap statistics and real hardware execution can reproduce SD target-efficiency curves. Per-layer IPF routing tracks real-model curves more closely than uniform random routing, especially at small batch sizes.

8 End-to-end validation via synthetic routing.

  • Synthetic-routing experiments replace learned routing with IPF-generated routing while keeping other inference settings fixed and running real kernels on target hardware.The setup uses dummy weights and captures hardware- and kernel-specific execution effects without modeling them explicitly.Source: §8 · p. 8
  • Per-layer IPF synthetic routing closely tracks real-model curves at small batch sizes and stays closer than uniform random routing across batch sizes; uniform random routing consistently underestimates target efficiency.Uniform IPF still captures the qualitative batch-size trends, though skewed IPF is more accurate.Source: §8 · p. 8, Figure 2 · p. 9
  • On MT-Bench with H100, Figure 3 compares SD speedup against batch size for K=3, 5, and 7; each model keeps its curve shape across K, while larger K mostly shifts curves down.Qwen3-235B is the exception: its stronger drafter keeps K=5 competitive.Source: Figure 3 · p. 9
  • Residual gaps are expected because synthetic routing does not reproduce the full real-model routing distribution, including higher-order statistics beyond pairwise constraints.The framework nevertheless substantially narrows the gap to real-model curves relative to uniform random routing.Source: §8 · p. 9

9 Discussion

The K-sweep preserves the distinct curve shapes of the three tested models, while long-context inference retains Qwen3-30B’s low-batch advantage. The discussion also bounds MoE-drafter implications to a synthetic design study and finds little leverage for routing-based rejection prediction.

9.1 Effect of the Speculative Window K

  • Across K=3, 5, and 7, Qwen3-30B and Qwen3-235B peak at batch size 1, while Mixtral-8x7B peaks near batch size 16.The curve taxonomy is therefore not an artifact of K=3.Source: §9.1 · p. 9

9.2 Long-Context Inference

  • Target efficiency exceeds MT-Bench at every batch size on 32k SPEED-Bench prompts, remaining non-concave and highest at batch size 1.The KV cache limits maximum batch size to 24; the authors attribute the relative efficiency to KV-cache reads diluting extra expert-weight costs.Source: Figure 4 · p. 9, §9.2 · p. 10

9.3 Implications for MoE Draft Layers

  • Synthetic analysis suggests MoE drafters may suit sufficiently sparse targets, especially at larger batch sizes, but does not measure trained-drafter acceptance gains.The 896-expert, under-1%-activation case is a design study, not direct evidence for trained MoE drafters.Source: §9.3 · p. 10

9.4 Routing Inertia in Rejected Draft Tokens

  • Routing overlap offers little leverage for early-exit rejection prediction: accepted and rejected distributions overlap by about 85% even in Qwen3-235B’s largest stable aggregate effect.Held-out slices generally show negligible or small effects, with some task- and language-specific reversals.Source: §9.4 · p. 10

10 Conclusion

Across 10 sparse MoE models, many fine-grained models reach high speculative-decoding speedup near batch size 1, contrary to the expected concave curve. The conclusion attributes verification cost to routing overlap and GPU execution dynamics as well as expert bytes, while identifying limits in synthetic routing and overlap transfer.

10 Conclusion

  • Many of 10 recent sparse MoE models attain high speculative-decoding speedup at or near batch size 1, contrary to the expected concave curve.The proposed maximum-entropy IPF framework predicts unique expert counts from overlap statistics with less than 4% error and narrows the gap to real-model curves under synthetic routing.Source: §10 · p. 10

A Limitations

  • IPF matches pairwise overlap statistics but omits higher-order routing structure, position-dependent expert marginals, and architecture-dependent changes in temporal overlap.How reliably overlap transfers across model families and sparsity patterns remains unresolved.Source: A Limitations · p. 14

B Experimental Setup

  • End-to-end sweeps use 80 MT-Bench prompts oversampled as needed, fixed 256-token outputs, and three independent sessions with speedup CV below 5%.Experiments use vLLM v1; all models run on H100 except Kimi-K2.5 and DeepSeek-V3, which use B200.Source: B Experimental Setup · p. 15, Table 7 · p. 15

C Consolidated Target Efficiency Measurements

  • Figure 5 compares Qwen3-30B speedup curves across BF16/FP8, TP1/TP2, and H100/B200 configurations.The comparison varies per-GPU expert shard size and hardware memory hierarchy.Source: C Consolidated Target Efficiency… · p. 15

C.1 Full 10-Model Speedup Grid

D Cross-Dataset Stability of Non-Concave Speedup Curves

  • Across four datasets, Qwen3-30B and Qwen3-235B retain non-concave curves peaking at batch size 1, despite variation in absolute speedups.The reported pattern supports robustness beyond the MT-Bench prompt distribution.Source: D Cross-Dataset Stability of… · p. 16

E Three-Step Correction: From Weight Model to Measured Target Efficiency

Profiling shows that the activated-expert-bytes model substantially underpredicts target efficiency, while temporal routing overlap and GPU execution dynamics explain why its estimates miss measured behavior. The section extends these findings across hardware and analyzes when routing overlap persists for rejected draft tokens.

E Three-Step Correction: From Weight Model to Measured Target Efficiency

  • Measured target efficiency for Qwen3-30B at batch size 1 on H100 is 0.87–0.90, versus 0.51 predicted by the activated-expert-bytes model.The kernel-level estimate is 0.872; the sweep wall-clock value is 0.896, with the remaining gap attributed to inter-kernel scheduling gaps within the CUDA graph.Source: E Three-Step Correction: From Weight… · pp. 16–17, Table 11 · p. 18
  • Correcting the MoE runtime share and scaling raises predicted efficiency from 0.51 to 0.84; correcting non-MoE scaling raises it to 0.87.MoE accounts for 32.9% of vanilla forward GPU time rather than the 66.7% predicted by weight bytes; measured MoE scaling is 1.57 rather than 2.44, with L2 reuse and overhead dilution contributing.Source: E Three-Step Correction: From Weight… · pp. 16–17

F Additional GPU Profiling: Qwen3-235B and B200

  • Across three additional configurations, the activated-expert-bytes model underpredicts target efficiency by 0.18–0.38, but the underlying mechanisms differ by hardware and kernel.On B200, observed versus predicted efficiency is 0.76 versus 0.51 for Qwen3-30B and 0.61 versus 0.43 for Mixtral; fixed per-step costs likely dominate at batch size 1.Source: F Additional GPU Profiling: Qwen3-235B… · pp. 18–19, Table 12 · p. 19

G Three-Level Overlap Decomposition Formulas

H.2 Deconfounded Results

  • To compare accepted and rejected tokens fairly, the analysis separates first-rejection pairs from successive-rejection cascades; fine-grained models show equal or higher overlap for first-rejection tokens.The comparison holds the predecessor token accepted in both the accepted and first-rejection populations.Source: §H.1 · p. 21, §H.2 · p. 21, Table 14 · p. 21

H.3 Effect Sizes (Cohen’s d)

  • For Qwen3-235B, accepted and rejected overlap distributions still overlap by approximately 85%, making expert overlap unreliable for per-token rejection classification despite p ≪0.001.The mean difference is 3–7× smaller than the per-sample standard deviation; held-out Spec-Bench and Aya results do not change the practical conclusion.Source: §H.3 · pp. 21–22, Table 15 · p. 22

H.4 Cascade Amplification

  • Cascade overlap exceeds first-rejection overlap by 1–5 percentage points, and the routing-inertia effect intensifies in deeper layers.For Qwen3-235B at step 3, 71 of 94 layers show |∆| > 0.05, all in the negative direction; Figure 8 visualizes per-layer differences.Source: §H.4 · p. 22, §H.5 · p. 22, Figure 8 · p. 22

H.6 Coarse Routing Exception

  • Mixtral-8x7B is the sole model with the opposite rejected-token overlap pattern, while anchor-overlap plots show the inertia is local rather than global.Figure 9 reports anchor overlap from step 0 to step j with SEM error bars and Cohen’s d annotations.Source: §H.6 · p. 22, Figure 9 · p. 23, §H.7 · p. 23

H.8 Held-Out Task and Language Slices

Held-out task and language slices largely preserve the measured overlap patterns, with low-acceptance Aya slices producing unstable outliers. Overlap remains above the independence baseline through step 4, and Aya prompts shift Qwen3-30B’s overlap curve upward without changing its decay shape.

H.8 Held-Out Task and Language Slices

  • The median consecutive delta is near zero at step 1 and negative at steps 2–3 on held-out Spec-Bench and Aya slices, matching the aggregate MT-Bench pattern.A handful of low-acceptance settings show slice-level sign reversals.Source: §H.8 · p. 23
  • Low-acceptance Aya outliers are too unstable for a useful routing-based rejection predictor; Qwen3-30B zho has AL ≈1.04, while Qwen3-235B zho and tel have AL ≈1.14 and 1.12.The paper notes the zho behavior may reflect model performance on that slice rather than a general language-level trend.Source: §H.8 · p. 23

I Overlap Decay

  • At step 4, Qwen3-Next-80B overlap is 7.8× baseline (0.281 vs. 0.036), and the decay remains above the independence baseline.Qwen3-235B retains 3.4× baseline overlap at step 4 (0.337 vs. 0.098).Source: I Overlap Decay · pp. 23–24, Table 17 · p. 24

I.1 Held-Out Task and Language Slices

  • Aya shifts Qwen3-30B’s overlap curve upward by roughly 0.03–0.05 at each step and raises the empirical verification ratio from 0.415 to 0.439, while the decay-curve shape is unchanged.The Spec-Bench mean is nearly unchanged from MT-Bench.Source: §I.1 · p. 24, Table 18 · p. 24

J Per-Layer Overlap Analysis

This section examines per-layer routing overlap and synthetic routing behavior, and reports the paper’s societal-impact, release, research, and compute disclosures.

J Per-Layer Overlap Analysis

  • Mid-network layers reach 4–5× the independence baseline, while early layers are near baseline and overlap declines in final layers.The independence baseline stays within 0.078–0.137 across layers, indicating that depth variation primarily reflects temporal-overlap strength rather than expert popularity.Source: J Per-Layer Overlap Analysis · pp. 24–25

L Synthetic Routing Generation via IPF

  • IPF fits each expert’s activation pattern across four tokens to marginal and pairwise-overlap constraints, then predicts the probability that the expert activates at least once.The skewed variant bins experts by empirical activation probability and solves separately per bin, capturing non-uniform expert popularity as well as temporal overlap.Source: L Synthetic Routing Generation via IPF · pp. 25–26

K Unique Expert Counts and IPF Prediction

L Synthetic Routing Generation via IPF

  • The framework generates synthetic token-to-expert assignments by sampling four-token patterns and repairing cardinality, matching per-layer routing probabilities with r>0.99 correlation.The generated assignments are used in the dummy-weight experiment.Source: L Synthetic Routing Generation via IPF · p. 26

M.1 Analytical Cost Model

  • The analytical cost model anchors MoE draft cost at batch size 1 and adds measured incremental per-layer cost, accounting for pipelining and framework overhead.This avoids double-counting attention costs while capturing the transition from bandwidth-bound to compute-bound regimes.Source: §M.1 · p. 26

M.2 Target Efficiency and Draft Cost for Qwen3-30B

  • For Qwen3-30B, a dense drafter is cheaper at small batches, but becomes more expensive than an MoE drafter at larger batches; the gap grows for multi-layer drafts.A one-layer MoE drafter has approximately the same per-step cost as the dense drafter at batch size 1, but this is a synthetic design study rather than a trained drafter evaluation.Source: §M.2 · pp. 26–27, §M.1 · p. 26, Figure 11 · p. 27

M.3 Synthetic Experiment with N=896

  • In the N=896 synthetic experiment, the MoE drafter remains bandwidth-bound to larger batches, suggesting MoE draft layers may be attractive across a broader batch-size range.The target-efficiency benefit is more pronounced with more draft layers; the experiment uses dummy weights and uniform IPF routing based on Qwen3-30B overlap statistics.Source: §M.3 · p. 27, Figure 12 · p. 27

2. Limitations

  • The authors say compute-resource details are provided in Appendix B.They indicate that the experimental setup describes the required compute resources.Source: §2 · p. 32
  • The authors report that they have read and sought to comply with the NeurIPS Code of Ethics.Source: §2 · p. 32
  • The authors report that the work has no societal impact and poses no high-risk release concerns.They state that safeguards are not applicable because the paper poses no such risks.Source: §2 · pp. 32–33
  • The authors use and cite vLLM, an open-source LLM inference framework, and report that they introduce no new assets.The paper answers that existing assets are properly credited and that it does not introduce new assets.Source: §2 · p. 33
  • The work does not involve crowdsourcing or research with human subjects.The authors also report that it does not involve human-subject research requiring IRB information.Source: §2 · p. 34
  • The authors state that LLMs are not an important, original, or non-standard component of the research.They therefore mark declaration of LLM usage as not applicable.Source: §2 · p. 34

Related papers

Loading 2610.38981v1…