PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang

arXiv:2610.15417v1cs.AI

TL;DR

With offloaded MoE weights, speculative decoding must account for the distinct experts activated by each verified token, not just the number of tokens. SpecFit searches subtrees by predicted reward per cycle time and uses the same objective to choose drafting depth. On summarization and GSM8K, it decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache, respectively.

  • Problem

    With offloaded MoE weights, each verified token can add experts to the H2D transfer demand, so verification cost depends on which draft tokens are selected as well as how many.

    Source: §1 · pp. 1–2

  • Method

    SpecFit exactly selects an ancestor-closed draft subtree by estimated committed-token reward divided by cycle-time cost, then uses that selection to determine when to stop drafting.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    21.6% and 7.0% faster than EAGLE-3 with the same expert cache on summarization and GSM8K, respectively; SpecFit is 1.97× and 1.72× as fast as vanilla EAGLE-3 on those tasks.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    SpecFit improves decoding throughput over vanilla EAGLE-3 and EAGLE-3 with an expert cache on summarization and GSM8K without altering target routing or expert execution.

    Source: §1 · p. 2

    The per-cycle ratio objective differs from throughput’s ratio of expectations; its analysis assumes independent, identically distributed cycle states independent of earlier selections.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.

1 INTRODUCTION

With expert offloading, MoE verification cost depends on which draft tokens are selected because each can add experts to the transferred union. SpecFit selects verification subtrees by predicted throughput, and the reported evaluations show faster decoding than EAGLE-3 baselines.

1 INTRODUCTION

  • A larger verified tree can commit more tokens but also transfer more experts, so verification cost depends on which draft tokens are selected, not just how many.Rejected branches still contribute experts to the activated union.Source: §1 · p. 2
  • SpecFit chooses an ancestor-closed verification subtree by predicted throughput rather than acceptance length, using draft-probability reward and estimated cycle time based on distinct activated experts.The exact search also determines how many draft nodes to verify and when to stop drafting.Source: §1 · p. 2
  • 21.6% and 7.0% faster than EAGLE-3 with the same expert cache, and 1.97× and 1.72× as fast as vanilla EAGLE-3, on summarization and GSM8K, respectively.The evaluations use 40 summarization articles and an 80-question GSM8K subset.Source: §1 · p. 2
  • The method leaves the target’s routing and expert execution unchanged, and the paper characterizes when smaller subtrees improve throughput while bounding the per-cycle ratio rule’s distance from the throughput-optimal rule.The selection problem is also shown to be NP-hard under a size limit.Source: §1 · p. 2

2 BACKGROUND

This section explains why selecting a speculative tree for an offloaded MoE target requires accounting for the experts activated by its verified nodes. Acceptance, cost, structural, and cycle-time considerations make this selection more complex than choosing the largest tree.

2 BACKGROUND

  • An offloaded MoE forward must execute the per-layer union of experts required by its query, and cache misses trigger host-to-device copies before the layer can finish.The cache may avoid later transfers for experts that remain resident, but each forward still has to meet its expert demand.Source: §2 · p. 2
  • Draft tokens on different positions and branches are routed separately, so rejected branches still incur expert demand and enlarge the union a verified tree activates.Thus, a forward's expert cost depends on the selected set, not solely on its number of tokens.Source: §2 · p. 3
  • Selecting a verified set is difficult because acceptance is uncertain, expert costs overlap, nodes require their prefixes, and the objective trades committed tokens against cycle time.The expert-union cost is a coverage function, so a node's marginal cost depends on the other selected nodes.Source: §2 · p. 3
  • Existing selectors use acceptance or tree size and depth, while MoE methods prefetch, limit, or add experts; the cited selectors do not account for the expert union.Tree-based decoding selects a subset of candidate nodes, and greedy verification commits the longest matching root-to-node path.Source: §2 · p. 3

3 METHOD

SpecFit selects an ancestor-closed draft subtree using estimated committed tokens and cycle time, where the cost estimate reflects the subtree’s per-layer expert union. It searches the candidate pool exactly and reuses the selection objective to decide drafting depth.

3 METHOD

  • SpecFit drafts a candidate pool level by level, selects the legal subtree with the largest predicted reward-to-time ratio, and stops drafting when the newest level contributes no selected node.The target then verifies the selected subtree.Source: §3 · p. 3, Figure 1 · p. 4

3.1 SELECTING A VERIFICATION SUBTREE

  • A legal subtree contains at most K nodes and includes every selected node’s ancestors; the candidate generator uses top-2 expansion, depth at most 5, and at most 18 nodes.K is set to min(7, |V|).Source: §3.1 · p. 3

3.2 ESTIMATING REWARD AND EXPERT COST

  • The estimated committed-token reward is one plus the sum of selected nodes’ commit probabilities, which SpecFit estimates using the drafter’s path probabilities without assuming independent commit events.The root contributes one committed token.Source: §3.2 · pp. 3–4
  • SpecFit estimates cost from node count and a per-layer expert-union feature; the resulting coverage function is monotone and submodular because each node adds only experts not already in the subtree.Selected branches contribute to estimated demand whether or not they are later accepted.Source: §3.2 · p. 4

3.3 SELECTION, VERIFICATION, AND AVAILABLE INFORMATION

  • The selector maximizes predicted reward divided by predicted cycle time over legal subtrees, jointly choosing node identities and subtree size.The cycle-time coefficients are fitted once on GSM8K calibration questions and then frozen.Source: §3.2 · p. 4, §3.3 · p. 5
  • The per-cycle ratio objective is not the throughput-optimal ratio of expectations; its stated comparison assumes independent, identically distributed cycle states independent of earlier selections.For the bounded pools used, the NP-hard selection is solved exactly by enumeration; pruning raises throughput iff it saves a larger fraction of cycle time than committed tokens.Source: §3.3 · p. 5
  • After each draft level, SpecFit continues drafting only if the selected subtree includes a node at that level; otherwise it stops and verifies the selection.The stopped selection equals the full-pool selection whenever the latter stays within the drafted levels.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments compare SpecFit with EAGLE-3 and autoregressive decoding across two workloads, cache settings, and GPUs, measuring throughput, verification and transfer counters, and selection overhead. The main comparison uses Qwen3-30B-A3B at batch size one, with 48 expert slots per layer when caching.

4.1 SETUP

  • The main setup uses Qwen3-30B-A3B-Instruct-2507 with a RedHatAI EAGLE-3 drafter on 40 CNN/DailyMail articles and 80 GSM8K questions, at request batch size one.All configurations use an offload backend; the primary cached comparison holds 48 experts per layer, and SpecFit and EAGLE-3 share a drafter.Source: §4.1 · p. 6

4.2 SUMMARIZATION

  • 8.27 tokens/s, 21.6% above cached EAGLE-3 and 1.77× AR on CNN/DailyMail; SpecFit is also 19.9% faster including prefill.It verifies 2.88 draft nodes and commits 13.1% fewer tokens per cycle, but each cycle takes 28.5% less decode time and transfers 22.1% fewer experts per output token.Source: Figure 2 · p. 6, §4.2 · pp. 6–7, Table 2 · p. 7

4.3 GSM8K

  • 10.79 tokens/s, 7.0% above cached EAGLE-3 on GSM8K; SpecFit is 6.3% faster including prefill and 2.29× AR.It commits 7.8% fewer tokens per cycle, while cycles take 13.8% less decode time and transfer 11.6% fewer experts per output token.Source: Figure 3 · p. 7, §4.3 · p. 7, Table 3 · p. 7

4.4 SENSITIVITY TO OFFLOAD PRESSURE

  • SpecFit’s decoding gain over EAGLE-3 rises from 21.6% at 48 slots to 24.3% at 24 slots, 29.2% at 12 slots, and 31.6% without cache.With cache, its cycles are 28.5%, 30.1%, and 32.7% shorter at 48, 24, and 12 slots, respectively.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 ANOTHER GPU

  • 6.70 tokens/s, 10.8% above cached EAGLE-3 and 2.51× AR on an RTX 4090; without cache, SpecFit is 18.1% faster than vanilla EAGLE-3.It verifies 4.53 draft nodes instead of 6.99 and commits 7.5% fewer tokens per cycle, while cycles take 16.5% less decode time.Source: Table 4 · p. 8, §4.5 · pp. 8–9

4.6 ROUTE PREDICTION AND SELECTION COST

  • With simulated 60%-accurate routes, SpecFit reaches 2.26× AR on GSM8K; its selection takes a median 2.82 ms per cycle.The measurements use atlas routes outside the request timer; training route predictors is left to future work.Source: §4.6 · p. 9

5 RELATED WORK

Prior MoE offloading work changes expert loading or execution, while prior tree-selection work generally values acceptance or hardware cost. SpecFit keeps the EAGLE-3 candidate generator and instead selects verification depth and nodes using expert-union cost.

Expert offloading

  • Offloading systems cache and prefetch expected experts or compute some on the CPU, whereas SpecFit changes which tokens create expert demand.SpecFit uses an on-demand expert cache without prefetching.Source: §5 · p. 9

Draft-tree construction and selection

  • SpecFit keeps a fixed EAGLE-3 drafter and candidate generator, changing only drafting depth and the subset sent to verification.It values both by expert-union cost rather than tree size or shape.Source: §5 · p. 9

Speculative decoding for MoE models

  • Unlike SP-MoE, MoE-Spec, and DraftExpert, SpecFit executes all routed experts for selected tokens and trains no model component.It evaluates a complete-subtree reward/cost ratio over a bounded feasible family, with cardinality included in the search.Source: §5 · p. 9

6 CONCLUSION

SpecFit selects verification subtrees and drafting depth by predicted reward relative to expert-union cost, without changing the drafter, router, or expert execution. It improves decoding over cached EAGLE-3 on both evaluated workloads.

6 CONCLUSION

  • With Qwen3-30B-A3B, SpecFit verifies fewer draft nodes when their expert cost outweighs their expected tokens.It selects each cycle’s verification subtree and drafting depth by draft-probability reward relative to expert-union cost, without changing the drafter, router, or expert execution.Source: §6 · p. 9
  • 21.6% faster summarization decoding than EAGLE-3 with the same expert cache.On GSM8K, decoding is 7.0% faster, with each cycle 13.8% shorter and throughput 2.29× that of AR decoding without the cache.Source: §6 · p. 10
  • As the cache shrinks from 48 to 12 slots per layer, the summarization gain grows to 29.2%; on an RTX 4090, the GSM8K gain is 10.8%.Source: §6 · p. 10

AI USE STATEMENT

The authors used AI assistants in software development, experiment inspection, mathematical exposition, literature search, and manuscript drafting, and reviewed all AI-assisted content.

AI USE STATEMENT

  • AI assistants supported software development, experiment inspection, mathematical exposition, literature search, and manuscript drafting; the authors reviewed the assisted content and take responsibility for the paper’s claims, references, and results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work accelerates existing MoE models with experts offloaded to host memory, using publicly released models and datasets and collecting no new data.

ETHICS STATEMENT

  • SpecFit changes which draft tokens are verified while the target verifies them unmodified, adding no capability or risk beyond the accelerated model.The work involves no human subjects and collects no new data.Source: ETHICS STATEMENT · p. 10

A SELECTION THEORY AND PROOFS

The section relates per-cycle subtree selection to long-run predicted throughput under an i.i.d. cycle-state assumption, then derives a stopping rule for adaptive draft depth.

A.1 THROUGHPUT OF A SELECTION RULE

  • Under the stated i.i.d. cycle-state assumption, the throughput-optimal rule selects the subtree maximizing expected reward minus λ* times expected cycle time.This rule attains the maximum predicted throughput λ* across selection rules.Source: §A.1 · p. 13
  • When λ is below or above the largest per-cycle ratio ρ, the λ-based selection has respectively no more or no less predicted time than the ratio-maximizing selection.The comparison follows from the lemma’s relation between selections maximizing bR(S)−λbT(S) and the selection with ratio ρ.Source: §A.1 · p. 13

A.2 ADAPTIVE DEPTH

  • Adaptive depth stops at the first level whose optimal subtree matches the previous level’s selection, unless the full-pool optimum includes a deeper node.The legal subtree families are nested as depth increases.Source: §A.2 · p. 13

A.3 COMPLEXITY AND EXACT SEARCH

The section establishes hardness for bounded subtree selection, identifies a tractable unbounded case under specified intercepts, and describes SpecFit’s exact enumeration.

A.3 COMPLEXITY AND EXACT SEARCH

  • Deciding whether a legal subtree reaches a given reward-to-time ratio is NP-complete, even with depth one, one MoE layer, and equal intercepts.Without the size limit and with affine, nondecreasing intercepts, the largest ratio is computable in polynomial time by parametric minimum cut.Source: §A.3 · p. 14
  • Without a size limit, the affine-intercept objective becomes a maximum-weight closure problem solvable through parametric minimum cuts.The construction links each node to its parent and routed expert pairs, then uses nested minimum cuts across λ values.Source: §A.3 · p. 14
  • SpecFit’s depth-first search reaches every legal ancestor-closed subtree exactly once by adding nodes in parent-before-descendant order.It carries expert bitmaps incrementally and ranks candidates by reward-to-time ratio with stated tie-breakers.Source: §A.3 · p. 14

A.4 EVALUATION ON RECORDED POOLS

On recorded candidate pools, the per-cycle ratio rule stays close to the predicted throughput optimum, while selections and adaptive stopping vary across cycles.

A.4 EVALUATION ON RECORDED POOLS

  • Below 1% of λ* was the optimality gap of Equation 5, within Equation 7’s bound, although the two rules chose different trees in about a quarter of cycles.With recorded acceptances, S* committed 0.5% and 0.7% more tokens per unit of predicted time than bS.Source: §A.4 · p. 15
  • Doubling β lowered both expert demand per committed token and the number of selected nodes.The evaluation used recorded cycles from 40 summarization articles and 80 GSM8K questions, with coefficients fitted using the cache.Source: §A.4 · pp. 14–15
  • Drafting stopped before level 5 in 85.8% of summarization cycles and 33.7% of GSM8K cycles.Table 5 reports selection-theory evaluation on candidate pools drafted by SpecFit, with coefficients fitted with the cache.Source: §A.4 · p. 15, Table 5 · p. 15

B EXPERIMENTAL DETAILS

The appendix specifies the model, workloads, offloading and timing setup, and coefficient calibration used in the experiments.

B EXPERIMENTAL DETAILS

  • The target has 48 MoE layers, 128 experts per layer, and top-8 routing; workloads are 40 even-numbered CNN/DailyMail articles and an 80-question GSM8K subset.The stated input and output token limits differ by workload.Source: B EXPERIMENTAL DETAILS · p. 15
  • The offload backend loads missing experts on demand with at most two copies in flight, while the no-cache configuration clears the cache before every target forward.The KV cache is retained, and experts shared by tokens in one forward are loaded once.Source: B EXPERIMENTAL DETAILS · p. 15
  • Measured request time includes prefill, drafting, atlas conversion, selection, verification, commit, and required synchronization, but excludes model loading and post-request scoring.The request stream starts with an empty expert cache, whose contents carry over between questions.Source: B EXPERIMENTAL DETAILS · pp. 15–16
  • Calibration fits cycle time as b_k plus βF from 144 probe points across nine GSM8K states, using probability-prefix and minimum-expert-union probe trees.The fit constrains β and the intercepts to be nonnegative; intercepts are fitted independently and need not increase with k.Source: B EXPERIMENTAL DETAILS · pp. 15–16

Related papers

Loading 2610.15417v1…