PaperlayerView exact revision on arXiv

Source-linked AI summary

ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan

arXiv:2608.24938v1cs.LGcs.AI

TL;DR

Existing training-free MoE acceleration methods address prefill’s token-compute and decode’s batch-memory bottlenecks without explicitly reconstructing excluded experts’ contributions. ExFold folds those contributions into retained experts using a shared mechanism across phases; it reports up to 1.41× TTFT and 2.45× TPOT speedups while retaining about 99% of original average quality.

  • Problem

    Existing training-free MoE acceleration methods address prefill’s token-compute and decode’s batch-memory bottlenecks but do not explicitly reconstruct excluded experts’ contributions.

    Source: Introduction · pp. 1–2

  • Method

    ExFold selects retained experts under phase-specific token and batch budgets, then uses one calibrated scalar-projector matrix and shared folding rule to fold excluded contributions into retained experts.

    Source: Introduction · p. 2, Unified Expert Folding · p. 4

  • Results

    Up to 1.41× TTFT and 2.45× TPOT speedups while retaining about 99% of the original average quality.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The results establish output recovery as a practical basis for unified MoE inference acceleration.

    Source: Conclusion · p. 8

    The evaluated artifact’s online vector transform and substantial state prevent reuse of the standard fused MoE path.

    Source: Implementation · p. 12

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.

Introduction

ExFold unifies prefill and decode acceleration by selecting phase-specific expert sets and folding excluded contributions into retained experts. This addresses prefill’s token-level computation bottleneck and decode’s batch-level expert-memory-traffic bottleneck.

Motivation

  • Prefill is dominated by token-level expert computation, whereas decode is dominated by batch-level expert memory traffic; both TTFT and TPOT matter for low-latency serving.Source: Introduction · p. 1

Limitations of prior methods

  • Prior training-free methods reduce token-wise execution or the active expert set, but leave excluded contributions discarded, frozen, or only implicitly approximated.Source: Introduction · p. 2

Expert Folding

  • Expert Folding calibrates per-layer scalar-projector and projection-loss matrices on unlabeled text, routing each excluded expert to a minimum-loss target and scaling its contribution into that target’s router weight.The method is motivated by directionally aligned expert outputs whose magnitudes differ substantially, making direct substitution insufficient.Source: Introduction · p. 2

Unified phase-aware design

  • ExFold selects Kpre dominant experts per token in prefill and D dominant experts per batch in decode, using the same folding mechanism to recover excluded contributions.The phases differ only in retained-expert selection, unifying them as output approximation under phase-specific selection constraints.Source: Introduction · p. 2

Results

  • ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while retaining about 99% of original average quality.The vLLM plugin uses a lightweight expert-folding CUDA kernel; evaluation covers prefill, decode, and joint acceleration across multiple MoE architectures on real serving workloads.Source: Introduction · p. 2

Background and Motivation

MoE prefill and decode have different resource bottlenecks: token-wise expert computation versus memory traffic from the batch-wise expert union. Existing training-free methods target one phase-specific budget, but do not preserve excluded experts’ contributions under a shared objective.

MoE Inference

  • Prefill is compute-bound, so its cost falls with fewer experts per token; decode is memory-bound, with cost tracking the distinct experts activated across the batch.Prefill’s expert weight-loading is amortized across many prompt tokens, while decode loads a broad expert union to serve few tokens.Source: MoE Inference · p. 2

MoE Acceleration Methods

  • Expert outputs can align across experts and layers while their L2 norms vary substantially.This figure-level observation motivates distinguishing output direction from magnitude when accounting for excluded experts.Source: Figure 3 · p. 3
  • Token-wise sparsification methods reduce experts executed per token, whereas expert-set consolidation reduces the active expert set; each relieves one phase’s budget but leaves the other uncontrolled.Token-wise methods leave decode’s expert union uncontrolled, while expert-set methods do not match the per-token prefill budget.Source: MoE Acceleration Methods · p. 3
  • Neither acceleration family preserves excluded experts’ contributions, so approximation error grows as budgets tighten and compounds across phases with different targets.The paper frames the challenge as unifying prefill and decode acceleration with minimal quality loss.Source: MoE Acceleration Methods · p. 3

Design Insight

Prefill and decode impose different expert-selection budgets but leave the same excluded-output recovery problem. ExFold’s design uses per-pair scalar correction to fold excluded contributions into retained experts, while each phase differs only in how it selects the retained set.

Design Insight

  • The residual from budget-excluded experts has the same form, αe(x)Ee(x), for prefill and decode, so both phases recover the same excluded contributions.Prefill is constrained by experts per token, while decode is constrained by experts per batch; the constraint selects the retained set, not the recovery target.Source: Design Insight · p. 3
  • 0.335 to 0.529 at layer 5 and 0.251 to 0.471 at layer 17: unit-scale normalization raises average expert-to-expert similarity.The similarity gap persists across all 48 layers, while raw output norms span over 3× within a layer (max / min = 3.04× and 3.27× for the two shown), motivating scalar rescaling of retained experts.Source: Design Insight · p. 3
  • ExFold uses one cheap per-pair scalar correction to recover excluded contributions across both phases, changing only how each phase selects retained experts.This design addresses the tension between prefill and decode budgets without recomputing excluded experts or relying on blind substitution.Source: Design Insight · p. 3

Method

ExFold treats prefill and decode as budgeted output-approximation problems: each phase selects retained experts, while one calibrated scalar-folding mechanism recovers omitted experts’ contributions. The method uses unlabeled calibration and phase-specific selection under a shared transfer criterion.

Unified Expert Folding

  • ExFold frames both phases as budgeted output approximation: omitted routed experts’ calibrated contributions are folded into retained experts, and coalesced routes let each retained target execute once.Prefill cost counts retained experts per token, while decode cost counts distinct retained experts per batch; folding changes neither expert parameters nor the original router.Source: Unified Expert Folding · p. 4

Training-Free Projector Calibration

Phase-Specific Selection and Unified Folding

  • Prefill and decode use phase-specific retained-expert selectors but share the calibrated transfer criterion and folding operator.The shared inference mechanism is parameterized by the scalar table S(l) and loss table L(l).Source: Phase-Specific Selection and Unified… · p. 4

Main Quality Results

  • At P4, ExFold preserves the original Qwen3 average and scores 69.26 versus Direct Top-4’s 66.64; jointly, P4+D64 retains 68.50 versus 56.17 for Direct Top-4 in both phases.The comparisons use matched execution budgets.Source: Table 1 · p. 5, Main Quality Results · pp. 6–7
  • At D32, ExFold remains within 0.07 points of the original; on GLM-4.5-Air it retains 97.2% and 98.4% under P4 and D64, respectively, and 96.7% under P4+D32.The GLM-4.5-Air results demonstrate quality retention beyond the primary model.Source: Table 2 · p. 5, Main Quality Results · pp. 6–7

Efficiency Evaluation

  • At 8 QPS, ExFold reaches 1.41× TTFT and 2.45× TPOT speedups, sustains about 2.4× TPOT through 12 QPS, and improves offline throughput by 1.20×.The decode benefit comes from bounding the batch-level expert union; token-wise Top-4 and Top-6 lose their benefit as QPS increases.Source: Figure 5 · p. 6, Efficiency Evaluation · p. 7

Ablations and Analysis

  • Mixed calibration yields the best average (73.00), though all domains remain within 1.9 points; ranking retained experts by Si × Hi gives the best average in both phases.Task-matched calibration is not consistently optimal; router score alone omits output scale, while magnitude alone omits routing relevance.Source: Table 4 · p. 7, Table 5 · p. 7, Ablations and Analysis · p. 8

Scaling to DeepSeek-V4-Flash

  • On DeepSeek-V4-Flash, ExFold P3+D64 retains 97.68% versus 88.17% for all-stage K=3; P3+D128 retains 99.32%, while decode-only D64 and D128 stay within 0.2 average points of Original.P3 reaches 1.32× mean 8K TTFT speedup at QPS 8; D64 stabilizes near 1.15× TPOT speedup and raises saturated output throughput from 5.22k to 6.72k tokens/s (1.29×), ahead of Direct K=3, REAP-D128, and D128.Source: Scaling to DeepSeek-V4-Flash · p. 7, Table 6 · p. 8, Figure 7 · p. 8

Conclusion

ExFold jointly accelerates MoE prefill and decode by combining phase-specific retained-set selectors with a shared directed projector that folds excluded contributions into router metadata. Across MoE architectures, it improves TTFT, TPOT, and serving throughput while preserving more quality than expert dropping.

Conclusion

  • Across MoE architectures, ExFold improves TTFT, TPOT, and serving throughput while preserving more quality than expert dropping.The conclusion provides no specific benchmark values.Source: Conclusion · p. 8
  • ExFold combines phase-specific retained-set selectors with a shared directed projector that folds excluded contributions into router metadata.It is a training-free framework for jointly accelerating MoE prefill and decode.Source: Conclusion · p. 8
  • Because folding changes only router metadata, ExFold remains complementary to kernel, scheduling, and parallelism optimizations.The conclusion presents output recovery as a practical basis for unified MoE inference acceleration.Source: Conclusion · p. 8

Reproducibility Details

The reproducibility details specify the evaluated model architectures, benchmark and scoring protocols, projector calibration, and serving configurations. Together, they describe the fixed conditions used for quality and efficiency comparisons.

Model architectures

  • The evaluated checkpoint configurations report each model’s exposed architecture; shared experts run alongside routed Top-K experts, and “Expert FFN” denotes each expert’s feed-forward hidden width.The architectures are summarized in Table 7.Source: Reproducibility Details · p. 11

Benchmarks and metrics

  • Evaluations use fixed benchmark snapshots, including 490 MMLU-Pro cases for GLM-4.5-Air, 294 IFBench cases, and the same 167-problem LiveCodeBench v5 subset for every method.LiveCodeBench uses eight generations per problem; Qwen3-30B-A3B uses 32 samples per AIME 2024 problem, while GLM-4.5-Air uses eight. LiveCodeBench reports pass@8 and mean correctness, with pass@8 primary in aggregate comparisons.Source: Reproducibility Details · p. 11

Calibration protocol

  • Qwen3 and DeepSeek-V2-Lite projectors use 32 unlabeled sequences of at most 4,096 tokens; Qwen3.5 uses four eight-sequence shards, and GLM-4.5-Air uses 16 sequences of at most 1,024 tokens.Calibration uses forward activations only, without labels, gradients, or evaluator feedback; coefficients use source-output-norm weighting with λ = 10^-3, are clipped to [−4, 4], and self-pairs are fixed at one. Unobserved pairs receive loss 1030.Source: Reproducibility Details · p. 11

Quality evaluation

  • Quality comparisons fix prompts, decoding parameters, sample counts, and scoring code within each model–benchmark pair, and aggregate individually scored generations into percentages and arithmetic-mean “Avg.” scores.IFEval uses prompt-level strict scoring with thinking disabled; IFBench averages prompt- and instruction-level accuracy under strict and loose checks. LiveCodeBench and HumanEval+ execute generated programs against benchmark tests using the LiveCodeBench and EvalPlus evaluators (Jain et al. 2024; Liu et al. 2023).Source: Reproducibility Details · p. 11

System and serving configuration

  • Efficiency tests use eight-GPU H800 80GB servers, with prefill reporting mean TTFT and matched decode reporting mean TPOT under distinct tensor-parallelism, request, and QPS settings.Prefill uses tensor parallelism four, 1–8 QPS, 8,192-token prompts, and one generated token; decode uses vLLM 0.11.0, tensor parallelism one, 2–12 QPS, 512 requests, one input token, and 256 generated tokens. Speedups are normalized to Original under matched hardware and settings.Source: Reproducibility Details · p. 11

Implementation

ExFold implements scalar projectors calibrated from weighted reconstruction losses, then uses phase-specific expert selection to fold excluded contributions into retained experts. Prefill limits experts per token, while decode constrains the batch-level expert pool; both reuse the same folding mechanism.

Implementation

  • Pairwise scalar projectors fit directed source-to-target differences, while global and layer-wise variants share coefficients more broadly.Calibration weights samples by source-output norm, and normalized reconstruction loss guides target assignment; pairwise sharing preserves directed expert-level differences as a scalar metadata update.Source: Implementation · p. 11
  • The evaluated scalar-plus-low-rank artifact uses a rank-32 basis and at most 1,024 directed pairs per layer; its basis and pair-specific correction require an online vector transform and substantial state, preventing standard fused MoE reuse.It evaluates the first R ∈ {8, 16} directions; pairs outside materialized support use only the scalar term.Source: Implementation · p. 12

Phase-Specific Routing and Folding

  • Prefill retains each token’s Kpre routed experts with the largest estimated contribution magnitudes and folds omitted routes into retained targets.The Triton operator coalesces omitted routes assigned to the same target without materializing their outputs, leaving the fused MoE kernel to execute only Kpre experts per token.Source: Phase-Specific Routing and Folding · p. 12
  • Decode selects either a dynamic pool ranked by aggregate routed contribution or a static pool formed from calibration-time norms, then remaps omitted experts to minimum-loss targets.Dynamic decode scans BK routed slots and searches D candidate targets; static decode caches target and scalar mappings for all source experts, giving O(BK) per-route remapping work.Source: Phase-Specific Routing and Folding · pp. 12–13

Cross-Model Quality Results

The section presents cross-model quality results for joint compression on DeepSeek-V2-Lite-Chat and phase-specific or joint folding on Qwen3.5-35B-A3B. The supplied passages describe evaluation settings but do not provide table-cell values or support a quality comparison.

Cross-Model Quality Results

Expert Geometry and Projector Structure

Across Qwen3 and GLM-4.5-Air, scalar alignment increases expert-output similarity, while common-input diagnostics show that many directed Qwen3 expert transfers have low normalized projection loss. The layer-wide views also reveal nonuniform pair structure and expert-output magnitudes.

Full-Layer Expert Geometry

  • Across 48 Qwen3 layers and 128 routed experts, projector matrices and expert magnitudes vary by layer; GLM-4.5-Air shows nonuniform co-routed pair structure and magnitude differences across 45 layers.GLM diagnostics cover only co-routed pairs, and gray cells mark pairs never jointly observed.Source: Full-Layer Expert Geometry · p. 13
  • Mean similarity rose from 0.449 to 0.613 for Qwen3 and from 0.899 to 0.932 for GLM-4.5-Air after scalar alignment.These geometry diagnostics use analysis-only scalar fits, separate from the fixed deployment projectors.Source: Full-Layer Expert Geometry · p. 13, Figure 10 · p. 17, Figure 12 · p. 19

Layer-Wise Projector Structure

DeepSeek-V4-Flash Extension Details

The DeepSeek-V4-Flash extension reports speed results with separate quality and saturation operating points, and qualifies them with benchmark-aware calibration and a confidence-aware P3 safeguard. It also specifies the model and runtime conditions used for evaluation.

  • DeepSeek-V4-Flash has 284B total and 13B activated parameters, 43 layers, 256 routed experts plus one shared expert, and Top-6 routing.Quality evaluation uses tensor parallelism eight, max_num_seqs=32, CUDA Graph, and the same prompts, formatting, decoding rules, and scorers across methods.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • Calibration uses 64 unlabeled inputs, including eight benchmark inputs, making it transductive and benchmark-aware rather than zero-contact held-out evaluation.It uses no benchmark answers or evaluator feedback and changes no model weights.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • The DeepSeek artifact uses common-input expert outputs, source-output-norm weighting, and unbounded least-squares scalars, without Qwen’s coefficient clipping.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • P3 transfers part of an omitted source to its minimum-loss target and distributes the remainder across retained Top-3 routes, still invoking exactly three routed experts.The safeguard is motivated by DeepSeek-V4-Flash’s wider range of router scores and expert-output magnitudes.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • Speed tests use one H800 server with tensor parallelism four and CUDA Graph; online load curves use max_num_seqs=32, while offline throughput uses max_num_seqs=128.Prefill tests use 8,192 input tokens and one output token; decode tests use one input token and 256 output tokens.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • The TTFT peak at QPS 4 includes queueing amplification, so it is a serving-level speedup rather than a pure-kernel claim.Source: DeepSeek-V4-Flash Extension Details · p. 22
  • At max_num_seqs=32, D64 stabilizes near 1.15× TPOT at QPS 8–32; at saturated max_num_seqs=128, it yields 1.286× output throughput.The saturated throughput result is reported separately from the quality–speed operating point.Source: DeepSeek-V4-Flash Extension Details · p. 22

Token-Level Reconstructions

Figure 15 uses selected token-level proxy-space diagrams to illustrate ExFold’s reconstructions relative to other routing approximations, not to provide aggregate evaluation evidence or hidden-state reconstruction errors.

Token-Level Reconstructions

  • At least 15% lower proxy distance than Direct Top-4, ExFold also has the smallest displayed proxy distance in each selected illustrative case.Cases were selected only when Direct Top-4 proxy distance was at least 0.15 and ExFold reduced it by at least 15%.Source: Token-Level Reconstructions · p. 23
  • Expert contributions form strict head-to-tail chains in a two-dimensional MDS proxy, with arrow lengths set by router weight times calibrated expert-output norm.Chain endpoints represent aggregates; background arrows show the original aggregate y and approximate aggregate by.Source: Token-Level Reconstructions · p. 23
  • Illustrations cover four task datasets and layers 0, 16, 32, and 47, but show proxy-space distances rather than hidden-state errors or aggregate evaluation results.Each dataset uses at most eight evaluation examples and 96 deterministically spaced tokens per example; the nearest qualifying layer and first qualifying case are shown.Source: Token-Level Reconstructions · p. 23

Related papers

Loading 2608.24938v1…