Source-linked AI summary
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan
TL;DR
MoE serving has separate prefill compute and decode memory bottlenecks, while existing training-free methods do not explicitly recover excluded expert contributions. ExFold unifies both phases through calibrated scalar-projector folding with phase-specific retained-set selection, achieving faster serving with near-original quality. Its evaluated artifact requires additional online state and cannot reuse the standard fused MoE path.
Problem
Existing training-free MoE accelerators optimize either experts executed per token or experts activated per batch, without explicitly reconstructing excluded contributions.
Method
ExFold calibrates scalar and loss matrices on unlabeled data, then folds excluded expert contributions into phase-specific retained experts through router-weight corrections.
Results
ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while retaining about 99% of original average quality.
Takeaways & Limitations
A shared output-recovery mechanism supports unified prefill and decode acceleration while changing only the retained-expert selection constraint.
Takeaways & Limitations
The evaluated artifact requires substantial online state and vector transforms, preventing reuse of the standard fused MoE path.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.
Introduction
ExFold addresses the distinct compute and memory bottlenecks of MoE prefill and decode by unifying phase-specific expert selection with explicit recovery of excluded expert contributions.
- Motivation: Prefill is dominated by token-level expert computation, whereas decode is dominated by batch-level expert memory traffic.These phases therefore require different resource budgets: experts per token for prefill and distinct experts per batch for decode.
- Research gap: Existing training-free methods reduce executed experts or active expert sets but do not explicitly reconstruct excluded expert contributions.Pruning discards contributions, static merging permanently compresses them, and similarity rerouting does not calibrate output-scale mismatch.
- Evaluation: ExFold is implemented as a vLLM plugin with a lightweight CUDA kernel and preserves excluded contributions while accelerating both phases.The design is motivated by expert outputs that can align directionally despite substantial magnitude differences.
- Evaluation: 1.41× TTFT and 2.45× TPOT speedups are achieved while retaining about 99% of original average quality.The reported implementation targets real serving workloads across prefill and decode.
- Framework: ExFold formulates prefill and decode as one budgeted output-approximation problem with phase-specific retained-set selection.Prefill uses token-level Top-K selection, while decode restricts the batch-level expert pool.
- Expert Folding: A training-free folding mechanism projects excluded expert contributions onto retained experts using directional and scalar projectors calibrated from unlabeled data.The scalar correction is absorbed into router weights, while a reconstruction-loss matrix selects minimum-loss targets.
Design Insight
The paper argues that prefill and decode share the same excluded-contribution recovery problem despite selecting retained experts under different budgets. Directional alignment and scale variation make scalar correction a shared solution.
- Shared recovery target: Different phase budgets select different retained expert sets, but excluded contributions have the same residual form in the MoE output.This makes recovery a common approximation problem rather than two unrelated phase-specific problems.
- Magnitude-separable redundancy: Normalized expert outputs become substantially more similar, showing that expert redundancy is concentrated in output scale.Average similarity rises from 0.335 to 0.529 at layer 5 and from 0.251 to 0.471 at layer 17 after normalization.
- Design consequence: ExFold uses a shared per-pair scalar correction while allowing each phase to change only how retained experts are selected.This provides the paper’s affirmative answer to unified acceleration under distinct prefill and decode constraints.
3. Decode: batch-level folding
ExFold accelerates decode by selecting a constrained batch-level expert pool while folding excluded expert contributions into retained experts with calibrated scalar projectors. This shared mechanism preserves quality as the decode budget tightens and supports practical serving speedups.
- Batch-level folding: Omitted experts are not executed; their calibrated contributions are folded into retained targets and coalesced before expert computation.Unlike static merging, folding changes neither expert parameters nor the original router.
- Batch-level folding: Decode selects D dominant experts per batch and reuses the same scalar matrix, loss matrix, and folding operator as prefill.The phases differ only in retained-expert selection; excluded contributions are recovered through the shared folding mechanism.
- Cross-architecture quality: 97.2% and 98.4% of original average quality are retained on GLM-4.5-Air under prefill-only P4 and decode-only D64, respectively.Joint compression at P4+D32 retains 96.7%, extending the quality-preservation result beyond the primary model.
- Online decode efficiency: 2.45× TPOT speedup is reached at 8 QPS, and about 2.4× is sustained through 12 QPS by bounding the batch-level expert union.The result distinguishes decode acceleration from token-wise FLOP reduction, which loses benefit as QPS increases.
- Serving efficiency: 1.20× offline throughput improvement is reported, exceeding Direct Top-4 while adding little runtime overhead under large batches.The implementation uses a lightweight expert-folding CUDA kernel in vLLM.
- Quality under decode budgets: 1.64 additional quality-retention points separate D64 from the tighter expert-pool trade-off described by the quality frontier, while D128 is near-lossless.Decode-only D64 and D128 remain within 0.2 average points of Original in the reported evaluation.
Conclusion
ExFold unifies training-free acceleration of MoE prefill and decode by combining phase-specific retained-set selectors with a shared directed projector that folds excluded contributions into router metadata.
- ExFold jointly accelerates MoE prefill and decode through phase-specific retained-set selectors and a shared directed projector.The framework folds excluded expert contributions into router metadata rather than discarding them.
- Across MoE architectures, ExFold improves TTFT, TPOT, and serving throughput while preserving more quality than expert dropping.
- Because folding changes only router metadata, ExFold remains complementary to kernel, scheduling, and parallelism optimizations.
- Output recovery is presented as a practical basis for unified MoE inference acceleration.
Reproducibility Details
The evaluation covers released MoE checkpoints, multiple benchmark protocols, unlabeled calibration procedures, and specified vLLM serving configurations.
- Model architectures: The evaluated checkpoints and their architecture details are summarized in Table 7.The table reports checkpoint-exposed architectures, while shared experts are executed alongside routed Top-K experts.
- Benchmarks and metrics: The benchmark suite specifies case counts, generation samples, fixed subsets, checking protocols, and reported metrics across evaluated tasks.The protocols include MMLU-Pro, IFBench, and LiveCodeBench configurations.
- Calibration protocol: Calibration uses unlabeled sequences and retains fixed projectors across downstream tasks, with model-specific sequence counts and token limits.Primary Qwen3 and DeepSeek-V2-Lite use 32 sequences of at most 4,096 tokens; Qwen3.5 and GLM-4.5-Air use separate shard configurations.
- Calibration protocol: Calibration uses forward activations without task labels, gradients, or evaluator feedback, with source-output-norm weighting and λ = 10−3.Scalar coefficients are clipped to [−4, 4], self-pairs equal one, and unobserved pairs receive loss 1030.
- Quality evaluation: Quality evaluation uses an OpenCompass-compatible generation pipeline with task-specific scoring protocols and benchmark test-suite execution.
- System and serving configuration: Efficiency experiments use eight-GPU NVIDIA H800 servers and report TTFT, TPOT, and completed-request throughput under specified vLLM serving workloads.Prefill and decode use distinct tensor-parallelism, query-rate, request, prompt, and generation configurations.
Implementation
ExFold calibrates directed scalar projectors and loss-based target assignments, then applies phase-specific expert selection and remapping before fused MoE execution.
- Projector calibration: Calibration represents co-routed source and target expert outputs and compares scalar-sharing patterns, with pairwise sharing preserving directed expert-level differences.Pairwise scalar metadata can be used directly for target assignment through normalized reconstruction loss.
- Projector parameterization: The deployed scalar projector is chosen because its coefficient can be absorbed into router weights without an online vector transform.Diagonal and scalar-plus-low-rank alternatives require different online operations or state.
- Projector parameterization: The evaluated low-rank artifact retains at most 1,024 directed pairs per layer, while unsupported pairs use only the scalar term.Its basis and pair-specific correction require an online vector transform and substantial state, preventing reuse of the standard fused MoE path.
- Phase-specific selection: Prefill ranks routed experts by estimated contribution magnitude and retains Kpre experts per token, while decode selects a bounded expert pool from aggregate batch contribution.
- Target assignment: For any retained set, each omitted source expert selects the target with minimum calibrated projection loss.The same rule applies to the prefill set and either decode pool.
- Prefill implementation: Prefill folding coalesces omitted routes assigned to the same retained target without materializing omitted outputs, so the fused kernel executes only Kpre experts per token.
- Decode implementation: Decode remapping occurs before dispatch, ensuring every routed identifier belongs to a pool of at most D experts.Dynamic and static decode paths differ in pool construction and lookup caching, while preserving the original [B, K] layout.
Cross-Model Quality Results
The cross-model quality-results section reports joint-compression results for DeepSeek-V2-Lite-Chat and separates prefill-only, decode-only, and joint folding for Qwen3.5-35B-A3B.
- DeepSeek-V2-Lite-Chat: Table 10 reports DeepSeek-V2-Lite-Chat quality under joint compression.
- Qwen3.5-35B-A3B: Table 11 separates prefill-only, decode-only, and joint folding quality for Qwen3.5-35B-A3B.The method labels include P4, D32, and D64 retained-expert or decode-pool constraints.
Expert Geometry and Projector Structure
Across Qwen3 and GLM-4.5-Air, expert outputs show layer-dependent pair structure and substantial magnitude variation. Scalar alignment and directed projector diagnostics characterize how excluded expert outputs can be transferred to retained experts.
- Cross-model geometry: Qwen3 diagnostics cover all 48 layers and 128 routed experts, revealing layer-dependent similarity structure and expert-wise output-magnitude variation.The corresponding GLM-4.5-Air diagnostics cover 45 layers and use co-routed expert pairs, including unavailable pairs that were never jointly observed.
- Cross-model geometry: GLM-4.5-Air matrices show nonuniform co-routed pair structure and output-magnitude differences across its 45 MoE layers.Gray cells identify pairs without joint observations, separating missing support from observed pair geometry.
- Projector structure: Common-input projector diagnostics use source experts as rows and candidate targets as columns, with directed normalized Frobenius loss measuring scalar-aligned transfer quality.Every off-diagonal pair processes the same hidden-state matrix X, while self-pairs are suppressed in the visualized projector geometry.
- Projector structure: The median directed off-diagonal loss is 0.407, and 79.9% of pairs fall below 0.5, indicating many compatible scalar transfers.Lower loss denotes more compatible transfers, and the diagnostic avoids conflating absent pair support with projection quality.
DeepSeek-V4-Flash Extension Details
The DeepSeek-V4-Flash extension evaluates ExFold under a specified calibration and serving setup, including a confidence-aware safeguard for poorly calibrated transfers. Its speed results distinguish quality evaluation from a separate saturated-throughput boundary.
- Model and evaluation: DeepSeek-V4-Flash contains 284B total parameters, 13B activated parameters, 43 transformer layers, and 256 routed experts with Top-6 routing.Quality evaluation uses tensor parallelism eight, max_num_seqs=32, and CUDA Graph, with identical prompts and scoring conditions across methods.
- Calibration: Calibration uses 64 unlabeled inputs and at most 64 observer tokens per sequence, making the setup transductive and benchmark-aware rather than zero-contact held-out evaluation.The calibration mix includes general instruction, code, mathematics, IFEval, and IFBench inputs without labels or evaluator feedback.
- Safeguard: A confidence-aware P3 safeguard distributes residual omitted-expert weight across retained Top-3 routes instead of forcing a poorly calibrated pair to absorb it.P3 still invokes exactly three routed experts and adds no additional expert execution.
- Serving protocol: The speed protocol uses one H800 server, tensor parallelism four, CUDA Graph, max_num_seqs=32, and fixed prefill and decode request schedules across QPS points.Prefill uses 8,192 input tokens and one output token; decode uses one input token and 256 output tokens.
- Speed boundary: D64 stabilizes near 1.15x TPOT at QPS 8–32, while saturated max_num_seqs=128 yields a 1.286x output-throughput gain.The throughput result is reported separately because quality uses max_num_seqs=32, and the TTFT peak includes queueing amplification.
Token-Level Reconstructions
Token-level reconstruction compares reduced routing strategies in a proxy space, tracing expert contributions head-to-tail and comparing their approximate aggregates with the original aggregate. The diagrams illustrate routing geometry rather than aggregate hidden-state reconstruction quality.
- Token-Level Reconstructions: Figure 15 compares Original Top-8, Direct Top-4, MoDES, REAP, and ExFold Top-4 for tokens from four task categories.Arrow lengths combine router weight with calibrated expert-output norm, and contributions are accumulated in a strict head-to-tail chain.
- Token-Level Reconstructions: The diagrams use two-dimensional MDS of all-expert cosine distances, with route-rank labels rather than global expert identifiers.Four datasets target layers 0, 16, 32, and 47 using the nearest qualifying layer and first deterministic case.
- Aggregate comparison: Each chain endpoint represents the corresponding aggregate, while background arrows mark the original aggregate y and each method’s approximate aggregate ŷ.ExFold omits per-segment labels for clarity, and dashed arrows identify approximate aggregates.
- Interpretation limits: The displayed approximation with the smallest proxy distance is highlighted conceptually, but these distances are proxy-space quantities rather than hidden-state reconstruction errors.The panels are qualitative examples, not aggregate evaluation evidence.