PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang

arXiv:2609.05228v1cs.AI

TL;DR

Fixed top-k routing executes the same number of expert slots for every token, although routed experts can contribute unequally. ACE provides training-free, calibration-free token-adaptive skipping, and at 50% skipping achieves the best average accuracy across three MoE models while reducing Qwen3.6 WikiText-2 perplexity by 7.96% versus the strongest competitor.

  • Problem

    Fixed top-k routing executes every token’s same number of expert slots, despite variation in expert contributions and resulting redundant computation.

  • Method

    ACE combines offline global transformation-capacity and router-conditioned directional-response estimates to conservatively skip low-contribution expert slots while retaining the top-1 expert.

  • Results

    At 50% skipping, ACE achieves the best average accuracy across three MoE models and reduces Qwen3.6 WikiText-2 perplexity by 7.96% versus the strongest competitor.

  • Takeaways & Limitations

    ACE is particularly effective under aggressive skipping, where router preference alone cannot reliably distinguish structurally weak or directionally specialized experts.

  • Takeaways & Limitations

    The calibration-free RCR formulation assumes a Gaussian layer-input distribution and fixes the input-scale prior at 1.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing activates the same number of expert slots for every token, causing substantial redundant computation. Existing expert-skipping methods often rely on router confidence, calibration data, or additional training, and therefore cannot reliably estimate the actual contribution of routed experts. To this end, we propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. ACE contains two complementary components: 1) Global Spectral Proxy (GSP), which estimates global transformation capacity from the coupled gate, up, and down projections together with RMSNorm scaling; and 2) Router-Conditioned Refinement (RCR), which constructs expert-specific direction prototypes from centered router weights and evaluates expert responses along routing-preferred directions. During inference, ACE combines both estimates with runtime router gates and skips an expert slot only when both views identify it as low-contribution, while always retaining the top-1 expert. All expert statistics are computed offline, leaving only table lookups and lightweight scalar operations online. Extensive experiments across three MoE-based LLMs and eight benchmarks demonstrate that ACE consistently outperforms existing static and dynamic baselines, with increasingly pronounced advantages under aggressive expert skipping. For instance, at a 50% skipping ratio on Qwen3.6-35B-A3B, ACE reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method.

1 Introduction

MoE models scale capacity through sparse expert activation, but fixed top-k routing still executes redundant expert slots because routing preference does not directly measure expert contribution. ACE addresses this with training-free, calibration-free, checkpoint-preserving skipping that combines global structural and routing-direction estimates for token-adaptive decisions.

  • MoE scaling: MoE replaces each dense FFN with multiple experts, routing each token to a small subset to expand capacity while controlling per-token computation.Qwen3.6-35B-A3B activates about 3B of 35B parameters, while DeepSeek-V4-Pro and DeepSeek-V4-Flash use 1.6T/49B and 284B/13B total-to-active configurations.
  • Motivation: Fixed top-k routing activates the same number of expert slots for every token despite variation in expert contributions, leaving low-contribution computation.Offline compression reduces the expert pool but typically retains fixed top-k execution, while dynamic methods may modify routers, add decision modules, or use post-training thresholds.
  • Challenges: Calibration-free skipping must distinguish routing preference from contribution while accounting for coupled expert structure and direction-dependent specialization.Router gates express relative assignment preference among top-k candidates, whereas experts with similar gate values can produce different responses because of their FFN structures and normalization scaling.
  • ACE: ACE combines Global Spectral Proxy and Router-Conditioned Refinement estimates with the pretrained router and original top-k candidates for safeguarded, token-adaptive expert skipping.GSP uses gate, up, and down projections with preceding RMSNorm scaling; RCR constructs expert-specific direction prototypes from centered router weights and measures responses along preferred routing directions.

2 Related Work

Prior work compresses MoE experts statically before deployment or skips them dynamically according to token-, layer-, or batch-specific states. ACE combines offline expert-capacity estimates with token-wise router gates to identify and skip low-contribution expert slots during inference.

  • Static Expert Compression in MoE-based LLMs: Static expert compression prunes, merges, or restructures experts before deployment to reduce model size and resident memory.Existing methods identify redundant experts using routing frequency, task statistics, or structural importance.
  • ACE: ACE combines Global Spectral Proxy and Router-Conditioned Refinement scores with token-wise router gates to skip low-contribution expert slots.GSP estimates transformation capacity from frozen SwiGLU weights, while RCR evaluates responses along router-induced directions.
  • Dynamic Expert Skipping in MoE-based LLMs: Dynamic expert skipping adapts expert activations to token-, layer-, or batch-specific states and can replace fixed top-k routing with variable-cardinality policies.Top-P Routing uses cumulative routing probabilities, Ada-K Routing learns an allocator through reinforcement learning, and DynMoE combines Top-Any gating with adaptive training.

3 Method

ACE estimates token-adaptive expert contribution without calibration data by combining a global, parameter-derived capacity proxy with router-conditioned directional refinement. It skips an expert slot only when both estimates identify low contribution, while retaining the top-1 expert.

  • Method overview: ACE combines Global Spectral Proxy and Router-Conditioned Refinement to estimate expert contribution without calibration data or additional expert computation.GSP captures global transformation capacity, while RCR addresses directional specialization using router-derived directions.
  • Inference procedure: At inference, ACE uses offline static scores, runtime gates, and lightweight normalization to skip slots identified as low-contribution by both GSP and RCR.The top-1 expert is always retained, and online GSP requires only table lookup, scalar multiplication, and normalization over the original top-k candidates.
  • Global Spectral Proxy: GSP combines runtime routing preference with static transformation capacity derived from the coupled gate, up, and down projections and RMSNorm scaling.This reflects the multiplicative structure of SwiGLU experts rather than relying on any single projection norm.
  • Global Spectral Proxy: GSP aggregates two complementary factorization paths with a geometric mean, then normalizes each expert’s score by the layer-wise mean.The geometric mean limits dominance by an exceptionally large single path and yields scores relative to the layer average.
  • Router-Conditioned Refinement: RCR centers router weight vectors within each layer to remove shared translation and extract expert-specific routing directions without real activations.RMS normalization preserves relative routing directions while avoiding redundant encoding of routing confidence already represented by runtime gates.

4 Experiments

Experiments across three MoE models and eight benchmarks show that ACE preserves quality increasingly well as expert skipping becomes aggressive, while its complementary estimators and lightweight inference support practical deployment. Ablations and efficiency measurements validate the conservative fusion rule and low threshold-construction cost.

  • Evaluation setup: ACE is evaluated on three MoE models, WikiText-2 perplexity, and seven zero-shot downstream tasks using a shared deterministic BF16 protocol.Evaluations use identical prompts and splits, greedy decoding, and seed 42.
  • Main results: As skipping increases, ACE increasingly outperforms router-only and single-view criteria, leading accuracy from 20% onward on Qwen3.6 and achieving the lowest PPL throughout.At 50%, ACE improves Qwen3.6 from MoDES’s 9.42/71.42 to 8.67/75.57 (PPL/Acc.).
  • Main results: ACE remains consistently strong on both perplexity and downstream accuracy, reflecting complementary quality measures and conservative protection against one-sided expert removal.The results identify divergence between language-modeling perplexity and reasoning or coding accuracy, particularly at aggressive budgets.
  • Component ablation: At 50% skipping, combining GSP and RCR reduces PPL from 8.99 to 8.85 and improves average accuracy from 74.10 to 74.30 over GSP.GSP is the stronger individual signal, while RCR alone degrades more rapidly.
  • Fusion ablation: Max performs best because it skips a slot only when both views deem it dispensable, preserving complementary evidence without an additional fusion hyperparameter.Min is overly aggressive, whereas averaging can suppress a strong signal from one view.
  • Efficiency: A single quantile pass constructs a threshold in 1.8 minutes, versus 16.5 minutes for binary search and 18.9 minutes for MoDES frontier search.ACE uses expert-indexed lookups and top-k scalar operations online; the 9.2–10.5× reduction concerns deployment preparation.

5 Conclusion … 16 Add AGSP

ACE is a training-free, checkpoint-preserving framework that combines global and router-conditioned expert contribution estimates for conservative, token-adaptive skipping. Its offline statistics enable lightweight inference-time decisions, while experiments show particularly strong results under aggressive skipping.

  • 5 Conclusion: ACE combines GSP’s global transformation-capacity estimate with RCR’s router-conditioned directional response, skipping a slot only when both views identify low contribution.GSP uses coupled SwiGLU projections and RMSNorm scaling, while RCR evaluates experts along directions derived from centered router weights.
  • 5 Conclusion: At 50% skipping, ACE achieves the best average accuracy on all three evaluated MoE models and reduces Qwen3.6 WikiText-2 perplexity by 7.96% versus the strongest competitor.The conclusion also reports accelerated prefill and decoding under aggressive budgets.
  • A.1 Detailed Algorithms of ACE; Appendix Overview; A Appendix: ACE computes expert-level GSP and RCR statistics once before deployment, then uses expert-indexed table lookups and lightweight scalar operations during inference.This workflow separates offline parameter analysis from online token-adaptive skipping and preserves the pretrained model parameters.
  • Global Spectral Proxy; 15 AGSP: GSP estimates each expert’s intrinsic global transformation capacity from frozen gate, up, and down projections together with RMSNorm scaling, without calibration samples or input activations.The resulting normalized scores are stored as a static lookup table reused for every token.
  • Router-Conditioned Refinement: RCR centers each expert’s router weight, RMS-normalizes the resulting direction prototype, and measures the corresponding frozen expert’s output-to-input norm ratio offline.ACE sets ρ_l = 1 for every layer to retain calibration-free prototype scaling.
  • Online Conservative Expert Skipping: During inference, ACE retains the router’s original top-k candidates, combines their gates with offline GSP and RCR scores, and normalizes both score sets within that candidate set.The dual-view procedure produces contribution estimates before expert execution.
  • Online Conservative Expert Skipping; 16 Add AGSP: ACE skips an expert only when both views fall below the threshold, always retains the original top-1 expert, restores candidates if needed to satisfy m_min, and renormalizes retained gates.This conservative rule uses the maximum of the two normalized probabilities as the final contribution score.
  • 16 Add AGSP: Because decisions precede execution, only the final active expert set performs high-dimensional SwiGLU computation, while ACE adds table lookup, scalar operations, normalization, and small-set comparisons.The method therefore preserves the pretrained router and expert parameters with negligible control overhead relative to expert forward computation.

A.2 Detailed Proofs · 16 ARCR

The detailed-proofs material introduces theoretical support for ACE and presents Algorithm 3, RCR, for constructing a router-conditioned amplification table across MoE layers and experts.

  • A.2 Detailed Proofs: The subsection states that it provides theoretical results supporting ACE’s design and begins deriving an output-response upper bound.The supplied passage ends before the bound is completed.
  • A.2 Detailed Proofs: Algorithm 3 is identified as RCR, or Router-Conditioned Refinement.The algorithm is presented within the detailed-proofs subsection.
  • A.2 Detailed Proofs: RCR takes a pretrained MoE-based LLM M with L MoE layers as input.The supplied algorithm excerpt specifies the model and layer count symbolically.
  • A.2 Detailed Proofs: The algorithm outputs a router-conditioned amplification table and uses a numerical constant ϵ.The excerpt associates the output with the model’s MoE layers.
  • A.2 Detailed Proofs: RCR iterates over MoE layers indexed l = 1, . . . , L and denotes each layer’s expert count by N_l.The algorithm’s loop and notation define the layer-wise processing scope.
  • A.2 Detailed Proofs: For each layer, RCR denotes the router input dimension by d and extracts router weight vectors {w_l,e} for the experts.The excerpt establishes the router-related inputs used by the procedure.
  • A.2 Detailed Proofs: The procedure then loops over experts e = 1, . . . , N_l during its computation.A later expert loop is explicitly shown in the supplied algorithm excerpt.

17 Add ARCR … 14 SACE

This section establishes ACE’s theoretical foundations and conservative skipping criterion, then specifies the online algorithm and its SACE-based active-expert selection. The presented material covers GSP, RCR, ARCR, ZGSP, ZRCR, SGSP, SRCR, and SACE components.

  • Output-Response Bound of a SwiGLU Expert: ACE derives a SwiGLU expert output-response bound motivating the branch-symmetric Global Spectral Proxy construction.The derivation considers RMSNorm-scaled inputs and ignores bias terms.
  • 5 ZRCR: ACE’s theory establishes router-weight shared-translation invariance and a Gaussian-approximation conditional mean shift under exponential tilting for RCR.These results provide the theoretical basis for Router-Conditioned Refinement.
  • 17 Add ARCR: The framework establishes conservativeness of the dual-view skipping criterion and analyzes the output perturbation caused by expert skipping.This supports combining the GSP and RCR views when deciding whether to skip experts.
  • Output-Response Bound of a SwiGLU Expert: The SiLU activation satisfies a scalar inequality used in the SwiGLU output-response analysis.The supplied passage states the activation property without reproducing the inequality.
  • Output-Response Bound of a SwiGLU Expert: ACE’s online procedure takes layer and token indices, routed experts, ARCR, threshold τq, minimum active count mmin, and numerical constant ϵ as inputs.It outputs an active expert-slot set Il,t and renormalized gates.
  • 5 ZRCR: The algorithm iterates over each routed expert slot to evaluate expert-specific skipping decisions.The supplied pseudocode begins this loop over slots i = 1, . . . , k.
  • 14 SACE: SACE forms the active set as Il,t ← {1, . . . , k} \ SACE_l,t before checking whether fewer than mmin experts remain.These operations implement the SACE selection stage.
  • 14 SACE: When the active set is too small, ACE sorts skipped indices in SACE_l,t,i in descending order for subsequent handling.The supplied pseudocode specifies sorting after the |Il,t| < mmin condition.

19 Zgate · Shared-Translation Invariance of Router Weights · Conditional Mean Shift Under Gaussian Exponential Tilting

The section establishes that GSP must jointly capture the gate, up, and down branches, while centered router weights provide translation-invariant expert directions. Under Gaussian exponential tilting, these centered directions approximate the conditional mean-shift directions used by RCR.

  • 19 Zgate: GSP preserves the SwiGLU expert’s joint gate, up, and down projection dependence through two complementary factorizations.A proxy based on only one projection matrix cannot fully capture the expert’s structural response capacity.
  • Shared-Translation Invariance of Router Weights: Softmax routing probabilities remain unchanged when the same vector is added to every router weight, or the same scalar to every router bias.The shared additive contribution cancels across the softmax logits.
  • Shared-Translation Invariance of Router Weights: Centering removes the non-identifiable shared router component and retains each expert’s relative discriminative direction within its layer.This makes centered router weights more appropriate than uncentered weights for representing expert-specific routing preference.
  • Conditional Mean Shift Under Gaussian Exponential Tilting: Under a Gaussian hidden-state approximation, exponential tilting by a router direction produces another Gaussian distribution.The derivation completes the square and absorbs x-independent terms into the normalization constant.
  • Conditional Mean Shift Under Gaussian Exponential Tilting: The tilted distribution’s conditional mean shifts from the original mean according to the hidden-state covariance applied to the tilting direction.This follows from the Gaussian mean expression and its difference from the original hidden-state distribution.
  • Conditional Mean Shift Under Gaussian Exponential Tilting: When hidden-state covariance is approximately isotropic, the conditional mean-shift direction aligns approximately with the centered router weight.The isotropic assumption reduces the covariance-weighted direction to a scalar multiple of the centered router direction.
  • Conditional Mean Shift Under Gaussian Exponential Tilting: RCR uses the RMS-normalized centered router weight as a data-free approximation to the characteristic input direction associated with each expert.This construction follows the Gaussian conditional mean-shift interpretation.

Conservativeness and Output Perturbation of Dual-View Skipping

ACE’s dual-view criterion is conservative because it skips only experts identified as low-contribution by both GSP and RCR, while safeguards further restrict skipping. Its output perturbation depends on skipped router mass and the discrepancy between skipped and retained experts’ average responses, with a worst-case bound no larger than either single-view rule.

  • Candidate-set intersection: ACE’s initial skipping set equals the intersection of the low-contribution candidate sets identified independently by GSP and RCR.The maximum-based criterion is equivalent to requiring both view-specific scores to fall below the threshold.
  • Conservativeness: An expert considered important by either GSP or RCR cannot be skipped by ACE’s dual-view criterion.Top-1 retention and minimum-active-expert safeguards can only remove additional indices from the initial skipping set.
  • Skipped router-mass comparison: Under the same threshold, ACE removes no more router probability mass than either individual-view rule.This follows because the ACE skipping set is a subset of either single-view skipping set when router gates are nonnegative.
  • Output perturbation after gate renormalization: Output perturbation is jointly determined by skipped router mass α and the discrepancy between gate-weighted mean responses of skipped and retained experts.Gate renormalization changes retained gates to gi/(1 − α), and the perturbation vanishes when α = 0.
  • Conservative perturbation bound: ACE’s worst-case output-perturbation bound is no larger than that induced by either single-view candidate set under the same threshold.Dual-view intersection, top-1 retention, and the minimum-active-expert constraint jointly safeguard against excessive perturbation while preserving token-adaptive sparsification.

B Additional Experimental Details · B.1 Budget-to-Threshold Mapping

ACE separates score computation from budget control: one unlabeled score-collection pass maps requested execution budgets to thresholds, while preserving frozen GSP and RCR statistics. The resulting realized skipping ratios and tightly clustered thresholds support reuse across workloads.

  • B.1 Budget-to-Threshold Mapping: ACE estimates expert importance exclusively from frozen model parameters, without calibration samples, evaluation inputs, labels, losses, or parameter updates.
  • B.1 Budget-to-Threshold Mapping: A separate unlabeled score-collection pass converts each requested execution budget into the numerical scale required by the data-free scores.
  • B.1 Budget-to-Threshold Mapping: One collected score distribution yields the complete threshold table for all budgets, with tied boundaries assigned the closest realizable count.
  • B.1 Budget-to-Threshold Mapping: Budget control does not alter the GSP or RCR lookup tables and accounts for safeguarded slots and boundary ties.
  • B.1 Budget-to-Threshold Mapping: The paper reports measured skipped routed-slot fractions, so the 10–60% operating points are realized rather than merely requested ratios.
  • B.1 Budget-to-Threshold Mapping: For a fixed model and realized skipping ratio, thresholds remain tightly clustered across datasets despite differing domains and output formats.
  • B.1 Budget-to-Threshold Mapping: Threshold scale is primarily determined by the pretrained model and requested execution budget rather than by a specific benchmark.
  • B.1 Budget-to-Threshold Mapping: A threshold obtained from one unlabeled workload can consequently be reused on another workload.

B.2 Complete Task-Level Results

Tables 9–26 provide the complete task-level results underlying the main-paper budget curves across three MoE models and 10%–60% expert-skipping ratios. They report task accuracies, Avg., and WikiText-2 perplexity, with pruning methods ranked against an unpruned BF16 reference.

  • Coverage: Tables 9–26 cover Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, and Gemma-4-26B-A4B-it across 10%–60% expert skipping.The tables are ordered by model and then by target expert-skipping ratio.
  • Metrics: Task metrics and Avg. are accuracies (%), while PPL is evaluated on WikiText-2.These metrics constitute the task-level results underlying the budget curves in the main paper.
  • Evaluation protocol: At each model and skipping budget, BF16 is an unpruned reference and is excluded from ranking.Ranking applies to the pruning methods rather than the unpruned reference.
  • Evaluation protocol: Boldface identifies the best result among all pruning methods for each metric, retaining ties.This convention applies throughout Tables 9–26.
Loading 2609.05228v1…