PaperlayerView exact revision on arXiv

Source-linked AI summary

MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation

Lie Li, Wen Li, Junxiao Shen, Gusheng Hu

arXiv:2608.15299v1cs.LGcs.AI

TL;DR

Uniform expert counts overlook layer-wise redundancy in MoE Transformers. MAPLE reallocates experts using sensitivity-guided optimization and, across four models, improves performance under a 75% routed-expert budget, surpassing full-budget uniform configurations on three models.

  • Problem

    MoE Transformers typically allocate the same number of routed experts across layers despite substantial layer-wise differences in redundancy and sensitivity.

  • Method

    MAPLE probes layer-wise sensitivity, derives a closed-form heterogeneous allocation under a global expert budget, and refines it with sensitivity-guided genetic search.

  • Results

    Using 75% of routed experts, MAPLE achieves the best average accuracy on all four evaluated models and surpasses full-budget uniform configurations on three.

  • Takeaways & Limitations

    Layer-wise capacity redistribution can better match heterogeneous MoE layer demands than uniform allocation while using fewer routed experts.

  • Takeaways & Limitations

    Evaluation remains to be extended across diverse hardware platforms and broader real-world inference scenarios.

Abstract

from arXiv · show

Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior to guide exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency.

1 INTRODUCTION

MoE models expose layer-wise expert capacity, but existing uniform or costly allocation methods overlook substantial differences in layer redundancy. MAPLE provides a plug-and-play, sensitivity-guided framework that reallocates experts heterogeneously in pretrained MoE Transformers without weight updates or retraining, improving performance under the same budget.

  • Motivation: Layer-wise redundancy in MoE models varies substantially, making uniform expert allocation an underexplored opportunity for improving pretrained MoE LLMs.Evidence from expert pruning, layer compression, and parameter-efficient fine-tuning indicates that adaptation demands and redundancy differ across depth.
  • Limitations of Existing Methods: Existing layer-aware allocation methods can require calibration data, weight reconstruction, recovery procedures, or permanently removed experts.These costs limit plug-and-play deployment in pretrained models.
  • MAPLE Framework: MAPLE determines heterogeneous per-layer expert allocations in pretrained MoE Transformers without modifying weights or retraining.It formulates expert reallocation as constrained optimization under a global expert budget.
  • Optimization Strategy: MAPLE combines a lightweight layer-wise sensitivity scan, a closed-form allocation under a global expert budget, and genetic search for refinement.Sensitivity guides capacity toward layers whose performance responds more strongly to expert-count changes.
  • Results: 25% fewer routed experts still lets MAPLE surpass the original full-expert MoE on multiple reasoning tasks.Under the same expert budget, it also outperforms uniform, pruning-based, and expert-reallocation baselines across five reasoning benchmarks.

2 RELATED WORK

Prior work establishes that layer redundancy is heterogeneous, motivating sensitivity-based non-uniform compression. Related research also addresses MoE expert compression and budget-constrained layer allocation through pruning, approximate optimization, and evolutionary search.

  • Layer-wise Sensitivity Analysis: Layer redundancy is substantial but highly non-uniform across layers, motivating capacity allocation according to functional importance rather than uniform reduction.This sensitivity-based perspective underpins non-uniform model compression.
  • Mixture-of-Experts Models and Expert Compression: Sparse expert activation improves MoE parameter efficiency, while prior work reduces deployment cost by pruning experts with dense-model and router-aware methods.Examples include SparseGPT, Wanda, and MoE-Pruner.
  • Budget-Constrained Architecture Search: Allocating a fixed budget non-uniformly across layers is a discrete combinatorial problem whose search space grows rapidly with network depth, making exact optimization impractical for large models.Prior work therefore relies on approximate search strategies.
  • Budget-Constrained Architecture Search: Approximate allocation methods include reinforcement learning, iterative greedy adaptation, and evolutionary approaches for discrete architecture-search spaces.AMC assigns layer-wise compression ratios under a global FLOPs budget, while NetAdapt trades global optimality for search efficiency.

3 METHOD

MAPLE allocates routed experts heterogeneously across MoE layers by measuring layer-wise sensitivity, deriving a budget-constrained closed-form assignment, and refining it with a lightweight genetic search. It is plug-and-play at inference time because it changes only per-layer activated expert counts without modifying model weights, routers, or architecture.

  • Layer-wise sensitivity: MAPLE probes each layer independently by varying its routed-expert count while holding all other layers at the baseline k0.The probe evaluates expert counts e ∈ {1, …, k0} for each layer l.
  • Layer-wise sensitivity: Three lightweight measures capture performance-, parameter-, and representation-level sensitivity, with accuracy range used by default.The selected sensitivity metric is denoted s_l, and alternatives are evaluated in an ablation study.
  • Closed-form allocation: MAPLE derives an allocation that stays close to each layer’s preferred expert count while making more sensitive layers deviate less under the global budget K.The implementation uses positive sensitivity weights w_l = s_l, and surplus or deficit is assigned preferentially to low-sensitivity layers.
  • Closed-form allocation: The continuous allocation is clipped to [kmin, kmax] and converted with largest remainder into an integer assignment satisfying per-layer bounds and the global budget.The resulting allocation is denoted k_sens.
  • Genetic refinement: A lightweight genetic algorithm refines k_sens using validation accuracy as fitness and sensitivity-controlled Gaussian mutation, with smaller perturbations for high-sensitivity layers.Each generation retains the top two candidates as elites, uses no crossover, and returns the best candidate k_best.
  • Serving implementation: MAPLE integrates into MoE serving engines by changing only per-layer inference-time activated expert counts, without modifying routers, expert weights, or model architecture.The SGLang implementation adds a dedicated path for k = 1 layers that evaluates only the highest-scoring expert.

4 EXPERIMENTS

Across four MoE architectures and seven benchmarks, MAPLE evaluates heterogeneous expert allocation under a 75% routed-expert budget. It improves accuracy and efficiency, while ablations show that sensitivity-guided initialization and lightweight refinement are effective, stable, and transferable.

  • Experimental Setup: 75% of routed experts are retained across DeepSeek-MoE-16B, DeepSeek-V2-Lite, Moonlight-16B-A3B, and OLMoE-1B-7B, while shared experts remain preserved when present.Evaluation spans RTE, BoolQ, PIQA, ARC-E, ARC-C, GSM8K, and BBH.
  • Overall Performance: 2.63, 6.44, and 4.13 points are MAPLE’s average-accuracy gains over the full-budget baseline on DeepSeek-MoE-16B, DeepSeek-V2-Lite, and OLMoE-1B-7B, respectively.On Moonlight-16B-A3B, MAPLE is 0.62 points below baseline but remains the best method at this budget.
  • Long-form Generative Tasks: 53.0% lower total evaluation time is achieved on GSM8K while preserving accuracy within 0.61 points of the full-budget baseline.On BBH, MAPLE improves accuracy by 0.90 points while reducing total evaluation time at the same 75% budget.
  • Ablations and Refinement: +0.71 accuracy is obtained by accuracy range alone in just 2 generations, whereas adding output deviation causes -0.72 and omitting accuracy range causes -1.08.Refinement improves the closed-form allocation on every benchmark, with gains from +0.82 on PIQA to +7.01 on ARC-E.
  • Transferability and Budget Scaling: MAPLE’s advantage over Wanda and SparseGPT holds at every budget and widens at 50%, while reusable and cross-task allocations remain effective without target-task recalibration.Heterogeneous allocation also improves over the uniform baseline at 100% budget, indicating that better capacity placement can help independently of budget reduction.

5 CONCLUSIONS AND LIMITATIONS

MAPLE is a plug-and-play method for heterogeneous expert allocation that combines layer-wise sensitivity estimation, closed-form allocation, and lightweight genetic refinement without retraining or architectural changes. Future work will evaluate its practical benefits across broader hardware and inference settings, including system-level constraints affecting sparse MoE execution.

  • Conclusions: MAPLE allocates experts heterogeneously using layer-wise sensitivity, a closed-form solution, and lightweight genetic search without weight updates, retraining, or architectural modification.The framework targets budget-constrained MoE Transformers.
  • Conclusions: Experiments on four MoE models and five benchmarks show MAPLE outperforming pruning-based baselines under a 75% routed-expert budget.The supplied conclusion passage also states that MAPLE surpasses the full-budget uniform configuration, though the passage is truncated before reporting further details.
  • Limitations and Future Work: Future evaluations will test MAPLE across GPUs, CPUs, edge devices, and broader real-world inference scenarios.These studies are intended to assess practical efficiency, deployment flexibility, and applicability beyond the current experimental setting.
  • Limitations and Future Work: Hardware-aware analysis will examine whether reduced routed-expert computation produces actual latency, memory, and energy benefits across deployment environments.The passage frames these benefits as questions for future evaluation rather than established results.
  • Limitations and Future Work: Deployment-oriented studies will analyze interactions between heterogeneous allocation and memory bandwidth, batching behaviour, and hardware support for sparse MoE execution.These system-level factors may affect practical inference performance.

A APPENDIX

The appendix may include other additional sections.

  • The appendix may contain other additional sections.

A.1 FULL ALGORITHM

MAPLE is a three-step pipeline that models layer-wise sensitivity, derives a closed-form expert allocation, and refines it with a sensitivity-constrained genetic search. The algorithm probes candidate expert counts, evaluates allocations on a validation set, and projects solutions into the feasible budget range.

  • Step 1: Layer-wise Sensitivity Modelling: MAPLE first probes each layer by varying its expert count from 1 to k0 while fixing all other layers at k0.It records A_l(e) for each candidate count and can quantify sensitivity using the accuracy range s_l = max_e A_l(e) − min_e A_l(e).
  • Step 2: Sensitivity-guided Closed-form Expert Allocation: The closed-form allocation directs surplus to less-sensitive layers, removes excess from less-sensitive layers, and produces integer counts after clipping to [kmin, kmax].The integer projection uses the largest-remainder method to obtain k_sens.
  • Step 3: Sensitivity-constrained Genetic Refinement: The genetic refinement initializes a population around k_sens by adding Gaussian perturbations to each layer’s allocation.Candidates are evaluated on D_val using fitness f(k) = A(k), while perturbation scale is sensitivity-controlled.
  • Step 3: Sensitivity-constrained Genetic Refinement: Each generation retains the top-2 candidates as elites, perturbs elite solutions, and projects the resulting allocations back to the feasible set.The algorithm’s inputs include the MoE model, base expert count k0, total budget K, valid range [kmin, kmax], validation set D_val, population size P, and generations G.
Loading 2608.15299v1…