PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Zachary Broveak, Christian Kubicka, Thierry Tambe

arXiv:2610.19232v1cs.LG

TL;DR

Dense-to-MoE conversion reduces active computation, while existing approaches use uniform per-layer allocation or a floating utilization budget. Spectral-MoE learns heterogeneous allocation under a fixed budget, and its allocators improve performance over uniform across analytic and learning-based conversion methods.

  • Problem

    Dense-to-MoE methods use uniform per-layer allocation or a floating utilization budget, leaving heterogeneous allocation under a fixed budget as an unexplored optimization axis.

    Source: §1 · pp. 1–2

  • Method

    Spectral-MoE learns per-layer allocation under a fixed utilization budget, using either token-shared per-layer scalars or dynamic per-token schedules.

    Source: §1 · p. 2

  • Results

    At 25% utilization, Spectral-MoE achieves SOTA mean accuracy and outperforms uniform allocation across budgets on analytic and learning-based conversion bases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Per-layer expert utilization adds a third optimization axis to dense-to-MoE conversion alongside expert partitioning and routing.

    Source: §1 · p. 2

    Sustained expert-assignment turnover may leave routers partially adapted at joint-training end; stabilizing partition learning remains future work.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.

1 Introduction

Dense-to-MoE conversion reduces active computation, but existing approaches use uniform per-layer allocation or a floating utilization budget. Spectral-MoE learns heterogeneous per-layer allocation under a fixed budget, with static and dynamic allocators that improve performance over uniform allocation.

1 Introduction

  • At 25% utilization, Spectral-MoE achieves SOTA mean accuracy and outperforms uniform allocation across budgets on analytic and learning-based conversion bases.Figure 1 presents benchmark accuracy versus utilization and shows the learned dynamic allocation extending the dense-to-MoE performance frontier.Source: Figure 1 · p. 1, §1 · p. 2
  • Dense-to-MoE conversion groups pretrained FFN neurons into experts and routes each token to a subset, reducing active computation.The fraction of experts active for a token is utilization.Source: §1 · p. 1
  • Existing dense-to-MoE methods use uniform per-layer allocation or a floating utilization budget, motivating investigation of heterogeneous allocation.Layer-dependent sensitivity in quantization and pruning motivates this potential optimization axis.Source: §1 · p. 2
  • Spectral-MoE learns per-layer allocation under a fixed utilization budget, adding a third optimization axis alongside expert partitioning and routing.Its static allocator shares one learned scalar per layer across tokens; its dynamic allocator learns a per-token schedule.Source: §1 · p. 2

2 Background and Related Work

Prior heterogeneous-allocation methods use sensitivity-based surrogates or vary utilization across tokens through routing and budget redistribution. Spectral-MoE learns per-layer utilization directly and applies it to analytic and gradient-based conversion methods.

2.1 Heterogeneous Allocation

  • Alloc-MoE’s Alloc-L and LExI optimize schedules using sums of per-layer sensitivity estimates, while Spectral-MoE learns per-layer utilization directly through gradients.Alloc-L uses dynamic programming over profiled sensitivities; LExI uses evolutionary search over output-perturbation estimates.Source: §2.1 · p. 2
  • Prior per-token methods vary utilization through confidence-based routing, null experts, or redistribution, whereas Spectral-MoE maps hidden states to schedules conserving a cross-layer budget.The cited redistribution methods include SeqTopK and Alloc-MoE’s Alloc-T.Source: §2.1 · p. 2

2.2 Dense-to-MoE Conversion

  • Analytic methods quickly construct neuron-to-expert partitions from heuristics; Spectral-MoE’s analytic base uses CMoE partitions and routers, then trains the allocator with both frozen.CMoE uses balanced clustering of activation patterns and a representative neuron from each cluster to construct the router.Source: §2.2 · p. 2
  • Learning methods use gradient-based optimization; Spectral-MoE learns per-layer utilization alongside DOT-MoE’s joint partition-and-router optimization while preserving each token’s total budget.DOT-MoE uses differentiable relaxations to learn the neuron-to-expert partition and token-to-expert router.Source: §2.2 · p. 2
  • In the dynamic learned-base architecture, hidden states revise token schedules, layers commit budget-constrained expert counts, and STE gradients train the allocator, routers, and partition affinities.The final layer consumes the remaining budget; gradients pass through selections and budget commitments.Source: Figure 2 · p. 3

3 Methods

Spectral-MoE learns per-layer expert allocation under a fixed utilization budget, using straight-through estimators and budget-preserving allocation. Its static allocator shares a learned count across tokens; its dynamic allocator revises per-token schedules from hidden states.

3 Methods

  • Spectral-MoE learns per-layer allocation under a user-targeted utilization budget, developing budget-preserving allocation and straight-through estimators for static and dynamic schedules.The method first develops these components for static allocation, then extends them to dynamic allocation.Source: §3 · p. 3

3.1 Utilization Budget

  • The user target determines a uniform initial routed count and total budget B; both allocators preserve sum El = B per token after reserving shared and minimum expert activations.The re-distributable budget is the total routed budget minus the minimum routed counts reserved across layers.Source: §3.1 · p. 3

3.2 Static Per-Layer Allocation

  • Static allocation learns a continuous routed count per layer, shared across tokens, then rounds counts to integers while repairing bounds and preserving B.Each router selects its top-El experts, so expert identities may still vary by token under the shared schedule.Source: §3.2 · p. 4
  • Soft inclusion adds count gradients to the inherited router estimator; the forward pass uses hard selections while router and count gradients flow through separate soft paths.The learned base jointly trains partition affinities, router weights, and the allocator; the analytic base trains the allocator after recovering weights with LoRA and freezing the other components.Source: §3.2 · p. 4

3.3 Dynamic Per-Layer Allocation

  • The dynamic allocator revises each token’s remaining schedule using current hidden states; the final layer consumes the remainder, preserving the total routed budget.A shared projection feeds an initial output head and subsequent revision heads.Source: §3.3 · pp. 4–5
  • Dynamic schedule logits distribute surplus among remaining layers, while feasibility bounds keep the remaining routed budget within later layers’ count limits.The current layer receives its softmax share of surplus after reserving minimum counts for uncommitted layers.Source: §3.3 · p. 5
  • Dynamic soft-inclusion gradients couple schedule shares and remaining budget; straight-through estimation preserves gradient paths through earlier integer commitments.Increasing a later layer’s logit reduces the current layer’s surplus share, and increasing committed budget reduces the current proposal.Source: §3.3 · p. 5

4 Results

Across analytic and learned dense-to-MoE bases, Spectral-MoE’s allocators improve benchmark performance over uniform allocation. The experiments examine performance across utilization budgets, learned layer and token schedules, and training stability.

4.1 Experimental Settings

  • Experiments use three model families and evaluate WikiText-2 perplexity plus six downstream benchmarks, with separate analytic and learned conversion setups.For the analytic base, the partition and router are frozen while the allocator is trained; on the learned base, partition affinities, router weights, and allocator parameters are trained together with pretrained weights frozen.Source: §4.1 · pp. 6–7

4.2 Benchmark Performance

  • At 25% utilization, dynamic allocation reduces perplexity and improves six-task mean accuracy over uniform on both analytic and learned bases.On the analytic base, it also exceeds published CMoE mean accuracy by 3.0 points on Llama-2-7B and Llama-3-8B; on the learned base, it exceeds published DOT-MoE on those models and matches Qwen2.5-7B at 63.3%.Source: §4.2 · p. 7
  • Across learned-base models, dynamic allocation improves perplexity and six-task mean accuracy over uniform at 12.5%, 25%, and 50% utilization.Mean-accuracy gains average 1.5, 4.2, and 3.6 percentage points at those budgets, respectively; perplexity falls by 1.06–2.46, 1.04–2.96, and 0.91–2.55 points.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3 Learned Heterogeneity

  • On learned Llama-2-7B at 25% utilization, dynamic allocation finishes at 10.57 perplexity, versus 11.61 for uniform and 11.50 for static allocation.The dynamic schedule develops stabilized heterogeneous layer averages, while sampled tokens—including repeated instances of the same token ID—receive different schedules under the same total budget.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 Training Stability Considerations

  • Neuron-to-expert turnover averages 8.92% over the final 500 training steps and drops to 0.33% at the final step.The late drop is consistent with learning-rate decay toward zero; sustained turnover may leave the router partially adapted, motivating further work on stabilizing partition learning.Source: §4.4 · pp. 8–9, Figure 4 · p. 9
  • Ridge-regression router refitting improves overlap and coverage for both checkpoints, but six-task mean accuracy changes by +1.0 and −0.4 percentage points for seeds A and B.The authors leave exploration of router-refitting approaches with minimal token overhead to future work.Source: §4.4 · p. 9, Table 3 · p. 9

5 Conclusion

Spectral-MoE learns per-layer utilization under a user-specified budget, with static schedules shared across tokens and dynamic schedules that vary by token. The conclusion reports gains across conversion bases and models, while noting future work on stabilizing expert assignments.

5 Conclusion

  • Dynamic allocation lowers perplexity on both conversion bases across all three models at nominal 25% utilization, with learned-base gains extending to 12.5% and 50%.It also exceeds published DOT-MoE mean accuracy on Llama-2-7B and Llama-3-8B and matches it on Qwen2.5-7B.Source: §5 · p. 9
  • Static allocation shares a learned schedule across tokens, while dynamic allocation revises each token’s schedule and preserves the same total routed budget.The learned schedules combine stable layer averages with context-dependent variation.Source: §5 · p. 9

AI Use Statement

  • The authors report that generative AI helped implement a manually written software specification and reword, condense, format, and identify material for manual review.They state that generated artifacts were audited and that they take responsibility for the paper’s content.Source: AI Use Statement · p. 10

Reproducibility Statement

  • Training parameters and evaluation procedures are provided in Appendix C, and the project repository and implementation release status are available online.Source: Reproducibility Statement · p. 10

A Notation

  • The notation distinguishes routed counts from FFN utilization, which includes the shared expert, and uses bold italic and bold upright symbols for vectors and matrices, respectively.Source: Table 4 · p. 13

B Methods Implementation Details

The appendix specifies how requested utilization maps to realizable budgets, how static counts remain bounded and budget-preserving, and how final WikiText-2 perplexity is evaluated.

B.1 Real-Model Utilization Numbers

  • Requested utilization is rounded to a realizable multiple of 1/Et; compared allocation methods hold the routed budget fixed and activate the same total number of FFN neurons per token.Llama-3-8B realizes all three reported tiers exactly, while the other two models have remaining tiers within 0.6 percentage points of the requested utilization.Source: §B.1 · pp. 13–14, Table 5 · p. 14

B.2 Enforcing Bounds

  • Static count updates are centered to keep the gradient tangent to the budget constraint, then projected back to restore the sum after optimizer updates.Violating coordinates are clamped and projection is repeated over the remaining free coordinates to enforce count bounds.Source: §B.2 · p. 14
  • Integer schedules restore the routed budget after rounding and clipping, while continuous counts allow sub-integer updates to accumulate before the hard schedule changes.Source: §B.2 · p. 14

C.1 Perplexity

  • WikiText-2 test perplexity uses each model’s tokenizer and complete, nonoverlapping 2,048-token blocks; trailing partial blocks are discarded and final perplexity is measured at end of training.All reported perplexities use the WikiText-2 test split.Source: §C.1 · pp. 14–15

C.2 Benchmark Evaluation

Benchmark scores use common settings and equal weighting across six tasks, while training configurations distinguish learned-base optimization from the analytic-base conversion and allocator training.

C.2 Benchmark Evaluation

  • The six-task mean weights tasks equally, cross-model gains average model-specific gains equally, and evaluation settings are identical across models and allocation methods.Training and calibration use Dolmino, while evaluation uses WikiText-2 test and the listed benchmarks.Source: §C.2 · p. 15, Table 6 · p. 15

C.3 Training Configurations

  • Each learned-base run processes 458,752,000 Dolmino tokens using bfloat16 on four H100 GPUs, and allocator parameters use zero weight decay.The runs in Table 1 use initialization seed B; partition affinities and router weights use a peak learning rate of 5 × 10^-4 and weight decay 10^-4.Source: §C.3 · p. 15
  • The analytic base uses CMoE’s published partition and router construction, followed by LoRA weight recovery and allocator training with weights, partition, and router frozen.The allocator trains for 1,024 steps over the same token budget; the analytic uniform baseline is the recovered checkpoint before allocator training.Source: §C.3 · p. 15

C.4 Existing Method Implementations

The implementation details describe a learned-base reimplementation of DOT-MoE and the analytic-base training configuration.

C.4 Existing Method Implementations

  • The learned base reimplements DOT-MoE because public code was unavailable at the time of writing, adding a shared expert for training stability.Source: §C.4 · pp. 15–16

C.3 Training Configurations

  • For the analytic method, the static rate is set from the step count, spreading 0.875 times the uniform routed count over half the run.Source: Table 7 · p. 16

D Run-to-Run Variation

Because dense-to-MoE conversion is seed-sensitive, the authors report limited seed variation and match initialization seeds across allocation arms to control differences within comparisons.

D Run-to-Run Variation

  • Estimating seed variation across models, budgets, and allocation methods requires many repeated experiments, whose computational cost is prohibitive.The authors ran a small number of seeds to provide an indication of variation.Source: D Run-to-Run Variation · p. 16
  • Matched initialization seeds across uniform, static, and dynamic arms control seed differences within each comparison.The reported variation statistics cover only a small number of seeds for learned Llama-2-7B at 25% FFN utilization.Source: D Run-to-Run Variation · p. 16, Table 8 · p. 16
  • The authors report that downstream accuracy can differ significantly between runs, despite the limited seed sample.Table 8 reports representative statistics from the learned Llama-2-7B base training run at 25% FFN utilization.Source: D Run-to-Run Variation · p. 16, Table 8 · p. 16

E.1 Detailed Experiment Information

The detailed experiment uses learned Llama-2-7B conversion at 25% target utilization to compare allocation schedules and perplexity; across three models, both allocators favor late layers.

E.1 Detailed Experiment Information

  • Figure 3 evaluates learned Llama-2-7B at 25% target utilization, with seed B and 3,500 steps, differing only in allocation.The run processes 458,752,000 tokens.Source: §E.1 · p. 16
  • Figure 3 tracks WikiText-2 test perplexity every 10M tokens across 46 evaluations after initialization.Panel (b) applies a centered 61-step rolling median to per-step mean counts, while panel (c) shows 16 final-step schedules.Source: §E.1 · p. 16

E.2 Per-Layer Profiles Across Models

  • Across all three models, static and dynamic allocation assign more capacity to late layers.Dynamic profiles are per-layer means over the 4,096 tokens of the final batch; Figure 5 shows learned per-layer FFN utilization at 25%.Source: §E.2 · p. 17, Figure 5 · p. 17

F Inclusion Temperature Selection

Inclusion temperature controls the rank range of soft inclusion sensitivity around the selection boundary. The authors select T = 1.5 empirically as the lowest-perplexity tested value, while leaving further search for future work.

Temperature Selection

  • Soft-inclusion sensitivity to the per-layer selection threshold peaks at the selection boundary and decreases with distance from it.The interval of sensitive ranks extends T log 19 ranks on either side of the boundary for soft inclusions between 0.05 and 0.95.Source: F Inclusion Temperature Selection · p. 17
  • T = 1.5 gives the lowest perplexity among the tested inclusion temperatures, while potentially superior values remain unsearched.The comparison uses learned static allocation on Llama-2-7B with an initial routed count of 10 per layer.Source: F Inclusion Temperature Selection · p. 17, Table 9 · p. 17
  • With an initial threshold of 10, T = 1.5 places ranks 6 through 13 in the stated sensitivity interval, while T = 0.25 includes only ranks 9 and 10.The interval moves with the threshold, so ranks outside it initially can enter the transition region during training.Source: F Inclusion Temperature Selection · p. 17
Loading 2610.19232v1…