PaperlayerView exact revision on arXiv

Source-linked AI summary

What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning

Ezza Ali, Sehrish Bibi, Momina Moetesum, Faisal Shafait

arXiv:2610.41366v1cs.LGcs.CL

TL;DR

Training-free expert pruning chooses both which experts to remove and what the layer computes when routing selects a removed expert, but published comparisons often change both decisions. This paper holds the mask fixed to compare replacement operators and routing scales. Its findings show that trained gate scale matters, while no operator dominates across models.

  • Problem

    Published pruning comparisons often change expert selection and replacement together, leaving controlled evidence across replacement families, router conventions, masks, and evaluation protocols missing.

    Source: §1 · pp. 1–2

  • Method

    The study holds the pruning mask fixed while comparing replacement operators and routing semantics across tested MoE models and router conventions.

    Source: §3 · pp. 6–7

  • Results

    At 50% pruning, rerouting at the router’s trained gate scale is better on all four routers, while no replacement operator dominates across models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Within tested configurations, use the router’s trained gate scale when rerouting; the tilted novice or a partial copy can recover some accuracy at aggressive pruning.

    Source: §7 · p. 19

    The study covers tested base models with 40–64 routed experts and compares only the two endpoint gate scales.

    Source: §7 · p. 20

Abstract

from arXiv · show

Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.

1. Introduction

Pruning comparisons confound expert selection with what happens when routing selects a removed expert; this work holds the mask fixed to compare replacement operators. The results show that trained gate scale matters, the tilted novice can improve on a constant replacement, and no operator wins across all models.

Study design

  • Holding the pruning mask fixed isolates the replacement computation used when the router selects a removed expert, addressing comparisons that otherwise vary both decisions.The study tests three 64-expert architectures and adds Granite for a routing-normalization test.Source: §1 · pp. 1–2

Main findings

  • At 50% pruning, rerouting at the router’s trained gate scale is the better scale on all four routers; using the other scale scores below MoNE’s constant on each.On OLMoE and DeepSeek-V2-Lite, renormalizing survivors costs 0.8–8.5 points and reverses whether rerouting beats the constant; the direction reverses on Granite and Moonlight.Source: §1 · p. 2
  • At 50% pruning, the tilted novice adds 1.2–1.8% of pruned-model parameters and improves over the constant by 3.45 points on OLMoE and 1.37 on Moonlight.The gains move by at most 0.20 points across three calibration draws; under 4- and 3-bit quantization, they are not detectably reduced in 11 of 12 settings.Source: §1 · p. 2
  • No replacement operator dominates: trained-scale rerouting beats the tilt on DeepSeek-V2-Lite, while the tilt beats rerouting on Moonlight and at 50% on OLMoE.A partial copy of each pruned expert is statistically indistinguishable from the tilt on OLMoE and DeepSeek-V2-Lite.Source: §1 · pp. 2–3

Evaluation and calibration

  • At 25% pruning, MoNE’s constant retains 92–97% of dense multiple-choice accuracy but only 5–65% of GSM8K and 0–17% of code scores.Changing the calibration domain moves the pruned set 4–9 times as much as resampling general text at a matched token budget.Source: §1 · p. 3

Contributions and scope

  • The paper provides a five-item reporting checklist and limits its claims to one-shot, training-free pruning of tested base models with 40–64 routed experts and two endpoint gate scales.It does not infer preservation of open-ended generation from multiple-choice scores.Source: §1 · p. 3

2. Related Work

Prior work studies expert selection, replacement, routing after compression, expert compression, and evaluation, but often fixes one pruning decision while varying another. This paper’s related-work framing distinguishes these lines and identifies the need for controlled comparisons across replacement families and router conventions.

Expert selection

  • Selection methods score or search for experts to remove, while many fix the replacement operator; this work fixes selection and varies the operator, also crossing four masks with operators at 50%.The cited approaches range from activation and reconstruction criteria to second-order, interaction-aware, and learned importance methods.Source: §2 · p. 3

Replacing a removed expert

  • Replacement approaches include MoNE’s mean-output novice, fitted proxy maps, workload assignment, and compensation of survivors’ weights, but published gains can conflate selection and replacement.The tilted novice is presented as an input-map extension of the constant, using classical reduced-rank ridge regression.Source: §2 · pp. 3–4

Expert merging

  • Merging shares parameters while retaining router targets; this study evaluates slot remapping, a merging-free mapping of a pruned expert’s router index to a retained expert.It does not run merging pipelines itself.Source: §2 · p. 4

Routing after compression

  • After deletion, routing refills the top-k from survivors and changes surviving gate weights; the appropriate gate scale depends on the router’s training convention.REAP and HOPE assume top-k-renormalized routers, whereas OLMoE and DeepSeek-V2-Lite were trained without that renormalization.Source: §2 · p. 4

Compressing experts

  • Compression work can store experts more cheaply without deleting them; this study uses activation-whitened SVD copies and compares quantization at matched storage.The paper also tests whether replacement gains survive quantization.Source: §2 · p. 5

Calibration and evaluation

  • Calibration domain and evaluation task can alter pruning conclusions, while many MoE pruning studies report single runs; this work examines domain effects and uses paired uncertainty estimates with correction.The related work notes that reasoning and knowledge-intensive tasks can degrade before multiple-choice benchmarks.Source: §2 · p. 5

3. Setup

The setup isolates replacement behavior by holding masks, calibration tokens, and evaluation fixed, then compares operators under defined routing scales and paired evaluation procedures.

3. Setup

  • Selection, calibration tokens, and evaluation stay fixed while only the computation assigned to pruned experts’ routing slots varies.The protocol separately measures calibration noise, test-item noise, and multiplicity.Source: §3 · p. 5, Figure 1 · p. 6
  • MoNE masks prune 25%, 50%, or 75% of routed experts across OLMoE, DeepSeek-V2-Lite, and Moonlight; Granite supplies an additional routing-normalization test.Masks use MoNE’s fusion score, per-layer ranking, and 1,000 calibration sequences; Granite uses its own tokenization and masks.Source: §3 · p. 5
  • Rerouting keeps the same selected survivors but either preserves their unpruned-router gates or rescales gates as if pruned experts were absent.The trained-scale arm is keep-scale for OLMoE and DeepSeek-V2-Lite, and renormalized for Granite and Moonlight.Source: §3 · pp. 6–7, Table 3 · p. 7
  • The comparison includes mean-output constants, a fitted low-rank map, survivor proxies and slot remapping, rerouting, and Fisher-weight compensation.ConMoE assignment and COMPEL compensation are tested on MoNE masks, so these are component comparisons rather than evaluations of the complete methods.Source: §3 · pp. 6–7
  • Evaluation covers nine-task zero-shot accuracy, a five-task large-sample mean, GSM8K and code generation, and WikiText perplexity.The five-task mean avoids giving small tasks the same weight as MMLU; generation includes HumanEval and, at 25% pruning, MBPP.Source: §3 · p. 7
  • Paired item bootstraps quantify test-item uncertainty, disjoint calibration slices assess operator-fit variability conditional on fixed masks, and Holm correction controls multiple comparisons.An interval containing zero is not treated as evidence of equivalence.Source: §3 · pp. 5, 7, 8
  • Fit share measures the fraction of a constant replacement’s squared output error removed, while recovery share measures the fraction of its model-level damage removed.The paper distinguishes in-sample unweighted fit from a held-out, weighted fit test.Source: §3 · p. 8

4. Does the Replacement Change the Result?

At MoNE’s masks, a better-than-constant replacement improves accuracy at 50% and 75% pruning on all three models, but the best operator depends on the model. Across masks, protocols, and budgets, the tilt offers useful gains in some settings, while rerouting and partial expert copies also have advantages.

4.1. Main Results

  • The tilt beats the constant at 75% pruning on all three models and at 50% on OLMoE and Moonlight, with gains significant where constant damage exceeds 10 points.The gain is not one curve across models: OLMoE gains +1.83 at 25% despite 5.6 points of damage, while DeepSeek-V2-Lite gains +0.48 at 50% with 7.3 points of damage.Source: §4.1 · p. 8
  • At 50% and 75% pruning, no operator wins universally: the tilt beats rerouting on Moonlight, while trained-scale rerouting wins on DeepSeek-V2-Lite.On OLMoE at 50%, the tilt’s largest gain over the constant is +3.45 points.Source: §4 · p. 8, §4.1 · p. 8, Figure 3 · p. 9

4.2. Does the Ranking Depend on the Mask?

  • Across four 50% masks, the tilt ranks highest on OLMoE and trained-scale rerouting on DeepSeek-V2-Lite, although weaker operators’ order and gaps vary by mask.Changing among competitive masks moves accuracy by 1.9–4.1 points on OLMoE and 1.3–2.0 on DeepSeek-V2-Lite; the authors caution these are observed ranges from two models, not a general decomposition.Source: §4.2 · pp. 9–10, Table 5 · p. 9

4.3. Accuracy per Added Parameter

  • At the tilt’s rank-32 budget, highest-energy-neuron copies are statistically indistinguishable from the tilt on OLMoE and DeepSeek-V2-Lite, while SVD copies stay near the constant.On Moonlight the tilt is ahead of the neuron copy by +0.66 points; at four times the budget, the neuron copy overtakes rank-32 tilt on OLMoE by +1.50 points.Source: §4.3 · p. 10

4.4. Robustness to the Scoring Protocol

  • Tilt gains on OLMoE and Moonlight persist under five-shot prompting, raw accuracy, and changing answer options; rerouting is more protocol-sensitive.At five shots, the tilt keeps 87% and 112% of its zero-shot gain on OLMoE and Moonlight, respectively.Source: §4.4 · pp. 10–11, Table 6 · p. 11

4.3. Accuracy per Added Parameter

  • On OLMoE, rank-8 tilt reaches 51.92 at 16.8 M added parameters, beating one extra expert per layer with a sixth of its parameters (+1.65‡).The rank-128 tilt scores 54.05, about three points above keeping two extra experts per layer with a third more parameters.Source: §4.3 · p. 10, Figure 5 · p. 11

4.5. What the Tilt Restores

  • At 50% pruning, the tilt recovers 14–34% of language-modeling damage, varying by model but stable within 1.2 points across calibration draws.On OLMoE at 75%, WikiText perplexity falls from 1,434 under the constant to 172 with the tilt, versus 1,280 with rerouting. At OLMoE 50%, the tilt removes 28% of the constant’s gate-weighted residual error, but this decomposition does not establish what causes accuracy gains.Source: §4.5 · pp. 11–12

5. Which Routing Semantics Matter?

Holding masks fixed, the paper finds that rerouting works best at the gate scale the router was trained with; changing that scale can reverse operator comparisons. No replacement dominates across models, and remapping or weight compensation does not consistently match the strongest alternatives.

5.1. Renormalized Gates

  • On OLMoE and DeepSeek-V2-Lite, renormalizing survivors costs 0.8–8.5 points and lowers two-choice ranking ability versus keep-scale rerouting.On DeepSeek-V2-Lite, the cost grows from −0.79 to −8.47 points as pruning increases; on OLMoE it is already large at 25%.Source: §5.1 · p. 12, Table 8 · p. 12
  • With renormalized gates, rerouting falls below MoNE’s constant in all six softmax settings, while trained-scale rerouting beats it in five of six.On the same DeepSeek-V2-Lite REAP mask, the comparison changes from −3.3 points with REAP’s renormalized semantics to a +0.6-point estimate at the trained scale.Source: §5.1 · pp. 12–13, Table 9 · p. 12

5.2. Slot Remapping and Weight Compensation

  • Nearest-slot remapping underperforms keep-scale rerouting on every model and falls 1.3–3.7 points below MoNE’s constant on every model.Functional remapping recovers 2.04 points versus nearest remapping on OLMoE, but remains 4.01 points below keep-scale rerouting.Source: §5.2 · p. 13

Weight compensation and cross-expert proxies do not

  • COMPEL’s Fisher-weighted compensation costs 3.26 points on OLMoE and changes DeepSeek-V2-Lite and Moonlight by +0.39 and −0.61 points, respectively, neither significant.On DeepSeek-V2-Lite and Moonlight, MoRA’s channel-wise proxies find at most 3 of 832 pruned experts passing its threshold, and alter accuracy by at most +0.19 points.Source: Weight compensation and cross-expert… · p. 13

5.3. Routers Trained with Renormalization

  • On Granite, renormalized rerouting beats keep-scale rerouting by +2.08 points at 50%, but the two scales are not detectably different at 25% or 75%.Granite’s nearest remapping scores −3.51 against renormalized rerouting, −1.43 against keep-scale rerouting, and −2.04 against the constant.Source: §5.3 · pp. 13–14, Table 10 · p. 13
  • On Moonlight, trained-scale rerouting beats keep-scale rerouting by +1.68 points at 50% and +1.60 at 75%; the difference is not detectable at 25%.The 50% contrast survives Holm correction over all 111 comparisons.Source: §5.3 · p. 14, Table 11 · p. 15
  • At 50% pruning, the trained gate scale is best on all four routers: keep-scale rerouting on OLMoE and DeepSeek-V2-Lite, and renormalized rerouting on Granite and Moonlight.At that ratio, the wrong scale scores below MoNE’s constant on all four routers; the trained scale is not detectably different from it or is above it.Source: Table 8 · p. 12, Table 9 · p. 12, §5.3 · p. 15

5.4. Across Replacement Families

  • Across replacement families, no operator wins on every model: trained-scale rerouting leads on DeepSeek-V2-Lite, while the tilt leads on OLMoE and Moonlight.A partial neuron copy and the tilt are not detectably different on OLMoE and DeepSeek-V2-Lite; the tilt leads on Moonlight.Source: §5.4 · p. 15

6. Does the Answer Survive Realistic Evaluation?

The evaluation finds that pruning outcomes depend on the task and calibration domain: multiple-choice scores can conceal severe generative losses, and domain changes can alter both masks and operator rankings. Quantization comparisons also vary by deployment domain and storage budget.

6.1. Multiple Choice against Generation

  • At 25% pruning, OLMoE’s constant retains 5% of dense GSM8K and none of its HumanEval score, despite retaining over 90% of multiple-choice accuracy.Across models, multiple choice retains 79–93% of margin above chance, but code collapses at 25% under every operator.Source: §6.1 · p. 15, Figure 7 · p. 16
  • At 25% pruning with general-text calibration, no operator preserves GSM8K or MBPP, and multiple-choice operator rankings can differ from generation rankings.On Moonlight GSM8K at 25%, rerouting gains ten points while the tilt gains none; on DeepSeek-V2-Lite MBPP, keep-scale rerouting leads and the tilt scores 2.80 points lower.Source: §6.1 · p. 16, Table 13 · p. 17

6.2. Calibration versus Ability on Fixed-Answer Tasks

  • On fixed-answer tasks, answer bias and ranking ability can diverge: the tilt’s OLMoE and Moonlight RTE losses leave AUC essentially unchanged, while its OLMoE BoolQ gain raises AUC by 0.089.The constant can be badly miscalibrated, predicting “True” on 96% of OLMoE 50% RTE items despite ranking above chance.Source: §6.2 · p. 16

6.3. Perplexity versus Accuracy

  • Perplexity is not a reliable test of whether an operator beats the constant: DeepSeek-V2-Lite rerouting lowers perplexity from 22.74 to 19.70 while increasing accuracy damage by 45%.On Moonlight, trained-scale rerouting worsens WikiText perplexity but improves C4 and Zyda-2 perplexity.Source: §6.3 · p. 16

6.4. Calibration Domain

  • Changing calibration domain moves the pruned set 4.9–9.3 times more than resampling at 25–50%, even with token-matched general-text masks.At 50% pruning, changing domain swaps about a third of the pruned set; the authors identify routing concentration as a likely mechanism, not a tested one.Source: §6.4 · p. 17
  • With GSM8K-calibrated masks, 25% pruning retains essentially all dense GSM8K scores: 46.3, 37.8, and 73.8 versus dense scores of 45.5, 37.5, and 73.8.At 50% on OLMoE, keep-scale rerouting gains +22.7 GSM8K points over the constant, versus +19.4 for the tilt.Source: §6.4 · p. 17

6.5. Pruning with Quantization at Matched Storage

  • The tilt’s gain over the constant is not detectably changed by 4- or 3-bit quantization in 11 of 12 settings, while rerouting’s gain falls in 11 of 12.The proposed mechanism for rerouting’s decline—sending every slot through quantized experts—was not tested.Source: §6.5 · p. 18
  • At matched storage on general tasks, uniform 3-bit quantization scores 1.33 and 1.72 nine-task points above the best pruned arm, the tilt, on OLMoE and Moonlight.At the smallest tested budget, 50% pruning with 4-bit weights scores 48.7–61.2 nine-task points, versus 39.2–45.7 for uniform 2-bit GPTQ.Source: §6.5 · p. 19

7. Discussion and Limitations

The discussion recommends choosing replacement operators in light of the router convention and task, rather than assuming a universal winner. The authors also describe compute trade-offs and scope limits, and propose a five-item reporting checklist.

7. Discussion and Limitations

  • Within tested configurations, trained-scale rerouting is the strongest parameter-free replacement, while a budgeted tilt or partial expert copy can recover accuracy damage under aggressive pruning.The best replacement depends on model, router convention, and target task; the authors frame these as observations, not a universal policy.Source: §7 · p. 19
  • The proposed checklist asks pruning studies to report router and post-pruning gate semantics, fixed-mask operator comparisons, a generative task, calibration domain and budget, and a matched-storage quantization baseline.Source: §7 · p. 19
  • At batch 32, the tilt keeps two thirds to three quarters of MoNE’s prefill saving, while rerouting keeps almost none; at batch 1, rerouting keeps 85–92%.The tilt adds 0.13 GB on OLMoE and 0.22 GB on DeepSeek-V2-Lite at 50% pruning.Source: §7 · pp. 19–20, Table 16 · p. 19
  • The study covers base models with 64 routed experts plus Granite with 40, not models with 128 or more experts or instruction-tuned checkpoints.The reverse trained-scale result relies on two renormalized routers, each with one calibration draw; the study also omits MATH and long-form tasks.Source: §7 · p. 20

Impact Statement

The impact statement cautions that memory reduction does not by itself establish inference-energy savings and recommends task-relevant evaluation and explicit deployment comparisons.

Impact Statement

  • Memory reduction is not automatically an inference-energy saving; the study measured selected memory and latency configurations, not energy across serving systems.It recommends task-relevant generative evaluation, explicit calibration and routing-semantic reporting, and matched-storage quantization comparisons.Source: Impact Statement · p. 20

AI Use Statement

The authors report AI assistance in literature research, methodological discussions, and manuscript polishing, with results manually reviewed. In the five listed settings, MMLU is usually at chance with the constant, while the Moonlight 50% tilt and rerouting lift it.

AI Use Statement

  • The authors used AI tools for literature research, methodological discussions, and manuscript polishing; they manually reviewed all results.Source: AI Use Statement · p. 20

A.1. Every operator at every ratio

  • Across five listed settings, the constant scores 23–27% on MMLU, and most operator changes are within 0.8 points.The settings are OLMoE at 50% and 75%, DeepSeek at 75%, and Moonlight at 50% and 75%.Source: §A.1 · p. 24
  • On Moonlight at 50% pruning, the tilt adds 4.9 MMLU points and rerouting adds 3.5.Table A.1 reports nine-task, big-5, and MMLU accuracy for MoNE masks on draw A; its rerouting scale is trained-scale.Source: §A.1 · p. 24, Table A.1 · p. 24

A.2. Per-task change over the constant

The appendix details task-level effects, calibration sensitivity, and evaluation boundaries for the operators, then specifies the tilted novice fit and its parameter cost. Multiple-choice accuracy can conceal severe generative losses, while operator comparisons vary across tasks and models.

A.2. Per-task change over the constant

  • On OLMoE, the tilt's task-level gains are largest on ARC-e (+13.4 to +13.9) and PIQA (+6.7 to +7.1), while Moonlight's MMLU rises 4.9–7.0 points across draws.Removing letter-specific mean preference leaves Moonlight MMLU at 37.7% for the tilt versus 28.1% for the constant; renormalization also hurts almost every task on OLMoE and DeepSeek-V2-Lite, especially BoolQ and WinoGrande.Source: §A.2 · p. 25

A.3. 75% pruning

  • At 75% pruning, both operators beat the constant on every model: rerouting leads on DeepSeek-V2-Lite, the tilt on Moonlight, and OLMoE shows no detectable difference.Source: §A.3 · p. 25, Table A.3 · p. 25

A.4. Calibration draws

  • The tilt's gain over the constant exceeds its 25% gain at 50% pruning on every model, but its 75% gain rises on DeepSeek-V2-Lite and Moonlight and falls on OLMoE.Across nine draw-A settings, constant damage and tilt gain have a descriptive rank correlation of 0.73; the passage cautions that the relationship also varies by model.Source: §A.4 · p. 26, Table A.4 · p. 26

A.5. Language modelling at every ratio

  • At 75% pruning, OLMoE perplexity is 1434 with the constant, 1280 with rerouting, and 172 with the tilt, although rerouting and tilt differ by only +0.90 accuracy points, not significantly.Across models, the tilt restores a larger share of language-modeling damage as pruning deepens, but its accuracy share does not follow the same pattern.Source: §A.5 · p. 26, Table A.5 · p. 26, Table A.6 · p. 27

A.6. Generation

  • With in-domain calibration, OLMoE at 50% pruning retains 44.28 GSM8K against 45.49 dense, while Moonlight's tilt reaches 73.31 against 73.84 dense.Every 50% model scores zero on HumanEval under every operator; multiple-choice accuracy can therefore mask generative damage.Source: §A.6 · p. 27, Table A.7 · p. 27, Table A.8 · p. 27
  • At 25% pruning, the constant retains 79%, 93%, and 91% of dense margin-above-chance accuracy on OLMoE, DeepSeek-V2-Lite, and Moonlight, versus only 5%, 65%, and 44% of GSM8K scores.For MBPP, the corresponding constant retains 0%, 17%, and 0%; at 75%, OLMoE's constant is only 1.9% of the way from chance to dense nine-task accuracy.Source: §A.6 · p. 28

A.7. MBPP at 25% pruning

  • At 25% pruning, Moonlight's constant generates 500 MBPP programs that are all syntax errors, and cutting generations at answer boundaries does not turn any pass into a failure.The text attributes code collapse to invalid code rather than an extraction artifact; keep-scale rerouting preserves syntax but still does not pass the tests.Source: §A.7 · pp. 28–29, Table A.10 · p. 28

A.8. Multiple-choice cost of task calibration; A.9. Mask drift at a matched token budget; A.10. Calibration versus ability, every arm

  • GSM8K-calibrated masks have model-dependent multiple-choice costs, while matched-token mask-drift analysis compares domain shifts with resampling and fixed-answer task evaluation separates accuracy from ranking ability.For fixed-answer tasks, renormalization lowers ranking ability in all eight OLMoE and DeepSeek-V2-Lite cells by 0.028–0.140 AUC; the anchor's WinoGrande result is a point estimate.Source: §A.8 · p. 29, Table A.12 · p. 29, §A.10 · p. 30

B.3. Why a per-expert map of the full input

The appendix motivates per-expert maps by showing that expert-output structure is high-rank and depends on the full input, then tests fit quality, calibration size, rank, implementation, and equal-budget copies. Results favor per-expert maps over shared maps and show that accuracy comparisons depend on the replacement family and model.

B.3. Why a per-expert map of the full input

  • A full-rank map removes 22–42% of held-out constant error, while the router-logit basis saturates below 19%; increasing ridge tenfold changes these values by less than 0.02.The full-input map is motivated by expert outputs being partly linear, high-rank, and structured in the full input; layer-shared maps retain only 9–32% of per-expert gain.Source: §B.3 · pp. 32–33

B.4. Fit quality by layer

  • At 50% pruning, mean in-sample R2 of the rank-32 tilt is 0.135 on OLMoE, 0.146 on DeepSeek-V2-Lite, and 0.131 on Moonlight.On OLMoE, tilting layer 15 alone gains +0.35 of the full +3.45; fit values in draws A and B agree within 0.003 in almost every layer.Source: §B.4 · p. 33, Table B.6 · p. 33

B.5. Calibration size

  • At 100 sequences, held-out R2 already reaches 78–89% of its 1,000-sequence value, yet accuracy gain varies by 2.5 points between two slices.The authors report that 1,000 sequences recover both the per-task pattern and the total replicate.Source: Table B.7 · p. 33, Table B.8 · p. 33, §B.5 · p. 34

B.6. Rank 8 against rank 32

  • Rank 8 captures 82% of rank-32 gain on OLMoE at one-quarter the parameters, but captures under half on Moonlight and incurs a small 50% loss on DeepSeek-V2-Lite.Source: §B.6 · p. 34, Table B.9 · p. 34

B.7. Correctness checks

  • The fitting checks show that streaming statistics match direct covariance, the full-rank fit matches OLS, and rank truncation matches optimal reduced-rank regression.The implementation also checks held-out residuals against direct evaluation and pooled analysis against per-expert fits.Source: §B.7 · p. 34

C.1. Construction

  • At equal budget, a partial copy retains the SwiGLU nonlinearity and stores a calibration-token mean-output bias; copy variants include plain SVD, whitened SVD, and top neurons.The copy budget counts stored factors and bias alongside the tilt's maps, feature mean, clip scale, and output mean.Source: §C.1 · p. 34, Table C.1 · p. 35

C.2. Accuracy

  • At 50% pruning, equal-budget neuron copies beat rerouting on OLMoE, while the tilt and neuron copy are not detectably different on OLMoE and DeepSeek-V2-Lite; the tilt leads on Moonlight.Post-hoc bounds rule out differences beyond 0.81 and 0.73 points on OLMoE and DeepSeek-V2-Lite, respectively.Source: §C.2 · p. 35, Table C.2 · p. 35

C.3. Fit quality

The appendix compares fit quality and language-modeling behavior across replacements, then documents scoring, re-implementation choices, and gate-scale semantics. It finds that better expert-output fit does not guarantee better accuracy, and that normalization choices can materially change rerouting comparisons.

C.3. Fit quality

  • At equal budget, neuron copies fit better than the rank-32 tilt at 4×, but the tilt has the best fit at 1×; across replacement types, fit quality does not reliably predict accuracy.The tilt wins accuracy over the neuron copy only on Moonlight, where its fit lead is largest; part of its fit advantage follows from directly minimizing the measured squared error.Source: §C.3 · pp. 35–36, Table C.3 · p. 35

C.4. Language modelling

  • On perplexity, neuron copies are ahead of the tilt on OLMoE, similar on DeepSeek-V2-Lite, and behind on Moonlight; rerouting's effect can change sign with evaluation text.On Moonlight, rerouting worsens WikiText perplexity but recovers 15–18% of damage on C4 and Zyda-2.Source: §C.4 · p. 36, Table C.4 · p. 36

C.5. Scoring protocol; C.6. Few-shot prompting

  • On Moonlight, rerouting loses 2.02 points on fixed-answer tasks, mainly BoolQ, while per-item-choice tasks show no detectable difference; at 25%, the tilt's only significant effect is mostly MMLU.The scoring appendix also reports comparisons at 0-shot and 5-shot, but the supplied passage gives no outcome for those comparisons.Source: §C.5 · pp. 36–37, Table C.5 · p. 36, Table C.6 · p. 37

D.1. What was re-implemented, and the settings the paper leaves open

  • The authors re-implemented MoRA from its paper because its announced code was unavailable, and did not run MC-SMoE or HC-SMoE baselines.Their stated re-implementation settings include 1,000 steps on 4 × 1,024 C4 tokens and a renormalized fallback.Source: §D.1 · p. 37

D.2. The diversity loss’s normalization decides the method

  • MoRA's diversity-loss normalization changes MoRA-9 accuracy by 1–8 points, and the best normalization differs between settings.At step 1,000, kept-set rankings had not settled; for DeepSeek-V2-Lite under the primary normalization, the step-900 kept-set share was 0.942 at 25% and 0.797 at 50%.Source: §D.2 · p. 37, Table D.1 · p. 37

D.3. In our re-implementation, almost no retained expert predicts a pruned expert’s output

  • In the re-implementation, almost no retained expert predicts a pruned expert's output well enough to pass MoRA's proxy threshold; the best real candidate scored at most 0.0164.For MoNE-C4 masks on DeepSeek-V2-Lite, the apparent proxy arm loss is attributed to renormalized rerouting, not the proxies.Source: §D.3 · p. 38, Table D.2 · p. 38

D.4. Reproduction of the paper’s rows

  • Dense results reproduce within 0.12 points, but MoRA's own rows are 1–4 points below the paper's reported rows, and the authors cannot explain the gap.Their MoNE-C4 rows match MoNE's published C4 table, while exceeding MoRA's MoNE row by +0.92 on DeepSeek-V2-Lite 25% and +1.39 on Moonlight 50%.Source: §D.4 · pp. 38–39, Table D.3 · p. 38

D.5. Gate scale on MoRA’s masks

  • On MoRA's DeepSeek-V2-Lite masks, keep-scale rerouting beats renormalized rerouting by +0.47 at 25% and +1.24 at 50%; Moonlight's renormalized gates match its trained scale.The paper's method assumes normalized selected weights, although DeepSeek-V2-Lite and OLMoE were not trained with that routing scale.Source: §D.5 · p. 39

D.6. Against our operators on MoRA’s tasks

On MoRA’s tasks, the calibration corpus accounts for much of the 50% result; the selection analyses then compare masks, their stability, and routing semantics across draws.

D.6. Against our operators on MoRA’s tasks

  • On MoRA’s task set, the best configuration matches or beats MoRA in every setting, while at 50% pruning the calibration corpus accounts for most of the margin.With the corpus fixed, tilt adds 1.35 points over the constant on Moonlight and keep adds 1.64 on DeepSeek; each setting has one run, so intervals are item-level only.Source: §D.6 · p. 39

E. Selection

  • For mask comparisons, every criterion is evaluated with keep-scale rerouting, and variation between draws therefore reflects mask variance alone.The criteria are computed per token by re-deriving routing and running each expert on its routed tokens.Source: §E.1 · p. 39

E.1. MoNE’s mask against seven other published criteria

  • Among seven published criteria, only SGE beats MoNE’s mask, and only on DeepSeek; frequency fails there because it prunes a super expert.The comparison uses keep-scale rerouting and a two-draw criterion; the task set and protocol differ from Liu et al. (2026b).Source: §E.1 · p. 40

E.2. Stability

  • Across calibration draws, criteria averaging over routed tokens retain 97–98% of pruned experts, while criteria carrying frequency retain 89–96%.Stability does not imply accuracy: the most stable criteria select worse masks here.Source: §E.2 · p. 40

E.3. REAP with its own semantics

  • On the same REAP mask, REAP’s renormalized routing semantics score 1.88 points lower on OLMoE and 3.88 lower on DeepSeek than keep-scale rerouting.The respective scores are 47.12 against 49.00 and 56.58 against 60.46.Source: §E.3 · p. 40

E.4. SGE mask with the tilt

  • The tilt’s nine-task advantage over MoNE’s mask is significant only on draw A, the outlier, although MMLU is significant on every draw.The SGE mask is compared with the MoNE mask across three draws.Source: §E.4 · p. 41, Table E.3 · p. 41

E.5. Operator- and interaction-aware selection

The selection and deployment analyses show that layer-level proxies can miss model-level damage, while operator performance, latency, and quantization effects depend on the setting.

E.5. Operator- and interaction-aware selection

  • Layer error correlates weakly with model-level damage: mean within-layer Spearman is 0.13 for the constant and 0.50 for the tilt.Retest values are 0.23 and 0.50, respectively.Source: §E.5 · p. 41

E.6. Fisher-weighted selection

  • Fisher masks reach MMLU 32.7–33.8 on four of five arms versus 27.5–30.8 for other masks, while five benchmarks fall (big-5 −2.2 against SGE + tilt); draw spreads are 2.0–4.6.Source: §E.6 · p. 42

E.7. Where the knowledge-critical experts sit

  • MMLU-critical experts sit in early and middle layer groups, yet their output magnitudes are ordinary, with median percentile 52.Swapping in either the early or middle group lowers MMLU; swapping the late group does not.Source: §E.7 · p. 42

E.8. Super experts

  • DeepSeek-V2-Lite has super experts, unlike OLMoE and Moonlight; keeping the frequency-pruned expert gains 5.46 and 5.28 points across two draws.Those gains close 91% and 83% of the deficit to MoNE’s mask.Source: §E.8 · pp. 42–43

F.1. Memory and latency

  • At batch 32, the tilt retains two thirds to three quarters of MoNE’s prefill saving, while rerouting retains almost none; at batch 1, the order reverses.Decode is launch-bound in the unfused implementation, and rank 32 costs about the same per token as rank 8 despite four times the arithmetic.Source: §F.1 · p. 43

F.2. Quantization setup

  • The quantization self-test halves relative output error against round-to-nearest, from 0.1065 to 0.0544; all 60 runs had zero tokens reach a pruned expert.Quantization is simulated, so these accuracy measurements do not establish speed.Source: §F.2 · p. 43

F.3. Accuracy against deployed size

  • At the smallest budget, every 50%-pruned w4 arm scores 48.68–61.23, versus dense GPTQ w2 scores of 45.71, 41.10, and 39.19.Part of the margin reflects attention precision; the study did not run a 2-bit-expert arm with 4-bit attention.Source: §F.3 · p. 44, Table F.2 · p. 44

F.4. Do the operators’ gains survive quantization?

  • The tilt’s gain survives 4- and 3-bit quantization, while rerouting’s erodes; 11 of 12 rerouting interactions are negative and five are significant.Only one of 12 tilt interactions is significant; the proposed explanation that rerouting sends every slot through quantized experts is untested.Source: §F.4 · p. 44, Table F.3 · p. 44

F.5. Pruning against quantizing at matched size

At matched size, pruning and quantization compare differently under general versus task-specific calibration; the appendix also documents reproducibility checks and implementation settings.

F.5. Pruning against quantizing at matched size

  • With general calibration, quantizing every expert scores above pruning on general and knowledge tasks; every pruned arm is below uniform bits on every model.The loss is mostly MMLU (−2 to −11); with in-domain generation, no task-calibrated pruned arm is behind uniform bits.Source: §F.5 · pp. 44–45, Table F.4 · p. 44

G.1. Environment and data

  • The experiments reuse cached Zyda-2 token blocks across pruning, fitting, scoring, and quantization runs, with separate GSM8K-train caches for GSM8K calibration.The appendix specifies the source, block slices, and cache counts for the three models.Source: §G.1 · pp. 45–46

G.2. MoNE reproduced

  • MoNE reproduces within 0.32 points on every average, with mean absolute differences of 0.22 across six OLMoE settings and 0.30 and 0.25 on DeepSeek.The DeepSeek 25% mask also matches the authors’ released checkpoint on all 416 pruned experts.Source: §G.2 · p. 46

G.3. Integrity checks

  • Integrity checks verify routing alignment and gate replays, confirm zero tokens reach pruned experts, and reproduce stored evaluation results before reporting new numbers.The checks also report that 27 rerun arms matched stored scores within 0.10.Source: §G.3 · pp. 46–47

G.4. Compute

The appendices document compute and availability, then specify the scope and limits of statistical comparisons and provide further mask and parameter-matched results.

G.4. Compute

  • A rank-32 fit pass over 1,000 sequences uses about 1.7 GB of float64 statistics per layer; one GPTQ quantization takes 5–17 minutes per model.Reported runs used four A6000 GPUs.Source: §G.4 · p. 47

G.5. Availability

  • Code, masks, and fitted operators are not yet public, although the appendices provide settings needed to reproduce each result..Source: §G.5 · p. 47

H. Mask grid, further operators and statistics

  • The appendix reports paired comparisons with Holm-adjusted p-values and intervals, covering specified comparison families rather than every comparison in the paper.The correction families were fixed after most results were known, so the correction is conservative over conclusion-driving contrasts rather than confirmatory.Source: §H.1 · p. 47, H. Mask grid, further operators and… · p. 47

H.1. Holm-corrected comparisons

  • Across 111 comparisons, 47 remain significant after Holm correction over all; the smallest attainable adjusted value is about 0.022, where entries are at bootstrap resolution limit B.Source: §H.1 · p. 49

H.2. Equivalence bounds for the comparisons the text calls close

  • The reported equivalence bounds are not tests of a prespecified equivalence hypothesis because no margin was set before seeing the data.The table gives paired differences and intervals for comparisons the text calls close.Source: §H.2 · p. 49, Table H.2 · p. 50

H.3. The crossed grid in full

  • The highest-scoring operator is unchanged across masks: tilt leads on OLMoE, while keep-scale rerouting leads on DeepSeek-V2-Lite.The ordering of the remaining operators varies by mask; the tilt’s gain over the constant on the SGE mask remains +3.17 to +4.52 across three draws.Source: §H.3 · pp. 50–51, Table H.3 · p. 50

H.4. Parameter-matched contrasts

  • The parameter-matched table compares nine-task accuracy against the added parameters for replacements, including compressed expert copies at the tilt’s rank-32 budget.It also reports same-process constants and describes extra-expert baselines.Source: §H.1 · p. 49, Table H.6 · p. 51

H.5. Evaluation check for the extra-expert arms

The appendices check evaluation consistency, detail alternative replacement implementations, and qualify cross-model comparisons. They show that Granite’s results depend on trained gate scale, while calibration domain, task baselines, and limited tests constrain interpretation.

H.5. Evaluation check for the extra-expert arms

  • In-process and saved-checkpoint MoNE evaluations use the same constants, whose relative L2 distance is at most 3.4 × 10^-3, yet their nine-task scores differ by +0.32 and −0.16.The differences are in opposite directions, and per-task discrepancies are at most 1.08 points.Source: §H.5 · p. 51, Table H.7 · p. 51

H.6. Slot remapping and COMPEL: implementation and details

  • Functional slot assignment changes about half the slots, but its two calibration folds disagree on 28–38% of pruned experts; COMPEL instead distributes pruned weights across survivors.The functional rule considers four distance-nearest candidates and accepts an alternative only when both folds prefer it; COMPEL’s median survivor weight changes are 1.5–1.6%.Source: §H.6 · pp. 51–52

H.7. Relation to numbers reported by ConMoE, COMPEL and HOPE

  • Reported comparisons with ConMoE and COMPEL use different pools, calibration data, or evaluation settings, limiting direct interpretation of their results.ConMoE’s reported OLMoE figures agree directionally with the slot-remapping comparison, while the COMPEL perplexity gap could not be resolved.Source: §H.7 · p. 53

H.8. Granite: the gate-scale test in detail

  • The Granite 50% rerouting reversal favors renormalized over keep-scale on all nine tasks, but the +2.46 big-5 difference carries the result because MMLU is below chance.At 25% only MMLU’s difference is significant; at 75% neither the big-5 nor nine-task difference has a consistent direction, and BoolQ may reflect answer bias.Source: §H.8 · p. 54
  • Granite’s 50% operator order is tilt (48.64), trained-scale rerouting (46.74), constant (45.27), other-scale rerouting (44.65), and nearest remapping (43.23).The trained scale is renormalized on Granite; the tilt beats the constant and renormalized rerouting at every pruning ratio, while keep-scale rerouting loses to the constant at 25%.Source: Table H.11 · p. 54, §H.8 · p. 55

H.10. Second-draw masks from MoNE’s own code

  • MoNE’s masks from its released code overlap slice A by 93.6% on OLMoE and 95.8% on DeepSeek-V2-Lite and Moonlight.Source: §H.10 · p. 55

I.1. Candidate explanations

  • Granite’s 25% and 75% rerouting nulls have untested candidate explanations; at 75%, results are near guessing and BoolQ shifts mix answer bias with ability.On Moonlight, five pruned experts receive only 39–128 GSM8K calibration tokens, too few to fit a rank-32 map, so they use the constant.Source: §I.1 · p. 56

I.2. Which results are exposed to calibration noise

Calibration-draw effects depend on whether an operator is fitted again. The constant and tilt show small draw-to-draw changes in the reported OLMoE tests, while several other results remain exposed to calibration noise.

I.2. Which results are exposed to calibration noise

  • Across draws, the constant’s nine-task score moves at most 0.39 points, and the tilt’s gain over it moves at most 0.20 points at 50% and 0.58 at 25%.Source: §I.2 · p. 56
  • Rerouting and nearest remapping fit nothing, so their results are identical across draws and their contrasts with the constant inherit its small draw noise.Source: §I.2 · p. 56
  • Other draw-A-only results remain exposed to calibration noise, including several fitted operators, generative and 5-shot tilt tests, crossed-grid results, and all Granite results.Calibration-domain results use one GSM8K draw, and the mask grid covers 50% pruning on two models.Source: §I.2 · p. 56
Loading 2610.41366v1…