PaperlayerView exact revision on arXiv

Source-linked AI summary

Higher-order pruning of experts in mixture-of-experts language models

Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia, Stefano Soatto

arXiv:2609.18916v3cs.LGcs.AI

TL;DR

MoE expert pools create a substantial GPU-memory bottleneck, while existing pruning methods score experts independently despite their cooperative interactions. HOPE incorporates pairwise interactions into pruning decisions, and it outperforms other methods most often, especially at higher pruning rates. Its calibration adds modest time and data requirements.

  • Problem

    MoE experts consume substantial GPU memory, while existing pruning methods score experts independently and ignore cooperative interactions between experts.

    Source: §1 · pp. 1–2

  • Method

    HOPE minimizes a tractable upper bound on pruning error using a second-order objective that accounts for pairwise expert interactions.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 4

  • Results

    HOPE beat each individual baseline in most conditions, with a 73% average head-to-head win rate.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    HOPE’s advantage is associated with retaining experts that have stronger cooperative structure, which first-order methods miss.

    Source: §1 · p. 2, §4.3 · p. 6

    Calibration takes about 6–7% longer than REAP, and reaching 99% prune-set agreement required 10k prompts for HOPE versus 8k for REAP in the reported setting.

    Source: §5 · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective which provably minimizes an upper bound on the error resulting from pruning. We show that REAP (a state-of-the-art first-order pruning method) is a special case of HOPE where interaction terms are ignored. Across three frontier MoE models (up to 122B parameters), two distinct calibration sets, and multiple benchmarks (including math, instruction following, coding, and an agentic suite), we demonstrate that HOPE produces better pruning decisions than existing methods, and its advantage is most pronounced at high pruning rates and on challenging agentic workloads. At 50% pruning, HOPE outperforms all baselines and achieves an average rank of 1.58 out of 5 methods (versus 2.42 for the next-best method, REAP), with gains of up to +6.1% on agentic coding. Over all conditions, HOPE again achieves the best average rank and surpasses every other method in the majority of head-to-head comparisons. By preserving cooperative expert structure that first-order methods ignore, HOPE enables aggressive compression with minimal degradation, particularly on complex tasks where diverse expert combinations are invoked over long sequences.

1 INTRODUCTION

MoE experts create a major memory bottleneck, and independent pruning scores miss the cooperative structure among experts. HOPE uses pairwise interactions to make pruning decisions, with its advantage growing at higher pruning rates and on complex agentic tasks.

1 INTRODUCTION

  • Expert parameters compete with the KV cache for GPU memory, limiting hardware affordability, concurrent requests, and budgets for context, tools, and reasoning.In Qwen3.5-122B-A10B, expert MLPs account for 116 billion of 122 billion parameters and about 240 GB.Source: §1 · p. 1
  • Existing first-order methods score experts independently, although co-selected experts can cooperate and their simultaneous removal can compound damage at 40–50% pruning.Halving the expert pool is especially useful for deployment because it frees memory for longer contexts, extended reasoning, or more concurrent requests.Source: §1 · p. 2
  • HOPE uses pairwise expert interactions in a second-order pruning objective, while REAP is its special case with interaction terms ignored.The objective is derived to minimize an upper bound on pruning error and uses a per-layer interaction matrix.Source: §1 · p. 2
  • Across three MoE architectures, HOPE most often outperforms existing methods, with its advantage growing at high pruning rates and on complex agentic tasks.The analysis attributes the advantage to retaining experts with stronger cooperative structure.Source: §1 · p. 2

2 RELATED WORK

MoEs route each token through a small set of experts, while expert pruning removes selected experts based on calibration statistics. Prior pruning methods score experts independently, despite evidence that experts cooperate through co-selection.

2.1 MIXTURE-OF-EXPERTS

  • A MoE layer replaces its MLP with expert MLPs and a learned router, which selects a subset of experts for each token.Load-balancing training encourages uniform use, but utilization remains non-uniform, enabling pruning.Source: §2.1 · p. 2

2.2 EXPERT PRUNING

  • Expert pruning uses calibration-set statistics to score experts, then removes the lowest-scoring fixed fraction in each layer.Frequency ranks experts by activation count, while EAN ranks by summed output norms over active tokens.Source: §2.2 · p. 2
  • REAP ranks experts by gate-weighted activation norms, whereas MAN uses activation norms without gate weighting; both belong to first-order methods that ignore interactions.REAP is described as a state-of-the-art method for generative tasks, with theoretical justification.Source: §2.2 · pp. 2–3

2.3 EXPERT INTERACTION AND COOPERATION

  • Prior work found stable, exploitable expert co-selection patterns, but had not used this cooperative structure to guide pruning decisions.Examples include co-selection patterns that stabilize during training and a router graph encouraging repeated activation of cooperative pairs.Source: §2.3 · p. 3

3 HOPE: HIGHER-ORDER PRUNING OF EXPERTS

HOPE formulates expert pruning as minimizing a tractable upper-bound surrogate that captures pairwise expert contributions. It estimates interactions from calibration data, solves a quadratic program, and contains REAP as the interaction-free case.

3 HOPE: HIGHER-ORDER PRUNING OF EXPERTS

  • HOPE seeks a prune-set of a prescribed size that minimizes pruning error for a MoE layer.The derivation focuses on a theoretical objective for selecting experts to remove.Source: §3 · p. 3, §3.1 · p. 3

3.1 SETUP AND NOTATION

  • A MoE layer selects top-K experts for each token, and pruning replaces removed active experts with the next-highest-logit experts.The layer output combines expert outputs weighted by router gates.Source: §3.1 · p. 3

3.2 ERROR DECOMPOSITION

  • Pruning error comprises substitution error from replacing removed experts and renormalization error from rescaling retained gates; HOPE focuses on substitution error because renormalization error is typically small..Source: §3.2 · p. 3

3.3 THE SECOND-ORDER OBJECTIVE

  • HOPE minimizes squared substitution error through Z, a tractable surrogate whose minimization provably reduces that error and whose expansion captures pairwise interactions.The upper bound includes a conservative replacement-expert factor, while the practical gap is far smaller than the bound suggests.Source: §3.3 · p. 4

3.4 MINIMIZING Z

  • HOPE estimates the expected surrogate objective over calibration data and formulates prune-set selection as a binary quadratic program using F, whose diagonal represents individual contributions and off-diagonal entries pairwise co-contributions.The method conditionally averages over tokens activating each pair to avoid undervaluing rarely co-selected pairs.Source: §3.4 · p. 4

3.5 CONNECTION TO REAP

  • HOPE’s diagonal captures individual expert importance, while its off-diagonal entries encode interactions; zeroing those entries empirically recovers REAP’s prune-set.The full matrix differs from REAP by including second-order interaction terms.Source: §3.5 · p. 5

3.6 PRACTICAL IMPLEMENTATION

  • HOPE records expert activations, gate values, outputs, and co-activation interactions in one calibration pass, then solves a per-layer QP with negligible added cost.The QP takes 1–2 seconds per layer for typical expert counts; the continuous relaxation is empirically tight.Source: §3.6 · p. 5

4 EXPERIMENTS

HOPE performs best overall, with its advantage strongest at aggressive pruning, and its interaction matrix captures cooperative expert structure. The experiments also find stable pruning decisions across trials and a HOPE advantage that remains after supervised fine-tuning.

4.1 EXPERIMENTAL SETUP •

  • The evaluation covered three MoE models, two calibration sets, pruning rates from 10% to 50%, and 54 configurations across standard and agentic benchmarks.Models ranged from 35B to over 100B parameters; benchmarks included math, coding, instruction following, and general knowledge.Source: §4.1 · p. 5

4.2 HOPE PERFORMANCE COMPARISON

  • Across 54 conditions, HOPE had a 73% average head-to-head win rate and finished last only once, with its strongest gains on agentic coding, code generation, and compositional reasoning.Its gains over the next-best method reached +2.0% on SWE-Bench Pro, +19.6% on LiveCodeBench, and +11.7% on BBH.Source: §4.2 · pp. 5–6
  • At 40–50% pruning, HOPE averaged rank 1.58, topped most conditions, and beat REAP in every high-pruning agentic experiment, by +2.8% mean and up to +6.1%.Figure 1 summarizes performance across conditions and pruning rates.Source: §4.2 · p. 5, Figure 1 · p. 6

4.3 STRUCTURE OF HOPE’S F -MATRIX

  • Off-diagonal F terms averaged 33% of diagonal magnitude; HOPE retained experts with stronger cooperative scores, as its QP penalizes pruning experts in cooperative clusters.Figures 2 and 3 show the interaction structure and expert-retention pattern.Source: §4.3 · pp. 6–7, Figure 2 · p. 7, Figure 3 · p. 7
  • HOPE’s prune-set overlap with other methods increased as pruning rates rose, but remaining disagreements mattered more; disagreements were distributed across layers.The interaction-aware decisions therefore remain distinct across the network, even as methods converge on commonly unimportant experts.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 CALIBRATION AND TRIAL ROBUSTNESS

  • Across three random trials, all methods had average prune-set Jaccard similarity above 0.95; across calibration sets, HOPE averaged 0.60 versus REAP’s 0.61.The paper reports that calibration-set variation affects all methods and does not make HOPE more sensitive than REAP.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 COMPATIBILITY WITH DOWNSTREAM TRAINING

  • On Qwen3.5-122B-A10B, HOPE-pruned models outperformed REAP-pruned models both before and after one billion tokens of supervised fine-tuning, though fine-tuning narrowed the gap.The matched-trial comparison used LiveCodeBench traces collected from the unpruned model.Source: §4.5 · p. 8, Figure 6 · p. 9

5 CONCLUSION

HOPE incorporates pairwise expert interactions into pruning and outperforms other methods, especially at aggressive pruning rates. Its calibration adds small costs, and cross-layer interactions remain outside the method’s current scope.

5 CONCLUSION

  • HOPE is a theoretically derived quadratic-programming objective that accounts for pairwise expert interactions and outperforms other methods, especially at aggressive pruning rates.The conclusion identifies high-pruning-rate settings as most relevant to real-world deployment.Source: §5 · p. 9
  • HOPE’s off-diagonal interaction terms guide retention of experts with high cooperative scores, a signal first-order methods miss.The authors report that these terms drive selective retention and validate their prediction that considering them yields better prune-sets.Source: §5 · p. 9
  • HOPE adds small calibration costs and still prunes each layer independently, without accounting for cross-layer interactions.The authors also identify extending beyond pairwise interactions as a possible direction for future work.Source: §5 · p. 9

A FULL DERIVATION OF HOPE

The derivation models expert replacement under top-K routing, separates pruning error into substitution and renormalization components, and establishes why the latter is generally small.

A.1 SETUP

  • Pruning replaces each removed selected expert with a next-highest-logit expert, preserving the number of selected experts.The derivation sets up an MoE layer with routed experts and gate weights before defining this replacement.Source: §A.1 · p. 12

A.2 ERROR DECOMPOSITION

  • Pruning error splits into substitution error from swapping experts and renormalization error from rescaling untouched selected experts’ gate values.The renormalization effect follows because gate values must sum to 1.Source: §A.2 · p. 12

A.3 GATE-MASS LEMMA

  • Under top-K softmax routing, replacement experts have lower total gate mass than the pruned experts they replace.Pruned experts had top-K logits while replacement experts did not, so their summed exponentiated logits are lower.Source: §A.3 · pp. 12–13

A.4 RENORMALIZATION ERROR IS SMALL

  • Assuming direct renormalization of old gate values, the renormalization error is generally small, so the derivation focuses on substitution error.The authors attribute this to replacing low-probability selected experts with experts just below the top-K cutoff.Source: §A.4 · p. 13

A.5 EXPANDING THE SQUARED SUBSTITUTION ERROR

The derivation expands squared substitution error into contributions from pruned, replacement, and retained experts, then upper-bounds it to obtain a tractable binary quadratic program.

A.5 EXPANDING THE SQUARED SUBSTITUTION ERROR

  • The squared substitution error separates into pruned-expert contributions, replacement contributions, and alignment between pruned and retained experts.Cross-terms in the pruned-expert component penalize jointly removing experts that reinforce each other; alignment with replacements contributes negatively to error.Source: §A.5 · pp. 13–14

A.6 UPPER-BOUNDING THE SUBSTITUTION ERROR VIA Z

  • The substitution-error bound is loose but strict and avoids needing replacement-expert information, making it tractable to optimize.This upper-bounding step supports minimizing the substitution error through the derived objective.Source: §A.5 · p. 13, §A.6 · p. 14

A.7 MINIMIZING Z AS A QUADRATIC PROGRAM

  • Minimizing the calibration-set objective over binary prune decisions yields a binary quadratic program for the target prune-set.Conditional normalization of F helps decouple interaction strength from co-selection frequency, so rarely co-selected but strongly contributing pairs are not undervalued.Source: §A.7 · p. 15

B SUPPLEMENTARY FIGURES AND TABLES

The supplementary results describe HOPE’s performance, non-random expert co-selection, and calibration behavior. The CHOPE appendix extends interaction-aware pruning across layers and describes how pruning errors propagate through the model.

B SUPPLEMENTARY FIGURES AND TABLES

  • HOPE is competitive or best across pruning rates, with the clearest separation at aggressive rates across the listed models, benchmarks, and calibration sets.Figure S1 compares mean Tulu and SWE-Bench Pro scores with the unpruned models.Source: Figure S1 · p. 16
  • Expert co-selection deviates from independence, especially at layer 0, supporting the presence of cooperative expert groups.The figure describes PMI and compares observed pairwise co-selection with the independence expectation.Source: B SUPPLEMENTARY FIGURES AND TABLES · p. 21
  • The Spearman correlation between HOPE’s diagonal scores and REAP scores is ρ = 0.988, confirming that REAP is a special case of HOPE.Source: B SUPPLEMENTARY FIGURES AND TABLES · p. 23
  • The mean binary-versus-continuous objective gap is 0.04%, compared with 65.6% for randomly sampled feasible binary solutions.The continuous solutions concentrate near 0 and 1, making rounding nearly lossless.Source: Figure S5 · p. 23, B SUPPLEMENTARY FIGURES AND TABLES · p. 23
  • HOPE and REAP pruning decisions stabilize within 10k calibration prompts, with HOPE requiring marginally more data to converge.Source: Figure S6 · p. 23

C CROSS-LAYER HOPE (CHOPE)

C.1 SETUP

  • After pruning a selected expert, the setup replaces it with the next-highest-logit expert for that token.The replacement count matches the number of pruned experts that would have been selected.Source: §C.1 · p. 24

C.2 ERROR PROPAGATION

  • The final-layer error includes both pruning error across MoE layers and error propagated from earlier layers; the derivation approximates each layer’s pruning error using clean inputs.Source: §C.2 · p. 25

C.3 UPPER-BOUNDING THE SUBSTITUTION ERROR

The cross-layer derivation bounds substitution error and minimizes a tractable objective over pruning decisions. Its quadratic program generalizes per-layer HOPE, while practical choices simplify propagation modeling and constrain layer-wise pruning.

C.3 UPPER-BOUNDING THE SUBSTITUTION ERROR

  • Theorem 4 bounds squared cross-layer substitution error by a factor times Z, and the factor is independent of the prune-sets, so minimizing Z suffices.Source: §C.3 · p. 27

C.4 MINIMIZING Z AS A QUADRATIC PROGRAM

  • CHOPE formulates joint prune-set selection as a quadratic program over binary decisions, using calibration-set expectations and a total pruning budget.Source: §C.4 · pp. 27–28

C.5 RELATIONSHIP TO PER-LAYER HOPE

  • CHOPE’s matrix includes within-layer and cross-layer interaction blocks; removing cross-layer terms recovers per-layer HOPE, and removing within-layer interactions further recovers REAP.Source: §C.5 · p. 28
  • A single global pruning budget lets CHOPE allocate pruning non-uniformly, potentially pruning more in layers with weaker cooperation or lower propagation impact.Source: §C.5 · p. 28

C.6 PRACTICAL CONSIDERATIONS

  • Computing propagation Jacobians at every layer is prohibitively expensive, so the practical simplification sets ΦL,j = I; per-layer minimum constraints can prevent over-pruning.The paper notes that this approximation is reasonable when residual-stream skip connections dominate.Source: §C.6 · p. 28

C.7 PRELIMINARY RESULTS

Preliminary experiments find that CHOPE’s non-uniform allocation can be consistent but does not outperform per-layer HOPE reliably. The appendix also describes calibration data, the QP solver, and physical checkpoint pruning.

C.7 PRELIMINARY RESULTS

  • CHOPE performed reasonably in many conditions but not as well as per-layer HOPE, and its pruned models occasionally experienced performance collapse.Tables S8–S9 compare the methods for two Qwen3.5 models using Evol-CA calibration.Source: Table S8 · p. 29, Table S9 · p. 29, §C.7 · p. 29
  • CHOPE consistently allocated pruning non-uniformly across trials but tended to prune many experts in early layers, which may introduce instability.The authors suggest limiting early-layer pruning as a possible way to improve performance and avoid collapse.Source: §C.7 · p. 29, Figure S7 · p. 30

D.1 CALIBRATION

  • Calibration used coding prompts or reconstructed SWE-bench trajectories, with three trials per model and calibration set using different generation seeds.The coding set contained 111k prompts; the agentic set contained 500 trajectories, with rare traces over 100k tokens truncated.Source: §D.1 · p. 31

D.2 QUADRATIC-PROGRAM SOLVER

  • HOPE’s binary QP was solved by continuous relaxation with SLSQP, then rounded by selecting the |P| entries with the highest values.The solver used [0, 1] bounds and a hard equality constraint on the sum.Source: §D.2 · p. 31

D.3 MODEL PRUNING

  • Pruning physically deletes expert MLP weights and corresponding router rows, producing a standard checkpoint for inference.Source: §D.3 · p. 31

D.4 TULU3 EVALUATION

The Tulu3-Dev evaluation uses eight tasks spanning math, instruction following, knowledge, reasoning, factuality, and coding.

D.4 TULU3 EVALUATION

  • Tulu3-Dev evaluation covers GSM8K, MATH, IFEval, MMLU, BBH, TruthfulQA, PopQA, and LiveCodeBench; Codex was dropped for insufficient variation between models.Models used an 8192-token maximum sequence length and a 2048-token generation budget per instance.Source: §D.4 · p. 31

D.5 SWE-BENCH PRO EVALUATION

The evaluation section specifies an agentic coding benchmark setup and a post-pruning fine-tuning protocol for 50%-pruned Qwen3.5-122B-A10B checkpoints.

D.5 SWE-BENCH PRO EVALUATION

  • SWE-Bench Pro evaluation measured mean resolve rate across 731 enterprise-level software engineering instances.The evaluation used the Harbor framework and OpenHands SDK agent, with the pruned model served via vLLM.Source: §D.5 · pp. 31–32
  • The evaluation used one attempt per instance, a 253952-token input limit, an 8192-token output limit, temperature 0.6, and top-p 0.95.These were also the parameters used for SWE-Bench Verified calibration on unpruned models.Source: §D.5 · p. 32

D.6 SUPERVISED FINE-TUNING

  • Post-pruning SFT compared HOPE and REAP checkpoints at 50% pruning using Evol-CodeAlpaca calibration and LiveCodeBench traces from the unpruned model.Full-parameter fine-tuning ran for 500 steps, totaling approximately 1B tokens; pre- and post-SFT evaluations used matched calibration trials.Source: §D.6 · p. 32

Related papers

Loading 2609.18916v3…