PaperlayerView exact revision on arXiv

Source-linked AI summary

RAZOR: Pruning Replaceable Experts in LLMs

Mingyang Song, Mao Zheng

arXiv:2609.30465v2cs.LGcs.CL

TL;DR

MoE pruning must reduce storage while preserving reasoning ability, but common frequency and magnitude scores do not capture whether surviving experts can replace deleted computation. RAZOR uses consensus-residual scores to estimate fixed-input deletion damage, including survivor reweighting and router refill. It leads the reported reasoning benchmarks across all model-budget settings, though pruning also changes generation behavior.

  • Problem

    MoE pruning must reduce expert-storage burden without eroding reasoning ability, while frequency and magnitude scores do not capture whether survivors can reproduce removed computation.

    Source: §1 · p. 1

  • Method

    RAZOR scores fixed-input single-expert deletion changes from consensus residuals, accounting for survivor renormalization and router refill, then aggregates scores over calibration tokens.

    Source: §1 · pp. 1–2

  • Results

    RAZOR had the highest nine-task macro average among evaluated methods in all eight model-budget settings and exceeded REAP by 2.12–5.59 points on two backbones.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Across four models and two removal budgets, replaceability-based pruning retained more reasoning ability at the same budget using forward passes alone.

    Source: §5 · p. 10

    On Qwen3.6-35B-A3B, pruned models still shift in response diversity, formatting, and termination; at 50% removal, length-limited finishes are more frequent than unpruned.

    Source: §1 · p. 2, §3.4 · p. 8

Abstract

from arXiv · show

Mixture-of-experts (MoE) models activate only a few experts per token yet store the entire expert pool. Whole-expert pruning shrinks that pool, but for reasoning models it must remove experts without eroding reasoning ability. Common scores rank experts by routing frequency or output magnitude, which measures isolated contribution rather than deletion damage. What decides the damage is functional replaceability, whether the surviving computation can reproduce what is removed. A large contribution may be replaceable by the remaining mixture, whereas a small one may carry a direction the survivors cannot recover. We introduce RAZOR, a training-free method that scores replaceability from consensus residuals, the deviations of individual expert outputs from their original weighted mixture. Holding the layer input fixed, these residuals yield the exact output change from deleting one expert, including survivor reweighting and the replacement expert promoted by router refill. RAZOR aggregates this change over calibration tokens and prunes to a layerwise budget using forward passes alone, without gradients, subset search, or recovery training. On GLM-4.7-Flash, Qwen3.6-35B-A3B, DeepSeek-V4-Flash-0731, and Hy3 at 25% and 50% expert removal, RAZOR attains the highest macro average over nine reasoning-centered tasks among the evaluated pruning methods in all eight model-budget settings. Against REAP on GLM-4.7-Flash and Qwen3.6-35B-A3B, it gains 2.12-5.59 points on this average and lowers reverse KL in all four comparisons. Retained accuracy is not the whole picture, as pruned Qwen3.6-35B-A3B still shifts in response diversity, formatting, and termination.

1 INTRODUCTION

The introduction frames reasoning-model pruning as preserving useful computation at a fixed expert budget: deletion damage depends on whether surviving experts can replace the removed computation. RAZOR estimates that damage and leads the reported reasoning benchmarks, while generation behavior remains a separate concern.

1 INTRODUCTION

  • MoE models activate only a small subset of experts per token, but sparse computation does not reduce the cost of storing the full expert pool.For reasoning models, pruning must preserve reasoning ability because errors can compound across long chains of dependent steps.Source: §1 · p. 1
  • Routing frequency and output magnitude miss how outputs combine: a large contribution may be replaceable, while a small one may carry an unrecoverable direction.Deletion damage depends on the computation available after removal, not isolated contribution magnitude.Source: §1 · p. 1
  • With router refill, deletion renormalizes survivors and promotes an unselected expert, so magnitude, fixed-support damage, and refill-aware damage can favor different deletions.Figure 1 contrasts deletion without refill, where proximity to the original mixture matters, with refill, where a promoted expert can compensate for a removed balancing contribution.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • RAZOR scores functional replaceability with consensus residuals, adding survivor renormalization and router refill to estimate fixed-input single-deletion changes.It aggregates scores over calibration tokens and prunes to layerwise budgets using forward passes alone, without gradients, subset search, or recovery training.Source: §1 · p. 2
  • RAZOR achieved the highest nine-task macro average among evaluated pruning methods in all eight model-budget settings, exceeding REAP by 2.12–5.59 points on two backbones.It also lowered reverse KL relative to REAP in all four matched model–budget settings.Source: §1 · p. 2
  • At 50% removal, all four backbones score below their originals, and Qwen3.6-35B-A3B still shifts in diversity, formatting, and termination.The introduction cautions that retained accuracy alone does not certify generation behavior.Source: §1 · p. 2

2 METHODOLOGY

The methodology treats pruning as minimizing local output change while retaining a layerwise expert budget. It derives fixed-input deletion counterfactuals from consensus residuals, then aggregates expert scores over calibration tokens to select retained experts.

2.1 PRUNING OBJECTIVE AND ROUTED MIXTURE

  • At a fixed budget, the objective is to preserve the original output distribution, using local Euclidean output change as a surrogate rather than a global guarantee.The consensus is the routed mixture; unchanged residual, dense, and shared-expert branches cancel in the local comparison.Source: §2.1 · pp. 2–3
  • Gates and output norms cannot identify the least harmful deletion because they omit output direction, which determines how survivors reproduce the removed computation.Two scalar examples with identical gates and norms yield different uniquely least-harmful deletions.Source: §2.1 · p. 3

2.2 FROM CONSENSUS RESIDUAL TO EXACT DELETION EFFECT

  • At fixed layer input, refill promotes the highest-ranked unselected expert and renormalizes its score with surviving scores; setting its pseudo-weight to zero recovers fixed-support deletion.The resulting single-deletion counterfactuals use routed outputs and, for refill, the promoted expert output without evaluating a separate end-to-end pruned model for each expert.Source: §2.2 · pp. 3–4
  • Single-deletion effects are not additive: residuals can reinforce or cancel, refill couples deletions, and deployment changes activations across layers.Thus matching the single-token refill rule does not guarantee checkpoint fidelity, and the local criterion remains a surrogate for joint pruning.Source: §2.2 · p. 4

2.3 EXPERT SCORING AND PRUNING

  • RCS, RCS-LOO, and RCS-Refill aggregate token-level quantities by RMS over routed calibration tokens and retain the highest-scoring experts within each layer’s budget.RAZOR names RCS-Refill under RMS aggregation; deployment uses top-k routing over the retained pools.Source: §2.3 · p. 4

3 EXPERIMENTS

The experiments evaluate predictive fidelity, nine-task performance, generation behavior, and the effects of scoring and aggregation choices across four MoE backbones and two removal budgets. RAZOR leads the macro benchmark comparison, while domain-level fidelity, generation metrics, and scoring diagnostics qualify that result.

3.1 EXPERIMENTAL SETUP

  • The evaluation compares residual criteria at 25% and 50% removal across four backbones without recovery training, using a nine-task suite that scores reasoning trajectories.Calibration uses 2,048 examples across seven domains; the task suite spans mathematics, coding, long-context inference, knowledge, instruction following, and tool use.Source: §3.1 · p. 5

3.2 PREDICTIVE FIDELITY (RQ1)

  • RAZOR has lower reverse KL than REAP in all four matched settings, but at 50% removal lowers KL in only 73 of 80 routing-stratified estimates and keeps PPL closer in 47.Across eight domains, the number favoring RAZOR varies by model and budget; Qwen3.6-35B-A3B’s 50% held-out mean is 0.2% higher than REAP’s, with an interval including zero.Source: §3.2 · pp. 6–7

3.3 DOWNSTREAM PERFORMANCE (RQ2)

  • RAZOR has the highest nine-task macro score among evaluated pruning methods in all eight model-budget settings, exceeding RCS-LOO by 0.74–2.58 points and REAP by 2.12–5.59 points where compared.The benchmark tables cover GLM-4.7-Flash and Qwen3.6-35B-A3B (Table 1) and DeepSeek-V4-Flash-0731 and Hy3 (Table 2).Source: Table 1 · p. 6, §3.3 · p. 7, Table 2 · p. 7
  • At 50% removal, all four backbones fall below their unpruned macro scores, although 25% RAZOR checkpoints remain close to or above the originals.For example, GLM-4.7-Flash scores 40.1 at 50% versus 44.8 unpruned; Qwen3.6-35B-A3B scores 58.7 versus 61.6.Source: §3.3 · p. 7

3.4 GENERATION BEHAVIOR (RQ3)

  • RCS-LOO has the highest Distinct-n in five of six budget-and-order combinations despite RAZOR’s stronger macro task performance.Under a 128-token prefix control that also changes question eligibility, the RCS-LOO–REAP Distinct-4 gap reverses, so the reversal cannot be attributed to length alone.Source: §3.4 · p. 8
  • At 50% removal, RAZOR emits 8.5 stray </think> delimiters per 10,000 completions versus REAP’s 13.4, but reaches the output cap more often than the unpruned model.Length-limited finishes are 25.8% for RAZOR, 24.2% for REAP, and 16.0% unpruned; loops account for under 1% of capped responses, leaving most cap finishes unexplained.Source: §3.4 · p. 8

3.5.1 SCORING COMPONENTS

  • Consensus subtraction lowers KL in all four matched settings, but LOO and refill refinements have model-dependent effects and a more complete local counterfactual need not improve the jointly pruned checkpoint.Under conditional RMS, adding LOO lowers GLM-4.7-Flash KL by 0.6%/5.6% but raises Qwen3.6-35B-A3B KL by 7.1%/11.8% at 25%/50% removal; refill improves on RCS-LOO in three of four settings.Source: §3.5.1 · pp. 8–9

3.5.2 EXPERT SELECTION

  • RMS changes which experts are retained, with model-dependent KL effects; fixed-support Gram diagnostics show reinforcement in four GLM layers for RAZOR but cancellation in all 46 GLM layers for RCS-LOO-selected sets.At 50% removal RMS retains an average of 11.0 experts per Qwen layer and 1.85 per GLM layer that mean would discard; these probes do not establish that swaps cause benchmark gains or that joint selection is optimal.Source: §3.5.2 · p. 9

4 RELATED WORK

Related work on MoE compression differs in what it removes, how it scores experts, and how it selects survivors. RAZOR scores whole-expert deletion using the fixed-input computation after renormalization and router refill, without recovery or retained-set search.

Expert scoring

  • Individual-expert methods use routing, activation, or weight statistics, while deletion-aware methods model post-removal computation; RAZOR scores the full fixed-input consensus-relative deletion-refill change.REAP includes substitution and survivor renormalization but retains the removed expert’s routing-weighted output norm; a unified framework derives exact damage but omits rerouting residuals from practical scores.Source: §4 · p. 10

Expert relations

  • Relational methods model shared expert structure through clustering, co-occurrence, parameter-space replaceability, or cross-layer transitions, whereas RAZOR scores deletion with renormalization and refill without recovery.The cited approaches include STUN, HC-SMoE, SHAPE, ConMoE, and MAESTRO (Lee et al., 2025; Chen et al., 2025; Zhang, 2026; Yao et al., 2026; Goel et al., 2026).Source: §4 · p. 10

Candidate-set search

  • Methods that evaluate retained sets use reconstruction, joint pruning-merging, or layerwise and blockwise search; RAZOR performs no subset search, and its exact single-deletion score does not solve joint pruning.Deletions can interact, and a refill candidate may itself be removed.Source: §4 · p. 10

5 CONCLUSION

The paper presents training-free RAZOR for expert pruning in reasoning MoE models, scoring replaceability through consensus residuals under survivor renormalization and router refill. It reports leading macro scores across eight model-budget settings, while noting that local fidelity, generation behavior, and retained-set stability limit what those results establish.

5 CONCLUSION

  • RAZOR achieves the highest nine-task macro average among evaluated methods across four post-trained MoE models and two removal budgets, and lowers reverse KL than REAP in all four matched comparisons.The matched comparisons use GLM-4.7-Flash and Qwen3.6-35B-A3B.Source: §5 · p. 10
  • Consensus residuals score functional replaceability under survivor renormalization and router refill, using forward passes alone.The conclusion states that the approach needs no recovery training and can be used in pipelines able to run calibration.Source: §5 · p. 10
  • The results do not establish that more faithful local counterfactuals always produce better rankings; response diversity, formatting, and termination also shift independently of reported gains.Selection under these criteria is less stable across calibration budgets than under REAP.Source: §5 · p. 10

AI USE STATEMENT

The authors report that generative AI tools assisted with academic prose refinement, literature search, and coding.

AI USE STATEMENT

  • Generative AI tools assisted with prose refinement, literature search, and coding.The stated prose goals were clarity, coherence, and precision of expression.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The appendix connects consensus residuals to fixed-input deletion damage, then explains why the resulting single-expert scores remain a ranking heuristic for joint pruning. It also describes calibration scoring and checkpoint execution, while cautioning that predictive fidelity does not establish safety or alignment.

ETHICS STATEMENT

  • Safety-related fidelity does not establish safety or preserved alignment, so independent evaluation is emphasized before deployment, particularly in safety-sensitive applications.The study uses previously released calibration data and involves no human-subject research.Source: ETHICS STATEMENT · p. 10

A.1 RESIDUAL DECOMPOSITION

  • The consensus residual encodes output alignment: positive inner products reduce it, while negative ones increase it, but self-inclusion still requires correction to measure distance to survivors.An aligned expert may be replaceable, while an expert balancing the others may be costly to lose.Source: §A.1 · p. 15

A.2 SURVIVOR-DISTANCE IDENTITY

  • With inputs and gates fixed, a common shift to all routed outputs leaves local deletion damage unchanged, even though output norms change.This translation invariance applies to local damage, not model predictions.Source: §A.2 · p. 16

A.3 EXACT PERTURBATION FOR A SELECTED SET

  • Single-expert deletion identities do not add into an exact joint objective because deletion sets introduce interference cross terms and a set-dependent denominator.Refill also makes the numerator and denominator depend on the whole removed set, leaving selection a fixed-budget ranking heuristic.Source: §A.3 · p. 16

A.4.1 ONLINE STATISTICS AND BOUNDED WORKING BUFFERS

  • RAZOR is RCS-Refill aggregated by conditional RMS; its scoring pass reconstructs the promoted expert as well as routed outputs, then accumulates token-level scores.The refill denominator is clamped at 10^-6, so an active clamp makes the computed score differ from exact local damage.Source: §A.4.1 · p. 17
  • Chunking reduces the working set for routed outputs from O(T(k + 1)d) to O(C(k + 1)d), with persistent RMS statistics requiring O(E) state per layer.The memory bounds exclude model weights and ordinary forward activations.Source: §A.4.1 · p. 17

A.4.2 LAYER-WISE EXECUTION

  • A layer-wise schedule loads and releases decoder layers sequentially, allowing the largest backbones to be scored on a fixed device budget without changing the score definition.The schedule propagates the calibration shard through each layer and collects its statistics before releasing the weights.Source: §A.4.2 · p. 18

A.4.3 ARCHITECTURE-SPECIFIC ROUTING

The implementation preserves each model’s native routing rule while normalizing selected scores for damage scoring, with a separate repair procedure for fixed-hash layers. Physical pruning slices expert parameters and router rows into smaller checkpoints, but the study does not measure serving acceleration.

A.4.3 ARCHITECTURE-SPECIFIC ROUTING

  • Native router selection is preserved across architectures, and selected unmodified scores are renormalized before evaluating the damage formula.Correction bias is not added to mixture weights; omitting the common positive routed-output scale leaves within-layer rankings unchanged.Source: §A.4.3 · p. 18
  • DeepSeek’s three fixed-hash layers retain frequent table experts and replace removed entries with cosine-nearest surviving router-weight vectors not already used in each row.This repairs routing rather than merging expert weights or solving an assignment problem.Source: §A.4.3 · p. 18

A.4.4 PHYSICAL PRUNING

  • Slicing expert-indexed tensors and matching router rows yields smaller checkpoints without runtime masks or custom sparse kernels; Qwen’s 256-to-128 reduction gives an approximately 19B-parameter model.Dense blocks and shared experts remain unchanged.Source: §A.4.4 · p. 18
  • Preserving top-k does not proportionally reduce active expert computation, and serving latency, energy use, and end-to-end acceleration are not measured.Source: §A.4.4 · p. 18

B DATA AND EVALUATION PROTOCOL

  • Only RAZORCAL influences expert retention; separate collections measure predictive fidelity, downstream performance, and response behavior.Fidelity and diversity collections each distinguish four calibration-covered axes from four held-out task types.Source: B DATA AND EVALUATION PROTOCOL · p. 18

B.1 CALIBRATION DATA

RAZORCAL is a 2,048-example calibration mixture drawn from existing instruction datasets across seven domains, with source sampling and preprocessing tailored to the pool. Its figure summarizes domain representation and character-length coverage.

B.1 CALIBRATION DATA

  • RAZORCAL contains 2,048 chat-format examples across seven domains and is assembled from existing instruction datasets rather than collected as a new corpus.Coding receives twice the allocation of each other domain.Source: §B.1 · p. 18
  • Examples undergo source-specific quotas, length filters, and deduplication, are capped at 32K characters, and are rendered with each checkpoint’s chat template before tokenization.The figure shows domain coverage over the full observed character-length range, with each dot representing one of the 2,048 examples.Source: §B.1 · p. 19, Figure 5 · p. 19
  • The coding allocation combines algorithm, competitive-programming, software-engineering, and tool-use examples from four instruction sources.The source mix includes 205 Python-algorithm examples, 141 Python and 115 C++ examples, 40 software-engineering examples, and 11 tool-use examples.Source: §B.1 · pp. 18–19
  • The remaining domains include mathematics, science, Chinese-STEM, instruction following, tool calling, and world knowledge, each allocated 256 examples.Mathematics uses high-reasoning-mode AoPS examples; the world-knowledge allocation uses a cleaned, non-thinking OpenHermes subset.Source: §B.1 · p. 19
  • Diagnostic calibration statistics include all nonpadding conversation positions, including prompts and role headers, and are compared within rather than across backbones because tokenizers yield different counts and source coverage.The reported token counts are 785,703 for Qwen3.6-35B-A3B and 785,632 for GLM-4.7-Flash.Source: §B.1 · p. 19

B.2 FIDELITY EVALUATION DATA

The fidelity protocol compares pruned and original predictions on reference prefixes across eight task axes, separating calibration-covered from held-out task types. Evaluation configuration and metric definitions specify the model settings and distinguish reference-token likelihood measures from other output statistics.

B.2 FIDELITY EVALUATION DATA

  • The fidelity pool has 8,192 conversations across eight axes, with four calibration-covered task types and four held-out task types, and scores assistant positions on fixed reference prefixes.Exact-message overlap with calibration is absent, but near-duplicates and benchmark contamination have not been ruled out.Source: §B.2 · p. 20

B.3 EVALUATION CONFIGURATION

  • For downstream benchmarks, sampling parameters are shared across all four backbones; evaluation retains original top-k and shared experts, with temperature 0 except SWE-bench Verified at 0.7.Qwen3.6-35B-A3B and GLM-4.7-Flash benchmark settings enable thinking. Fidelity and response diversity use their own protocols.Source: §B.3 · p. 21

B.4 METRIC DEFINITIONS

  • Reverse KL, reference-token NLL, and excess perplexity compare predictions on observed reference tokens, while predictive entropy summarizes the vocabulary distribution.Negative ∆NLL indicates higher reference-token likelihood under the pruned model, not necessarily better downstream capability.Source: §B.4 · p. 21
  • Absolute entropy drift averages predictive entropies by routing cell and then measures the absolute pruned–original difference across cells, distinguishing it from mean per-token absolute entropy change.Source: §B.4 · p. 21

B.5 DOWNSTREAM BENCHMARK PROTOCOL AND SCOPE

The benchmark compares pruning across four backbones and two removal budgets, while later analyses examine score definitions, benchmark scope, and how RMS aggregation changes expert selection. Results favor RAZOR overall in the recorded task comparisons, but individual capabilities and evaluation scope qualify that pattern.

B.5 DOWNSTREAM BENCHMARK PROTOCOL AND SCOPE

  • The benchmark covers four backbones at two removal ratios, but DeepSeek-V4-Flash-0731 and Hy3 are compared with unpruned references rather than REAP.LiveCodeBench’s “2026 latest” label denotes a rolling contest window, so absolute scores are not comparable with externally reported numbers.Source: §B.5 · pp. 21–23

C SCORING PROXIES AND EXPERT SELECTION

  • Frequency, EAN, and REAP differ in how they measure expert use or output magnitude, and EAN’s expert-specific normalization can change rankings.The scoring discussion separates token quantity from its reduction across tokens; conditional RMS, conditional mean, and corpus sum are distinct aggregation choices.Source: §B.5 · p. 22, C SCORING PROXIES AND EXPERT SELECTION · p. 23, §C.1 · p. 24

B.5 DOWNSTREAM BENCHMARK PROTOCOL AND SCOPE

  • The nine tasks score multi-step solutions across mathematics, instruction following, knowledge and tools, long context, and coding; Overall weights all nine tasks equally.Task-score averages and paired macro gaps use unrounded task scores; AIME’26 reports avg@8 and the other eight benchmarks report avg@3.Source: §B.5 · pp. 22–23
  • RAZOR records 36 wins, no ties, and no losses against REAP across two backbones and two budgets, but these comparisons reuse one task suite and report no task-level uncertainty.The counts summarize recorded task scores; they do not extend to the two backbones without a REAP benchmark run.Source: §B.5 · p. 23
  • At 50% removal on GLM-4.7-Flash, EAN exceeds RAZOR on LCB (33.9 versus 29.8), while favorable macro comparisons coexist with capability-specific trade-offs.At 25% on Qwen3.6-35B-A3B, RCS slightly exceeds RCS-LOO in macro average (61.0 versus 60.8); SWE has the largest proportional loss at 50% for RCS-LOO on Hy3 and DeepSeek-V4-Flash-0731.Source: §B.5 · p. 23

C.1 SCORING CRITERIA AND BUDGET COMPARISONS

  • Adding refill lowers reverse KL in three of four settings, whereas adding the leave-one-out denominator raises it on Qwen3.6-35B-A3B.The reported estimates pool tokens within each axis and weight eight axes equally.Source: §C.1 · p. 24

C.2 RMS AGGREGATION AND SCORE DISTRIBUTIONS

  • RMS-only selection exchanges differ between models, with Qwen3.6-35B-A3B showing more exchanges at both budgets; refill and fixed-support swap counts stay within one expert per layer.In Qwen3.6-35B-A3B layer 0, experts with conditional means 0.03323 and 0.03359 crossed the 192-expert retention boundary under RMS ranking.Source: §C.2 · p. 25

C.3 CONDITIONAL RMS, MEAN, AND CORPUS-SUM REDUCTIONS

Conditional RMS and conditioning on expert use are separate aggregation choices, and their effects vary across budgets and models. The refill analysis also finds that local score-profile summaries do not reliably predict checkpoint quality.

C.3 CONDITIONAL RMS, MEAN, AND CORPUS-SUM REDUCTIONS

  • Conditional RMS measures severity when an expert is used, whereas all-token RMS adds an exposure factor and targets total calibration error.The conditional choice does not minimize damage to capabilities the calibration set never exercises.Source: §C.3 · p. 26
  • For the RCS quantity, reverse KL follows RMS < mean < sum in all eight model–budget settings, while RMS improves over the conditional mean in six of eight settings.For the leave-one-out residual, RMS lowers KL in seven of eight settings, with the exception at 37.5% removal on Qwen3.6-35B-A3B (0.20504 versus 0.19196).Source: §C.3 · pp. 26–27
  • The reduction choice spans KL by 2.4× and 1.7× on Qwen3.6-35B-A3B at 25% and 50% removal, exceeding the 1.12× and 1.13× spans among REAP, RCS, and RCS-LOO.On GLM-4.7-Flash, the reduction span is comparable at 25% and smaller at 50%, motivating a factorial comparison of quantity and aggregation.Source: §C.3 · p. 26

C.4 WHAT THE REFILL TERM CHANGES

  • Refill shifts median scores in opposite directions: a 2.81% increase on Qwen3.6-35B-A3B and a 2.56% decrease on GLM-4.7-Flash.Despite these opposite median shifts, refill lowers KL at both Qwen budgets and at only one GLM budget; local summaries therefore do not predict checkpoint quality.Source: §C.4 · p. 27
  • Refill and fixed-support residual profiles have median cosine similarities of 0.9942 on Qwen3.6-35B-A3B and 0.9952 on GLM-4.7-Flash, but some experts show substantial directional changes.Minimum cosine values are 0.7062 and 0.5440, respectively; directional similarity alone does not fix expert rankings.Source: §C.4 · p. 27

C.5 REFERENCE, ROUTING FACTOR, AND AGGREGATION

The component study finds that scoring choices interact, so local exactness does not determine empirical ordering after joint pruning. Diagnostics and calibration studies further show limits in how local scores transfer and how stable retained sets are.

C.5 REFERENCE, ROUTING FACTOR, AND AGGREGATION

  • In all four mean-aggregation settings, refill beats REAP, while interactions prevent a uniform component ordering after joint pruning.The broader factorial comparisons likewise show that local exactness does not determine empirical ordering.Source: §C.5 · pp. 29–30

C.6 MATCHED COMPONENT CONTRASTS AT TWO REMOVAL BUDGETS

  • Consensus subtraction lowers reverse KL in every matched setting, while later refinements are backbone-dependent.RMS can lower reverse KL while raising excess perplexity on Qwen3.6-35B-A3B, so the proxy metrics do not share one ordering.Source: §C.6 · p. 30

C.7 SELECTION DIAGNOSTICS

  • Three diagnostics probe criterion mechanisms through selection-disagreement outputs, joint residual interactions, and deployed-routing single-expert ablations; none measures retained-set redundancy.The budgeted benchmark comparisons test retained-set redundancy indirectly through task utility.Source: §C.7 · p. 31

C.7.1 OUTPUT DIFFERENCES CONDITIONED ON SELECTION DISAGREEMENTS

  • Conditional on selection disagreements, tokens in the highest routing-concentration, consensus, or relative-impact bins tend to show the largest criterion gaps.These associations do not generalize to unconditioned tokens.Source: §C.7.1 · p. 31

C.7.2 INTERACTIONS WITHIN THE SELECTED DELETION SETS

  • At 50% removal, net cancellation appears in 29 of 40 Qwen3.6-35B-A3B layers and all 46 GLM-4.7-Flash layers for RCS-LOO-selected sets.Ratios are not straightforward quality signals because the Gram matrices omit removed routing mass.Source: §C.7.2 · p. 31

C.7.3 SINGLE-EXPERT REMOVAL WITH DEPLOYED ROUTING

  • RAZOR's deployed-routing single-expert damage correlations are 0.31 on Qwen3.6-35B-A3B and 0.18 on GLM-4.7-Flash, with no consistent advantage across models.The probe measures transfer from local output shifts to sample-wide prediction KL after router reselection.Source: §C.7.3 · p. 32

D CALIBRATION STABILITY AND SENSITIVITY

  • The budget study tests retained-set agreement using per-expert accumulators, while the corpus study measures reference-token perplexity using pruned checkpoints.Their differing costs determine which criteria each study covers.Source: D CALIBRATION STABILITY AND SENSITIVITY · p. 32

D.1 CALIBRATION BUDGET AND SELECTION STABILITY

  • At 50% removal, RCS-LOO disjoint-subset agreement rises from 85.72% to 93.39% as calibration grows from approximately 65,500 to 1.08 million tokens, but remains below REAP.The gap narrows but does not close; four subset-pair means without intervals or an ordering test cannot establish a plateau or explain the difference.Source: §D.1 · pp. 32–33

D.2 CORPUS CONFIGURATIONS AND EVALUATION COMPARABILITY

Corpus comparisons show different perplexity outcomes across backbones, but input matching limits prevent treating them as controlled calibration-source rankings. The broader fidelity analyses report aggregate gains alongside domain-level and response-property qualifications.

D.2 CORPUS CONFIGURATIONS AND EVALUATION COMPARABILITY

  • Self-generated calibration yields lower PPL than full RAZORCAL on matched Qwen3.6-35B-A3B inputs at both removal ratios, but the contrast also changes token budgets and conversation structure.Across the larger collection, self-generated calibration has lower PPL than tool-filtered calibration on Qwen3.6-35B-A3B but higher PPL on GLM-4.7-Flash at both budgets.Source: §D.2 · p. 33
  • Calibration-source comparisons reflect combined effects of source, token budget, and conversation structure rather than a controlled ranking of sources.The self-generated variants also change conversation structure, while later turns are not preserved.Source: §D.2 · pp. 33–34

E OUTPUT FIDELITY AND RESPONSE DIVERSITY

  • The aggregate fidelity gain coexists with axis-level regressions, weaker held-out fidelity on Qwen3.6-35B-A3B, and shifts in response diversity and generation formatting.The reported 95% intervals are pointwise and unadjusted for multiple comparisons.Source: E OUTPUT FIDELITY AND RESPONSE DIVERSITY · p. 34

E.1 PER-AXIS FIDELITY RESULTS

Per-axis comparisons show aggregate fidelity gains alongside identifiable regressions. Domain-wise results are favorable on most domains, with Qwen3.6-35B-A3B's held-out group a consistent weak spot.

E.1 PER-AXIS FIDELITY RESULTS

  • RCS-LOO has lower reverse-KL point estimates than REAP in 11 of 16 Qwen3.6-35B-A3B axis–ratio cells and 13 of 16 GLM-4.7-Flash cells.Four adverse cells remain, including creative writing on Qwen3.6-35B-A3B at both budgets; the comparisons decompose the same predictions rather than replicate the experiment.Source: §E.1 · p. 34
  • RAZOR has lower reverse-KL point estimates than REAP in 11 of 16 Qwen3.6-35B-A3B cells and 14 of 16 GLM-4.7-Flash cells, while three intervals remain above zero.Refill removes the adverse Qwen3.6-35B-A3B creative-writing interval at 25%, but the other three adverse cells remain.Source: §E.1 · p. 34

E.2 DOMAIN-WISE OUTPUT FIDELITY

  • At 50% removal, RCS-LOO has lower-KL estimates than REAP in 5 of 8 domains on Qwen3.6-35B-A3B, while its held-out equal-domain mean is 2.3% higher.Refill narrows this gap without closing it; neither view evaluates full generated trajectories.Source: §E.2 · pp. 34–35

E.3 ROUTING-STRATIFIED OUTPUT FIDELITY

This section examines output fidelity across routing groups and free generations while situating RAZOR’s importance-signal-only whole-expert pruning among alternative compression methods. It limits the conclusions of local single-deletion scores to comparisons that vary importance signals under fixed compression conditions, leaving combinations with other compression choices untested.

E.3 ROUTING-STRATIFIED OUTPUT FIDELITY

  • RAZOR leads five of eight routing panels by 1.2–4.3 points, while RCS leads three; refill’s fidelity effect varies by model and budget, not routing concentration.On Qwen3.6-35B-A3B OOD, RCS-LOO and RAZOR fall below REAP on average, whereas RCS improves on it.Source: §E.3 · pp. 35–36

E.4 REFERENCE-TOKEN PERPLEXITY ON THE SAME ROUTING GROUPS

  • Across 80 cells, RAZOR and RCS-LOO lower reverse KL than REAP in 73 and 72, respectively, but each preserves original PPL more closely in only 47, showing distinct fidelity objectives.On DeepSeek-V4-Flash-0731, negative ID NLL changes mean lower PPL can represent a larger deviation from the original.Source: §E.4 · p. 36, Figure 8 · p. 37

E.3 ROUTING-STRATIFIED OUTPUT FIDELITY

  • At 50% removal, held-out reverse KL is 2.1–3.0× its in-distribution value, while concentration widens RCS-LOO’s advantage by 15.6/17.2 points on Qwen ID/OOD and 4.3 points on GLM ID.Only these three endpoint contrasts have nominal paired-bootstrap intervals excluding zero, giving mixed evidence for a concentration-dependent advantage.Source: §E.3 · p. 35, Figure 8 · p. 37

E.5 RESPONSE DIVERSITY

  • At 50% removal, RCS-LOO exceeds REAP in all nine benchmark–order diversity cells, while refill lowers Distinct-n in five of six benchmark-averaged cells; prefix control reverses Qwen’s Distinct-4 gap.The Qwen3.6-35B-A3B RCS-LOO–REAP gap changes from +4.30% on full responses to −0.37% under prefix control; the change also reflects altered question eligibility.Source: §E.5 · pp. 38–39

E.6 RESPONSE DELIMITERS, FORMATTING, AND TERMINATION

  • Across ten RCS-LOO budgets, unfenced, length-limited, and stray-tag rates generally rise with removal; at 75%, Qwen produces 25.8% unfenced and 42.0% length-limited completions.At that budget, GLM-4.7-Flash has 78.7% length-limited completions and a median length of 30,444 characters.Source: §E.6 · pp. 39–40, Table 11 · p. 40
  • At 50% removal, Qwen3.6-35B-A3B RCS-LOO emitted stray </think> at 68.5 [22.8, 133.7] per 10,000 completions, versus REAP’s 13.4 [7.6, 20.1]; refill measured 8.5 [4.2, 13.6].On GLM-4.7-Flash, the corresponding RCS-LOO rate was 2.9 [1.1, 4.9] per 10,000; REAP’s rate was 1.6 [0.4, 2.7].Source: §E.6 · pp. 40–41

F.1 WHOLE-EXPERT SALIENCY AND CONDITIONAL AGGREGATION

  • RAZOR varies only the importance signal, pruning whole experts under fixed layerwise budgets without changing retained weights or routers, unlike frequency, activation, weight-concentration, or output-magnitude rankings.REAP scores weighted output magnitude; consensus residuals retain the removed expert’s direction relative to the mixture, while RCS-Refill also includes the promoted expert’s weighted residual.Source: F EXTENDED RELATED WORK · p. 41, §F.1 · pp. 41–42

F.2 EXPERT RELATIONS AND CANDIDATE-SET SELECTION

  • Other relational approaches cluster router behavior (STUN; Lee et al., 2025), cluster expert outputs (HC-SMoE; Chen et al., 2025), or use co-occurrence, prototype-reassignment, and routing-transition signals (SHAPE; ConMoE; MAESTRO).MAESTRO’s main results use attention-only LoRA recovery while experts and routers remain frozen.Source: §F.2 · p. 42
  • Single-deletion residuals measure output change at a fixed layer input, but do not capture joint-removal interactions or solve combinatorial expert-set selection.Residuals may reinforce or cancel under joint removal, so exact single-deletion quantities do not make the resulting ranking an exact set-selection algorithm.Source: §F.2 · p. 42

F.3 MERGING, ROUTER ADAPTATION, AND BUDGET ALLOCATION

  • The experiments use uniform layerwise removal, retain original expert weights, and perform no recovery training, so they do not establish benefits from combining scores with merging, router adaptation, or non-uniform budgets.Router KD is a recovery procedure rather than a ranking criterion; merging changes surviving functions, unlike router refill, which promotes an existing expert without averaging parameters.Source: §F.3 · p. 42

F.4 FINER-GRAINED COMPRESSION AND COMPARISON SCOPE

  • Finer-grained methods prune weights or intermediate neurons, so their reported sparsity levels are not matched whole-expert budgets.MoE-Pruner sparsifies within-expert weights, while HEAPr treats feed-forward intermediate neurons as atomic units rather than original router-selectable experts.Source: §F.4 · p. 43
  • The comparison fixes removal unit, layerwise budget, retained weights, and router, varying only the importance signal across RCS variants and magnitude- and frequency-based baselines.RAZOR denotes RCS-Refill under conditional RMS; the evidence addresses local signals for identifying replaceable experts, not their combinations with other compression choices.Source: §F.4 · p. 43
Loading 2609.30465v2…