Source-linked AI summary
Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking
Pradeep Kumar Sharma, Shantanu Godbole, Hritvik Shrivastava
TL;DR
The relative importance of MoE layers for selective expert removal remains unclear, limiting evidence for depth-aware compression. This paper conducts a multi-phase layer-wise masking study and finds that late layers are more tolerant, with held-out validation favoring very-late masking.
Problem
The relative importance of individual MoE layers and whether selective expert removal preserves output quality remain insufficiently characterized for compression.
Method
The paper conducts a multi-phase empirical study using regional ablations, policy discovery, and held-out validation on Qwen3.6-35B-A3B.
Results
419/500 Good+Similar outputs were retained by masking layers 35-39 at 50% while masking only 640 experts in held-out validation.
Takeaways & Limitations
The findings support a depth-aware principle that final MoE layers are the dominant near-term compression target, while flat all-layer masking is poor.
Takeaways & Limitations
The findings are limited to one model, Qwen3.6-35B-A3B, and one XLCoST code translation task.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation. Despite their widespread adoption, the relative importance of individual MoE layers remains insufficiently characterized, particularly for model compression. This paper presents a systematic layer-wise sensitivity analysis of the Qwen3.6-35B-A3B model (40 MoE layers, 256 experts per layer, top-8 routing) using magnitude-based expert masking on the XLCoST cross-lingual code translation benchmark. We conduct a multi-phase study spanning 100, 300, and 500 prompt evaluation scales across three H100 GPU servers. Our central finding is that layer sensitivity is strongly depth-dependent: early layers (0-9) and middle layers (10-29) are highly fragile to expert masking, while late layers (30-39), and especially very-late layers (35-39), tolerate aggressive masking of low-magnitude experts. Flat all-layer masking at 30% retains only 150/300 Good+Similar outputs at 300-prompt scale, whereas late-focused policies retain 249-255/300 while masking 640-1,145 experts. On a later 500-prompt held-out validation slice, the narrow very-late policy (layers 35-39 @ 50%) achieves the strongest quality/masked-expert tradeoff among tested candidates, retaining 419/500 Good+Similar outputs while masking only 640 of 10,240 total experts. We additionally characterize top-k routing width reduction from 8 to 6 active experts per token, which shows a large observed wall-clock reduction on a 100-prompt probe with no Good+Similar loss, though it does not yet compose cleanly with aggressive expert masking. These findings provide an empirical foundation for depth-aware MoE expert masking and establish a practical path toward physical weight surgery, activation-based expert scoring, and training-based recovery.
1. Introduction
This paper addresses the limited empirical characterization of layer-wise sensitivity in large-scale MoE systems through a multi-phase study of Qwen3.6-35B-A3B. It finds that sensitivity varies with depth and that late-focused masking preserves output quality better than uniform pruning.
- Motivation: The study targets open questions about whether MoE layers contribute equally, whether some depth regions are redundant, and whether selective expert removal can preserve output quality.These questions have direct implications for compression, deployment efficiency, and future architecture design.
- Depth-dependent sensitivity: Early layers (0-9) and middle layers (10-29) are fragile under expert masking, while late layers (30-39) are substantially more tolerant.This establishes depth-dependent sensitivity across the model’s MoE layers.
- Rejection of uniform pruning: At 300-prompt scale, flat 30% all-layer masking retains only 150/300 Good+Similar outputs.The result motivates rejecting uniform pruning schedules.
- Rejection of uniform pruning: At 300-prompt scale, late-focused schedules retain up to 255/300 outputs under expert masking.Late-focused policies therefore preserve more outputs than flat 30% all-layer masking in the reported comparison.
2. Background and Related Work
MoE models replace dense feed-forward sub-layers with routed parallel experts, evolving from single-expert to top-k and fine-grained decomposition. Because routing is independent across layers, different depth regions may develop distinct redundancy profiles, motivating layer-wise sensitivity analysis.
- MoE foundations: MoE layers replace dense feed-forward sub-layers with parallel expert networks controlled by a learnable routing mechanism.The paradigm traces back to decomposing complex problems into specialized subnetworks.
- MoE foundations: Modern MoE designs span Switch Transformer’s single-expert routing, Mixtral’s top-k routing, DeepSeek-MoE’s finer-grained decomposition, and Qwen3.6-35B-A3B’s 256 experts with top-8 routing.The Qwen-MoE configuration provides the architectural context for this study.
- Study motivation: Independent routing decisions at each layer allow different depth regions to develop different redundancy profiles, forming the study’s central hypothesis.This motivates testing masking sensitivity by layer depth rather than treating all layers identically.
- Pruning foundations: Sensitivity analysis evaluates how parameter changes affect model behavior, while magnitude-based pruning assumes smaller-magnitude parameters contribute less to predictions.Earlier methods linked removability to second-order loss contributions, and magnitude pruning has also been combined with iterative pruning and fine-tuning.
- MoE compression: MoE compression is complicated by input-dependent expert use, motivating studies of expert importance heterogeneity, structured pruning, redundancy and merging, and routing efficiency.The related approaches frame expert masking as part of a broader MoE compression literature.
- Evaluation setting: XLCoST supplies aligned code snippets across multiple programming languages, enabling assessment of structural validity and semantic fidelity in cross-lingual code translation.The task preserves algorithmic semantics across a non-trivial transformation and provides a sufficiently large test pool for evaluation.
3. Experimental Setup
Experiments use Qwen3.6-35B-A3B with controlled, reversible magnitude-based expert masking on deterministic XLCoST subsets. Primary evaluation measures baseline-relative Good + Similar retention, while routing heatmaps remain descriptive rather than causal.
- Experimental Setup: Qwen3.6-35B-A3B has 40 MoE layers, 256 experts per layer, top-8 routing, and approximately 3.3B active parameters out of 35B total.Runs use bfloat16 on NVIDIA H100 80GB HBM3 GPUs, with each run isolated to a dedicated two-GPU pair.
- Experimental Setup: The study evaluates deterministic XLCoST subsets at 10, 100, 300, and 500 prompts for validation, regional ablations, policy discovery, and held-out frontier-policy testing.Each sample is formatted as a translation instruction requesting only the target-language output.
- Experimental Setup: Good + Similar count is the primary quality-retention metric, comparing each masked generation with the unmasked baseline rather than measuring absolute reference correctness.Good requires normalized code exact match with syntactic validity; Similar requires syntactic validity and normalized code similarity to baseline of at least 0.85 in the applicable case.
- Experimental Setup: Phase-separated routing heatmaps aggregate selected expert frequencies across prompt, think, and answer tokens but do not by themselves prove causal importance.The primary generation-based scoring uses max_new_tokens = 1200 after diagnostic testing of alternative token budgets.
- Experimental Setup: Experts are ranked by ascending L2 norm of their feed-forward weight tensors, and the bottom-p% are selected for masking.Masking forces selected experts’ router logits to a large negative value before top-k selection, leaving weights intact and routing to remaining experts.
- Experimental Setup: Hard masking is the primary intervention, completely excluding selected experts, while soft masking applies router-logit penalties of -2.0 or -5.0 as an auxiliary branch.A mandatory check confirms masked_selection_hits_total == 0 for hard-mask experiments before quality conclusions.
4. Results
Results show that MoE routing and masking tolerance vary strongly with depth: late layers have diffuse routing and support substantially better quality–masking tradeoffs than flat or earlier-layer interventions. Auxiliary phase-aware routing further identifies overlapping, phase-skewed expert usage, while top-k reduction provides a promising but preliminary efficiency probe.
- Routing behavior: Late-layer routing confidence collapses, with the top-1 margin falling from approximately 0.024 at layer 20 to approximately 0.006 at layer 37.Early layers are diffuse, while mid-depth layers peak in routing concentration near layer 20.
- Phase-aware routing: Reasoning-phase routing is broad, answer-phase routing is sparser, and phase-skewed experts overlap in a shared core rather than forming disjoint sets.The auxiliary study used Qwen3.5 because Qwen3.6 thinking boundaries were not reliably recoverable, and it remained descriptive without causal ablations.
- Masking sensitivity: 249/300 G+S late-layer 40% masking exceeds 150/300 for flat all-layer 30% masking at 300-prompt scale.At 100 prompts, late-layer 40% masking achieves G+S = 80/100 versus 63/100 for all-layer 30% masking.
- Masking sensitivity: 250/300 G+S from layers 35-39 at 50% masks 640 experts, nearly matching the late ramp while masking 505 fewer experts.Adding masking below layer 30 consistently reduces quality relative to late-only policies.
- Held-out validation: 419/500 Good+Similar outputs from layers 35-39 at 50% establish the strongest tested held-out quality/masked-expert tradeoff.The policy masks 640 of 10,240 experts, or 6.25% of all experts.
- Routing-width probe: Top-k reduction from 8 to 6 causes no Good+Similar loss on a 100-prompt slice and shows a large observed wall-clock reduction, but remains a probe.Combining k=6 with aggressive late-layer masking introduces substantial quality concerns, and the timing comparison is not a controlled latency benchmark.
5. Discussion
The discussion supports depth-aware MoE masking: early and middle layers are fragile, whereas late layers may tolerate low-magnitude expert removal because their routing is more diffuse. The findings also establish important limits on interpretation, evaluation, generalization, and practical pruning claims.
- Interpretation: Early and middle layers build foundational semantic, syntactic, structural, and algorithmic representations whose perturbations propagate through later computation.Late layers operate closer to the output distribution, where near-maximal routing entropy and collapsed margins may make experts more interchangeable.
- Policy implications: 300-prompt discovery favored the late ramp, while the 500-prompt held-out validation favored the narrower very-late-only policy, which masked fewer experts.The comparison reframes the late ramp as one point on a frontier rather than establishing a universally optimal schedule.
- Policy implications: Useful masking policies should be depth-aware, with the safe region beginning much later than uniform global pruning assumes.The discussion presents depth awareness as the stronger supported claim, rather than universal optimality of one fixed schedule.
- Failure modes: 30% flat all-layer masking can be mechanically valid yet behaviorally unsafe, producing semantic drift, task-fidelity loss, nearby-but-incorrect solutions, and reasoning leakage.Reasoning leakage occurs when intermediate computation spills into the required code block, suggesting compounded representational damage in early and middle layers.
- Limitations: All experiments manipulate router logits at inference time rather than permanently removing weights, so physical pruning and actual memory savings remain untested.The findings are also limited to Qwen3.6-35B-A3B on one XLCoST code-translation task and may not generalize across architectures or domains.
- Limitations: Baseline-relative heuristic scoring, single-run evaluation without confidence intervals, and zero-shot masking constrain absolute quality claims and strict comparisons among close candidates.AST similarity, CodeBLEU, execution-based evaluation, human review, bootstrap intervals, and post-masking training are identified as needed extensions; reasoning-like artifacts also complicate extraction and phase-routing claims.
6. Conclusion and Future Work
The study finds that MoE expert-masking sensitivity is strongly depth-dependent, with early and middle layers fragile and late layers more tolerant. Results support depth-aware compression focused on final layers rather than a single universal pruning recipe.
- Conclusion: Expert-masking sensitivity is strongly depth-dependent: early and middle layers are fragile, whereas late and very-late layers tolerate substantial expert exclusion.The conclusion synthesizes regional ablations, 300-prompt policy discovery, and 500-prompt held-out validation.
- Conclusion: 255/300 Good+Similar outputs were achieved by the best 300-prompt late-ramp policy, masking layers 30-34 at 35% and layers 35-39 at 55%.This was the best policy identified during 300-prompt discovery.
- Conclusion: 419/500 Good+Similar outputs were achieved by the narrower held-out policy masking layers 35-39 at 50%, while masking only 640 experts.Held-out validation favored this very-late policy over the broader discovery policy.