Source-linked AI summary
Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM
Md Nurul Absar Siddiky, Liuwan Zhu, Yingfei Dong
TL;DR
Activation frequency measures expert use, not influence, leaving open which experts are better suppression targets for weakening MoE refusal behavior. The paper compares router-gradient and activation rankings across five architectures under expert-count and routing-traffic budgets; gradient selection reduces refusals more often than activation and random selection in the reported conditions.
Problem
Activation frequency is commonly used to rank MoE safety experts, but it measures use rather than influence, and the better suppression targets are not known in advance.
Source: §I · p. 1Method
The study ranks experts by router-gradient sensitivity and activation on benign and malicious prompts, then evaluates suppression against held-out malicious prompts under matched expert-count and nominal routing-traffic budgets.
Source: §I · p. 1, B. Activation and Router-Gradient Scores · p. 2Results
Under each tested budget, gradient selection reduces refusals more than activation in 24 of 25 conditions and more than the ten-trial random mean in all 25.
Source: B. Held-Out Suppression · p. 4Takeaways & Limitations
With expert counts matched in every layer, gradient selection produces greater refusal reduction than activation in 23 of 25 conditions and ties in two.
Source: §V · p. 5The traffic-target comparison is a complementary check, not an independent replication, because only activation and random selections are rebuilt to those targets.
Source: B. Held-Out Suppression · p. 4
Abstract
from arXiv · showhide
Suppressing a small set of routed experts can weaken the safety behavior of a sparse Mixture-of-Experts (MoE) language model without retraining. Which experts to suppress is therefore a security question, and the usual answer is activation frequency, but frequency measures use, not influence. We test an alternative: router-gradient sensitivity, the sensitivity of the sequence loss to the gate weights that select an expert. Across five MoE architectures, we rank experts by each signal on 500 benign and 500 malicious prompts and measure refusal on 100 held-out malicious prompts under two budgets: equal expert counts and equal nominal malicious routing traffic (1%-5%). Under each of the two budgets, router-gradient selection reduces refusals more than activation in 24 of 25 conditions, and more than a ten-trial random mean in all 25. The largest effect is in OLMoE, where refusals fall from 34 to 9 of 100 prompts (73.53% relative) with no degraded outputs, indicating substantive compliance rather than broken generation. After matching expert counts in every layer, gradient selection still produces greater refusal reduction than activation in 23 of 25 conditions, with two ties. An exploratory cross-model analysis links larger malicious-versus-benign concentration gaps to greater peak gradient effects (rho = 0.90; exact two-sided p = 0.083, n = 5). Together, the results support gradient selection under the tested budgets.
I. INTRODUCTION
Sparse MoE routing makes expert suppression a security question, but activation frequency measures use rather than influence. The paper tests whether router-gradient sensitivity can identify better suppression targets than activation or random selection.
I. INTRODUCTION
- Router choices are discrete and per-token, so suppressing selected experts can weaken refusal behavior without retraining.The choice of which experts to disable matters to both attackers and defenders.Source: §I · p. 1
- Activation frequency measures how often malicious prompts use an expert, not how strongly that expert influences safety behavior.Frequently selected experts may matter little to a behavior, while rarely selected experts may be pivotal.Source: §I · p. 1
- The study compares gradient, activation, and random expert selection across five instruction-tuned models using separate analysis and held-out evaluation prompts.The two budget protocols control for expert count and routing traffic.Source: §I · p. 1
- The contribution includes an exact per-layer-count control to test whether gradient selection's advantage is explained by layer placement.The analysis also explores whether malicious-versus-benign routing concentration gaps relate to peak suppression effects.Source: §I · p. 1
II. RELATED WORK
Prior MoE work studies routing, specialization, safety evaluation, and expert-level interventions, with most safety-oriented selection based on activation or masking. This study compares router-gradient sensitivity with activation frequency for suppression under matched budgets and a random baseline.
II. RELATED WORK
- Prior MoE research examines conditional computation, load balancing, token-level specialization, interpretable architectures, steerable routing, and activation-magnitude “super experts” –[3], –.These studies motivate treating an expert by its layer and in-layer index, while leaving router-parameter sensitivity distinct from frequency.Source: §II · p. 1
- Safety evaluations use AdvBench and JailbreakBench, while StrongREJECT motivates separating restricted, compliant, and degraded outputs [18]–.Dense-model work also reports low-dimensional refusal mechanisms.Source: §II · p. 1
- Prior MoE safety interventions mainly use activation strength or masking, and the paper identifies its comparison of gradient and activation selection under matched budgets with a random baseline as novel.Related work includes gradient sensitivity for backdoor-expert localization rather than safety-behavior suppression.Source: §II · p. 2
A. Models, Analysis Sets, and Evaluation Sets
The study uses prompt sets for routing analysis and separate, non-overlapping sets for behavioral evaluation. Responses in the behavioral evaluation are manually categorized by safety and output quality.
A. Models, Analysis Sets, and Evaluation Sets
- Each model's analysis set contains 500 benign AEGIS2.0 prompts and 500 malicious AdvBench prompts [18].These prompts are used for routing analysis, malicious-minus-benign scoring, ranking, and traffic calibration.Source: A. Models, Analysis Sets, and… · p. 2
- Behavioral evaluation uses additional AEGIS2.0 sets of 100 malicious and 100 benign prompts [25], with no overlap with the analysis set.The malicious set evaluates baseline and suppression methods; the benign set audits gradient-based suppression.Source: A. Models, Analysis Sets, and… · p. 2
- Responses are manually labeled as restricted, compliant, or degraded, with degraded outputs including empty, incoherent, repetitive, or unrelated responses.The labels are used only for behavioral evaluation and do not influence expert identification.Source: A. Models, Analysis Sets, and… · p. 2
B. Activation and Router-Gradient Scores
The section defines activation and router-gradient signals for ranking experts by malicious-versus-benign differences. Gradient scoring uses sequence-loss derivatives for router-gate weights and requires a more costly one-time profiling pass.
B. Activation and Router-Gradient Scores
- Activation counts tokens routed to each expert within a sparse layer and averages those counts over benign or malicious prompt sets.The prompt groups are denoted benign and malicious.Source: B. Activation and Router-Gradient Scores · p. 2
- The prompt-level gradient score is the mean absolute sequence-loss gradient for a router row, collected without updating parameters.The score measures local sensitivity of formatted-prompt token prediction to router weights, not a refusal objective or direct safety gradient.Source: B. Activation and Router-Gradient Scores · p. 2
- Gradient and activation targets are ranked by their respective malicious-minus-benign scores.Gradient malicious and benign scores are averaged over the corresponding 500-prompt groups.Source: B. Activation and Router-Gradient Scores · p. 2
- Gradient profiling requires a backward pass per analysis prompt, unlike activation frequency's forward pass, but adds no inference-time cost after target selection.The absolute value in the gradient score prevents cancellation.Source: B. Activation and Router-Gradient Scores · p. 2
C. Suppression Budgets and Evaluation
The evaluation compares gradient-selected experts with activation and random selections under expert-count and malicious-routing-traffic budgets, then tests an activation control matched to gradient selection’s layer counts.
C. Suppression Budgets and Evaluation
- Gradient-ranked sets target approximately 1%–5% of malicious routing traffic, with each set built by accumulating expert shares and excluding the first expert that would exceed its target.The sets contain different expert counts across models and targets.Source: C. Suppression Budgets and Evaluation · pp. 2–3
- The same-count comparison suppresses equal numbers of gradient-, activation-, or randomly selected experts, but does not match cumulative traffic or layer-wise distribution.Source: C. Suppression Budgets and Evaluation · p. 3
- The traffic-target comparison matches nominal 1%–5% routing traffic across methods, while expert counts and layer-wise distributions may still differ.Achieved traffic can fall below nominal targets because experts are discrete.Source: C. Suppression Budgets and Evaluation · p. 3
- The depth-matched activation control selects exactly as many experts as the gradient set in each layer, matching total count and the full layer-count distribution.Routing traffic is not additionally matched in this control.Source: C. Suppression Budgets and Evaluation · p. 3
- Selected experts receive a negative router-logit bias before Top-k selection, and restricted responses are evaluated on the same 100 malicious prompts.Positive relative refusal reduction indicates fewer restricted responses after suppression.Source: C. Suppression Budgets and Evaluation · p. 3
- Random selection averages restricted-response counts across ten independently seeded trials per model and protocol.Sampling is without replacement and retains enough experts per layer for native Top-k routing.Source: C. Suppression Budgets and Evaluation · p. 3
A. Routing Concentration
Malicious prompts show stronger model-wide routing concentration in all five models, yet benign and malicious expert usage substantially overlaps. Within-layer concentration differences are less uniform.
A. Routing Concentration
- The analysis compares model-wide and within-layer concentration and expert overlap; k90 counts experts covering 90% of activation mass, while effective-expert counts derive from per-prompt entropy.Effective-expert counts are averaged over prompts and layers, and lower values indicate more concentrated usage relative to the available pool.Source: A. Routing Concentration · p. 3
- 4–7 experts overlap between benign and malicious Top-10 sets despite stronger malicious routing concentration in all five models; Jensen–Shannon divergence remains 0.0074–0.0408.Source: A. Routing Concentration · p. 3
- Malicious effective-expert ratios are lower in four models, with the largest differences in OLMoE and DeepSeek; Qwen1.5 is essentially unchanged.Greater model-wide concentration does not imply stronger concentration within every layer.Source: A. Routing Concentration · p. 3
- Layer profiles vary: Mixtral has 6.906 versus 7.002 effective experts at L8–15, while Qwen1.5 has 40.2 versus 39.8 at L23 for malicious versus benign prompts.OLMoE has lower malicious effective-expert counts at L3 and L14, alongside a shared minimum at L6; Qwen3 shows smaller, mixed group differences.Source: A. Routing Concentration · p. 3
B. Held-Out Suppression
On held-out malicious prompts, gradient-selected suppression reduces refusals more than activation and random selection under both tested budgets. Its advantage also persists after exact per-layer expert counts are matched.
B. Held-Out Suppression
- 24 of 25 conditions favor gradient over activation under each budget, and gradient beats the ten-trial random mean in all 25.The comparisons use matched expert counts and common nominal routing-traffic targets.Source: B. Held-Out Suppression · pp. 3–4, Figure 1 · p. 4
- 73.53% at K = 21 is OLMoE’s largest relative refusal reduction, with restricted responses falling from 34 to 9 and no degraded outputs.Activation and random suppression increase OLMoE refusals at every setting.Source: B. Held-Out Suppression · pp. 4–5
- The traffic-target comparison is complementary, not an independent replication, because gradient sets and results are reused while activation and random sets are rebuilt.Traffic targets span 1%–5%.Source: B. Held-Out Suppression · p. 4
- Gradient-selected experts are deeper on average than activation-selected experts: 68.2% versus 50.9% of sparse-layer depth under same-count budgets.Gradient sets are deeper in all 25 settings; settings are not independent replications because budgets reuse models and overlapping sets.Source: B. Held-Out Suppression · p. 4
- 23 of 25 conditions favor gradient over exact layer-count-matched activation, with ties for Qwen3 at K = 10 and Mixtral at K = 11.At the largest budgets, DeepSeek reductions are 28.05% versus 4.88%, and OLMoE reductions are 73.53% versus −29.41%.Source: B. Held-Out Suppression · p. 4, Table III · p. 5
- Spearman ρ = 0.90 (exact two-sided p = 0.083) links concentration gaps with peak gradient effects across five architectures, but does not establish causation.The exploratory association uses maxima across tested budgets and relative reductions normalized by differing baseline refusal counts.Source: B. Held-Out Suppression · pp. 4–5
V. CONCLUSION
This study finds that activation frequency is not a consistently reliable proxy for suppression-induced behavioral effects in sparse MoE models. Across five architectures, router-gradient sensitivity identifies targets that reduce refusals more effectively, including after exact per-layer counts are matched.
V. CONCLUSION
- Across five architectures, malicious inputs show greater routing concentration, while gradient sensitivity identifies suppression targets that reduce refusals more effectively than activation or random selection.The largest effect is a 73.53% relative reduction in OLMoE; exact per-layer-count matching favors gradient in 23 of 25 conditions and ties in two.Source: §V · p. 5
- The exploratory association between malicious-versus-benign concentration gaps and peak gradient effects is suggestive, but requires validation across more models.The analysis covers five architectures.Source: §V · p. 5
- The study aims to support identifying and hardening safety-sensitive experts; the mechanism behind larger-budget non-monotonic effects and general capability preservation remain future work.The reported refusal reductions do not establish general capability preservation.Source: §V · p. 5