PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs

Yukun Jiang, Hai Huang, Mingjie Li, Yage Zhang, Michael Backes, Yang Zhang

arXiv:2602.08621v1cs.LGcs.AIcs.CR

TL;DR

MoE safety risks remain underexplored despite routing adding a distinct structural attack dimension. The paper introduces RoSais to locate safety-critical routers and F-SOUR to search token- and layer-specific unsafe routes. Across four representative MoE families, F-SOUR achieves average ASR of 0.90 on JailbreakBench and 0.98 on AdvBench.

  • Problem

    Prior work has largely focused on MoE utility and efficiency, leaving the safety risks of sparse routing underexplored.

    Source: §1 · p. 1

  • Method

    The paper uses RoSais to identify safety-critical routers and F-SOUR to search token-by-token and layer-by-layer for unsafe routes.

    Source: §1 · pp. 1–2

  • Results

    Across four representative MoE LLM families, F-SOUR achieves average ASR of 0.90 on JailbreakBench and 0.98 on AdvBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify MoE routing as a distinct attack surface and point to safety-aware route disabling and router training as defensive directions.

    Source: §6 · p. 8

    The evaluation covers one representative model per MoE family, and RoSais uses maximum affirmative-token log-probability gain rather than jointly suppressing refusals or optimizing answer targets.

    Source: G Limitations and Future Work · p. 14

Abstract

from arXiv · show

By introducing routers to selectively activate experts in Transformer layers, the mixture-of-experts (MoE) architecture significantly reduces computational costs in large language models (LLMs) while maintaining competitive performance, especially for models with massive parameters. However, prior work has largely focused on utility and efficiency, leaving the safety risks associated with this sparse architecture underexplored. In this work, we show that the safety of MoE LLMs is as sparse as their architecture by discovering unsafe routes: routing configurations that, once activated, convert safe outputs into harmful ones. Specifically, we first introduce the Router Safety importance score (RoSais) to quantify the safety criticality of each layer's router. Manipulation of only the high-RoSais router(s) can flip the default route into an unsafe one. For instance, on JailbreakBench, masking 5 routers in DeepSeek-V2-Lite increases attack success rate (ASR) by over 4$\times$ to 0.79, highlighting an inherent risk that router manipulation may naturally occur in MoE LLMs. We further propose a Fine-grained token-layer-wise Stochastic Optimization framework to discover more concrete Unsafe Routes (F-SOUR), which explicitly considers the sequentiality and dynamics of input tokens. Across four representative MoE LLM families, F-SOUR achieves an average ASR of 0.90 and 0.98 on JailbreakBench and AdvBench, respectively. Finally, we outline defensive perspectives, including safety-aware route disabling and router training, as promising directions to safeguard MoE LLMs. We hope our work can inform future red-teaming and safeguarding of MoE LLMs. Our code is provided in https://github.com/TrustAIRLab/UnsafeMoE.

1 Introduction

MoE’s sparse routing creates a distinct route-based safety risk that prior utility-focused and diagnostic work has not fully explored. The paper introduces RoSais to identify safety-critical routers and F-SOUR to discover unsafe routes, showing that manipulating routing can increase harmful outputs.

1 Introduction

  • MoE safety risks remain underexplored relative to utility and efficiency, while prior expert-level analyses are largely diagnostic and rely on prior data.The paper frames route itself as a structural jailbreak dimension beyond manipulating inputs or model internals.Source: §1 · p. 1
  • Masking 5 routers in DeepSeek-V2-Lite raises JailbreakBench ASR by over 4×, from 0.15 to 0.79.The result illustrates how a small number of router manipulations can reroute harmful questions toward unsafe outputs.Source: §1 · pp. 1–2
  • RoSais quantifies router safety importance, and manipulating a small number of high-RoSais routers can reroute harmful questions to unsafe routes.The paper reports that this manipulation increases ASR by more than 4×.Source: §1 · p. 2
  • F-SOUR searches token-by-token and layer-by-layer for concrete unsafe routes, achieving average ASR above 0.90 across multiple MoE families and benchmarks.It explicitly considers the sequentiality and dynamics of input tokens.Source: §1 · pp. 1–2
  • The authors identify a gap in safety approaches applied uniformly across architectures, because sparse routing introduces vulnerabilities that can be adversarially exploited.They propose this as motivation for exploring ways to build safe MoE LLMs.Source: §1 · p. 2

2 Background and Related Work

This section contrasts dense and sparse Transformer computation, surveys LLM safety attacks and defenses, and specifies the adversary’s access and route-manipulation capability.

2.1 Dense and Sparse Models

  • Dense Transformers apply a full FFN to each token, whereas MoE layers use a router to select top-k experts for each input, enabling sparse computation.MoE retains total parameter capacity through multiple expert parameterizations while executing only selected experts.Source: §2.1 · p. 2

2.2 LLM Safety

  • LLM safety research includes jailbreak attacks, refusal-mechanism analysis, training-time alignment, and deployment-time guardrails; MoE studies examine expert-level safety effects.SAFEx and SteerMoE rely on observational analysis and predefined unsafe or jailbreak data, while this work searches routing selections across tokens.Source: §2.2 · pp. 2–3

2.3 Threat Model

  • The adversary can submit arbitrary queries and access per-layer routing scores, then apply token- and layer-specific masks at inference to restrict Top-k selection.The threat model permits per-instance or universal manipulated routes without modifying expert weights or training data.Source: §2.3 · p. 3

3 Sparse Safety in MoE LLMs

This section frames MoE safety as sparse across routing decisions and introduces RoSais-guided router manipulation to discover unsafe routes. Experiments show that manipulating a small number of routers can increase harmful-output rates, while safety-critical routers concentrate in a few layers.

3.1 Router Safety Importance Score (RoSais)

  • RoSais estimates each router’s safety importance by measuring the best-case increase in affirmative-token probability under random routing-mask manipulations.The method samples masks because exhaustive enumeration of expert selections is infeasible.Source: §3.1 · p. 3

3.2 RoSais-Based Unsafe Route Discovery

  • The discovery procedure ranks routers by RoSais, selects the top-LΦ routers, randomizes their masks, and assembles the resulting manipulated route.Because shallow manipulations may alter deeper routers’ RoSais, the paper points to a finer-grained approach to mitigate this issue.Source: §3.2 · p. 4

3.3 Experimental Setup

  • Experiments evaluate four representative MoE model families on JailbreakBench and AdvBench, using ASR and deterministic generation at temperature 0.The setup considers question-specific sample-level routes and a universal dataset-level route.Source: §3.3 · pp. 4–5

3.4 Experimental Results

  • Safety-critical routers concentrate in a few layers; the largest dataset-level RoSais peaks are 8.60 for Mixtral-8x7B and 8.18 for OLMoE-1B-7B.AdvBench shows the same pattern of safety-critical layers concentrated in a few layers.Source: §3.4 · p. 5
  • On JailbreakBench, dataset-level manipulation raises DeepSeek-V2-Lite ASR to 0.79, about 5.3× its baseline, while changing one layer raises sample-level ASR from 0.16 to 0.49 on average.The results also report increases on AdvBench, with dataset-level ASR peaking at 0.90 for DeepSeek-V2-Lite.Source: §3.4 · p. 5, Table 1 · p. 6

4 Fine-Grained Unsafe Route Discovery

F-SOUR searches token by token and layer by layer for routing configurations that increase the likelihood of harmful targets. Across four evaluated MoE models, it achieves the highest JailbreakBench ASR, and it also attains near-perfect ASR on AdvBench.

4 Fine-Grained Unsafe Route Discovery

  • F-SOUR addresses RoSais-based discovery’s static next-token focus and layer-level granularity by modeling token-wise routing dynamics.RoSais-based measurement also overlooks few-token-depth alignment and interactions between shallow and deep routers after interventions.Source: §4 · p. 6

4.1 Our Proposed F-SOUR

  • F-SOUR searches routing scores sequentially for each input token and layer, retaining changes that increase the harmful target’s log-probability.It repeats random masking up to a set limit, and a shadow judge checks generated answers; unsafe answers are accepted, while others trigger a restart.Source: §4.1 · p. 6
  • F-SOUR progressively accumulates token-layer manipulations to maximize harmful-target likelihood, producing a more optimized unsafe route than the RoSais-based approach.Source: §4.1 · p. 6

4.2 Experimental Setup

  • The evaluation uses the same LLM, dataset, and ASR settings as Section 3.3, with up to 10 randomizations per token-layer pair and five attempts.The threshold is τ = log(0.8); comparisons include four white-box methods and one black-box method.Source: §4.2 · p. 6

4.3 Experimental Results

  • F-SOUR achieves the highest JailbreakBench ASR across four MoE models, reaching 0.94 on DeepSeek-V2-Lite and at least 0.86 on the others.It surpasses other attacks by 40% to 61%; Table 2 presents the ASR comparison.Source: §4.3 · p. 6, Table 2 · p. 7
  • On AdvBench, F-SOUR achieves ASR of at least 0.94 on all evaluated LLMs, following the same trend as on JailbreakBench.F-SOUR consistently yields higher ASR than RoSais-based methods, while RoSais-based methods are more cost-efficient because of their coarser granularity.Source: §4.3 · p. 7

5 Defensive Perspectives

The paper evaluates prompt adversarial tuning and Self-Reminder against its attacks, then discusses route disabling and safety coverage during training as defense directions.

5.1 Existing Defenses

  • Prompt adversarial tuning and Self-Reminder reduce ASR by up to 0.80 against RoSais-based attacks, but by at most 0.06 against F-SOUR.The paper characterizes them as effective against coarse-grained attacks but ineffective against F-SOUR.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Unsafe Route-Inspired Defenses

  • The authors attribute unsafe routes partly to safety concentrated in specific layers and partly to safety alignment concentrated in subsets of experts.They suggest that routing away from safety-carrying experts toward poorly aligned ones could create unsafe routes.Source: §5.2 · pp. 7–8
  • The authors propose disabling dataset-level unsafe experts in high-RoSais layers at inference and improving safety coverage for rarely activated experts during training.The training direction uses routing randomization or coverage objectives; the authors implement route disabling as a case study and leave the training direction for future work.Source: §5.2 · p. 8

6 Conclusion

The conclusion presents sparse safety as a structural vulnerability in MoE LLMs and summarizes RoSais and F-SOUR as tools for finding unsafe routes.

6 Conclusion

  • Unsafe routes are routing configurations that flip refusals into harmful outputs, and manipulating a few high-RoSais routers can sharply increase harmfulness.Source: §6 · p. 8
  • F-SOUR models input-token sequence and dynamics to discover concrete unsafe routes, achieving approximately 0.90 ASR across four MoE families and surpassing existing baselines.Source: §6 · p. 8
  • The paper identifies MoE routing as a distinct attack surface and outlines route disabling and safety-aware router training as potential defenses.Source: §6 · p. 8

Ethical Considerations

Experiments use public LLMs and datasets under controlled local conditions, and the appendix describes affirmative-token construction and baseline configurations.

Ethical Considerations

  • The study uses publicly available LLMs and datasets, runs experiments on local models under controlled conditions, and releases only minimal unsafe-content demonstrations.The authors note that their techniques could potentially be misused to elicit harmful outputs.Source: Ethical Considerations · p. 8

A Affirmative Tokens

  • Affirmative words are tokenized separately for each model, retaining only words that map to exactly one token for probability measurement.The word list is based on affirmative words provided in [48].Source: A Affirmative Tokens · p. 12

C Ablation Studies for RoSais

RoSais-guided router manipulation outperforms random masking, and the resulting unsafe routes transfer across datasets. Its computational cost grows primarily with activated parameters and routed layers, while measured attack time also rises with model size.

Comparison with Random Masking

  • RoSais methods increase ASR by up to 0.64 on JailbreakBench and 0.88 on AdvBench, versus random masking’s 0.09 and 0.18, respectively.The comparison attributes the larger ASR improvements primarily to RoSais-based routing manipulations rather than random masks.Source: C Ablation Studies for RoSais · p. 13

Transferability

  • Cross-dataset ASR decreases by less than 0.10 in most cases, with decreases ranging from 0.00 to 0.16; five-layer transfers reach 0.69 on JailbreakBench and 0.86 on AdvBench.The routes are evaluated on a different dataset from the one used to obtain them.Source: C Ablation Studies for RoSais · p. 13

Scalability

  • RoSais-based attack complexity is O(PAct · (S1 · LRouted + S2 · LΦ) · (N + 1)), primarily scaling with PAct · LRouted as model size increases.The passage treats S1, S2, LΦ, and N as generally independent of model size.Source: C Ablation Studies for RoSais · p. 13
  • Attack time increases with model size, but its percentage increase need not match the percentage increase in computational complexity, and the results suggest potential application to larger MoE LLMs.The runtime comparison used NVIDIA A100-SXM4-80GB GPUs and default settings S1 = 20, S2 = 100, and LΦ = 1.Source: C Ablation Studies for RoSais · p. 13

D Ablation Studies for F-SOUR

On DeepSeek-V2-Lite, F-SOUR remains effective when its randomization budget is reduced, and its ASR improves with a few shadow-judge attempts before stabilizing.

Impact of S3

  • On JailbreakBench, increasing S3 from 5 to 10 raises ASR from 0.92 to 0.94, then ASR fluctuates mildly; on AdvBench, ASR saturates at 1.00 for S3 ≥10.At S3 = 15 and 20, JailbreakBench ASR is 0.90 and 0.93, respectively.Source: D Ablation Studies for F-SOUR · p. 13

Impact of S4

  • Increasing S4 from 1 to 5 raises JailbreakBench ASR from 0.85 to 0.94 and AdvBench ASR from 0.90 to 1.00, after which results gradually stabilize.Source: D Ablation Studies for F-SOUR · p. 13
  • With the minimal attempt setting, S4 = 1, F-SOUR still achieves ASR of 0.85 on JailbreakBench and 0.90 on AdvBench.The authors identify this result as indicating computational efficiency.Source: D Ablation Studies for F-SOUR · p. 13

E Case Study: Disabling Dataset-Level Unsafe Route as A Defense

Disabling selected experts in the top-five RoSais routers reduces the ASR of GCG and TAP on both evaluated datasets, though the authors note possible utility trade-offs and adaptive attacks.

Defense results

Defense scope and caveats

  • The patch disables 6 of 64 experts in each of 5 of 25 MoE layers, but may trade off utility and can be compromised by adaptive attacks.The authors describe combining route disabling with training-time safety coverage and router regularization as a promising next step.Source: E Case Study: Disabling Dataset-Level… · p. 13

F Case Study: Utility

Dataset-level RoSais attacks and defenses generally preserve the reported general-QA utility metrics, but changing five layers can substantially reduce math accuracy. The authors connect this difference to task sensitivity to routing choices and identify utility maintenance as a challenge for dataset-level interventions.

Utility outcomes

  • After dataset-level RoSais attack and defense, TruthfulQA informativeness and truthfulness generally decrease by ≤0.03, while changing five layers causes GSM8K accuracy drops exceeding 0.25 and 0.05, respectively.The accuracy drops exceeding 0.25 and 0.05 are reported for attack and defense, respectively.Source: F Case Study: Utility · pp. 13–14

Task sensitivity

  • The authors report that mathematics is more sensitive than general QA to routing choices, and that forcibly selecting or removing experts can reduce utility.General QA is described as more robust to routing manipulation.Source: F Case Study: Utility · p. 14

Implications

  • Maintaining utility while activating or disabling unsafe routes is a challenge for dataset-level interventions; sample-level F-SOUR can produce higher-quality harmful answers and higher ASR.The authors point to the relationship between task-specialized experts and unsafe experts as future work.Source: F Case Study: Utility · p. 14

G Limitations and Future Work

The study limits its evaluation and attack design to expose MoE architectural vulnerabilities, while identifying further model, objective, manipulation, mitigation, and domain directions for future work.

G Limitations and Future Work

  • Evaluation covers one representative model per MoE family, and RoSais uses affirmative-token log-probability gain rather than multi-objective or target-aware signals.The authors note that broader model coverage and richer RoSais objectives remain worth exploring.Source: G Limitations and Future Work · p. 14
  • Masking router scores before Top-k selection leaves direct score tuning, such as gradient steering, untested and may understate the attack surface.Source: G Limitations and Future Work · p. 14
  • F-SOUR uses bounded randomizations and attempts; ablations suggest larger or adaptive budgets could discover better unsafe routes, while white-box routing-score access may limit closed-API applicability.Source: G Limitations and Future Work · p. 15
  • The authors frame internal router access as intrinsic to route-level study and complementary to black-box attacks; unsafe routes also motivate exploring stronger black-box attacks.Source: G Limitations and Future Work · p. 15
  • Future work includes scaling to more MoE models, broadening RoSais objectives, finer routing manipulations, adaptive F-SOUR budgets, mitigations, and domains such as hallucination and bias.Suggested mitigations include safety-aware router training, route auditing, and randomized or hardened routing.Source: G Limitations and Future Work · p. 15
Loading 2602.08621v1…