PaperlayerView exact revision on arXiv

Source-linked AI summary

MASCing: Configurable Mixture-of-Experts Behavior via Activation Steering Masks

Jona te Lintelo, Lichao Wu, Marina Krček, Sengim Karayalçin, Stjepan Picek

arXiv:2604.27818v1cs.CR

TL;DR

MoE routing creates safety-control challenges, while retraining is costly when developers must rapidly configure behavior for changing objectives. MASCing uses surrogate-guided sparse routing masks to reconfigure behavior without retraining, improving both jailbreak defense and domain-specific compliance across seven open-source MoE models. Its routing-only intervention assumes the underlying experts already possess the capacity to produce safe responses.

  • Problem

    MoE safety behavior depends on sparse expert routing, while fine-tuning or retraining is costly for rapidly changing safety objectives.

  • Method

    MASCing trains an LSTM surrogate on routing logits, identifies behavior-relevant expert circuits, and applies sparse steering masks to routing gates without retraining.

  • Results

    Across seven open-source MoE models, average defense success rose from 52.5% to 83.9%, while adult-content generation success rose from 52.6% to 82.0%.

  • Takeaways & Limitations

    MASCing provides a lightweight mechanism for selectively strengthening or relaxing MoE safety behavior with negligible computational overhead while preserving general language utility.

  • Takeaways & Limitations

    MASCing assumes the model’s experts already have the inherent capacity to generate safe responses because it intervenes only at routing level.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) architectures in Large Language Models (LLMs) have significantly reduced inference costs through sparse activation. However, this sparse activation paradigm also introduces new safety challenges. Since only a subset of experts is engaged for each input, model behavior becomes coupled to routing decisions, yielding a difficult-to-control mechanism that can vary across safety-relevant scenarios. At the same time, adapting model behavior through full fine-tuning or retraining is costly, especially when developers need to rapidly configure the same model for different safety objectives. We present MASCing (MoE Activation Steering Configuration), the first framework that enables flexible reconfiguration of MoE behavior across diverse safety scenarios without retraining. MASCing uses an LSTM-based surrogate model to capture cross-layer routing dependencies and map routing logits to downstream behaviors. It then optimizes a steering matrix to identify behavior-relevant expert circuits and, at inference time, applies steering masks to the routing gates to override expert selection. This enables targeted enhancement or suppression of specific behaviors while preserving general language utility. To demonstrate its reconfigurability, we apply MASCing to two different safety-related objectives and observe consistent gains with negligible overhead across seven open-source MoE models. For multi-turn jailbreak defense, it improves the average defense success rate from 52.5% to 83.9%, with gains of up to 89.2%. For adult-content generation, MASCing enables models to comply with such requests that would otherwise be refused, increasing the average generation success rate from 52.6% to 82.0%, with gains of up to 93.0%. These results establish MASCing as a practical, lightweight, and flexible framework for scenario-specific safety reconfiguration in MoE models.

1 Introduction

MASCing is a lightweight, training-free framework for reconfiguring MoE safety through sparse routing intervention. It models routing-dependent behavior and applies steering masks to support both stronger jailbreak defense and selective policy relaxation.

  • MoE sparsity lowers inference costs but makes safety behavior depend on expert-routing decisions that can be manipulated to bypass alignment.Adapting safety through fine-tuning, alignment tuning, or retraining is costly, particularly when behavior depends on learned parameters and routing.
  • MASCing uses an LSTM surrogate to model routing patterns and cross-layer dependencies, then identifies behavior-relevant expert circuits.The framework optimizes a steering matrix and applies sparsity through L1 regularization and magnitude pruning.
  • 89.2% peak gains were achieved for multi-turn jailbreak defense while MASCing strengthened safety constraints.Average defense success increased from 52.5% to 83.9%.
  • 93.0% peak gains were achieved for domain-specific policy compliance while MASCing selectively relaxed safety boundaries.The application targeted adult-content generation under appropriate conditions.

2 Preliminaries

MoE models reduce inference computation by routing each token to a sparse subset of experts. Activation steering instead changes internal representations or routing-related components to influence target behaviors.

  • Mixture-of-Experts Architecture: MoE layers replace dense feed-forward networks with multiple experts and activate only a sparse subset for each token.This preserves large total parameter capacity while reducing active computation.
  • Mixture-of-Experts Architecture: The router converts per-expert logits into probabilities and uses top-k selection before combining the selected experts’ outputs.Only the k highest-scoring experts process each token.
  • Mixture-of-Experts Architecture: Standard MoE routes all experts through top-k selection, whereas Shared Expert MoE architectures divide experts into distinct structural roles.
  • Activation Steering: Activation steering intervenes on internal representations by adding vectors or targeting components such as attention heads, MLP layers, or expert routing.Its effectiveness depends strongly on intervention depth and steering strength.

3 Threat Model

The threat model represents developers who can inspect and modify deployed MoE models but need faster responses to changing safety requirements. Attackers have API-only access and cannot inspect internals or alter interventions.

  • Developers have full access to model inputs, activations, and weights while maintaining alignment with changing safety requirements.The setting reflects updates driven by new jailbreaks, regulations, or company policies.
  • Training-free interventions are important because developers may need to address new risks or policy changes faster than full retraining permits.
  • Attackers can submit prompts and observe outputs but cannot inspect internal states, modify weights, or revert developer interventions.This models public or hosted LLM services.

4 MASCing Framework

MASCing configures MoE behavior by learning routing-based behavioral circuits, sparsifying their steering masks, and applying those masks to gate logits during inference.

  • 4.1 Sequential Modeling of Behavior: MASCing uses an LSTM surrogate to map continuous routing logits to target behaviors while capturing temporal and cross-layer routing dependencies.The surrogate processes normalized, projected routing representations across tokens and network layers, then classifies whether a sequence leads to the target behavior.
  • 4.1 Sequential Modeling of Behavior: The surrogate is trained on routing-logit sequences with binary behavior labels using binary cross-entropy with logits loss.The trained model classifies whether an input routing sequence leads to a specified behavior.
  • 4.2 Steering Mask Creation: MASCing identifies behavior-relevant experts by optimizing a dense steering matrix through the differentiable surrogate and then pruning it into a sparse mask.The objective is a minimal expert subset because dense router manipulation can degrade untargeted language utility.
  • 4.2 Steering Mask Creation: Magnitude thresholding preserves positive components that boost target-aligned experts and negative components that suppress experts associated with opposing behavior.The resulting mask specifies which experts should be activated or deactivated.
  • 4.3 Steering Mask Application: During inference, pre-forward hooks add the static steering mask to gate logits with layer-wise scaling and amplitude control before standard top-k routing.The intervention modifies expert selection while confining the change to the routing mechanism.

5 Implementation and Evaluation Setup

The evaluation spans seven open-source MoE models, two opposing safety-configuration objectives, behavior-specific datasets, and utility benchmarks. Success rates measure target behavior, while MMLU and GSM8K assess whether masking affects general capabilities.

  • Dataset construction: Behavior-specific logit datasets combine AdvBench and Multi-Turn Human Jailbreaks for jailbreak defense with EroticaAnalysis and another dataset for adult-content generation.AdvBench contains 520 harmful behaviors, MHJ contains 2,912 prompts across 537 conversations, and EroticaAnalysis contains 14,886 instructions.
  • Models: Seven open-source MoE models are evaluated, including DeepSeek-MoE-16B-Chat, GPT-OSS-20B, Hunyuan-A13B-Instruct, Mixtral-8x7B-Instructv0.1, Phi-3.5-MoE-Instruct, Qwen1.5-MoE-A2.7B-Chat, and Qwen3-30B-A3B-Instruct-2507.Their architecture specifications are provided in Table 1.
  • Implementation: Implementations and evaluations use CUDA-enabled GPUs, including two NVIDIA GH200 120 GB Grace Hopper Superchips paired with NVIDIA H100 GPUs.The models and LSTM are implemented with PyTorch, Hugging Face Transformers, and Hugging Face Datasets.
  • Safety configuration scenarios: The safety evaluation covers opposing objectives: strengthening defenses against adversarial bypasses and selectively relaxing safety boundaries for specialized applications.The scenarios are multi-turn jailbreak defense and adult-content generation.
  • Evaluation metrics: Success rate measures the percentage of prompts or conversations producing the desired behavior, with human verification marking incoherent or nonsense outputs as unsafe.For defense, success measures safe responses on contexts that previously produced unsafe responses.
  • Utility evaluation: General utility is evaluated before and after masking on MMLU for broad knowledge and reasoning and GSM8K for multi-step mathematical reasoning.MMLU spans 57 subjects, and both benchmarks use 5-shot prompting.

6 Experimental Results

MASCing improves safety-task success across jailbreak defense and adult-content generation while altering expert-routing patterns and preserving most general utility. Its effectiveness depends on calibrated steering, and it outperforms direct expert steering for multi-turn defense.

  • Multi-turn Jailbreak Defense: 83.9% average defense success rises from 52.5% after MASCing across tested MoE models.Qwen3-30B-A3B-Instruct-2507 and GPT-OSS-20B approach 90% defense success.
  • Adult-Content Generation: 82.0% average adult-content generation success rises from 52.6% after MASCing.DeepSeek-MoE-16B-Chat, Mixtral-8x7B-Instruct-v0.1, and Qwen1.5-MoE-A2.7B-Chat were excluded because they already comply with these requests.
  • Steering Analysis: MASCing produces behavior-specific routing changes, with jailbreak defense showing more top-k expert-selection deviations than adult-content steering.Most models primarily activate experts associated with the desired behavior, whereas GPT-OSS-20B mixes activation and deactivation.
  • Hyperparameter Analysis: High α values collapse success by overriding natural expert selection, while low α values yield marginal gains; optimal performance lies in a narrow intermediate range.Excessive steering can produce incoherent, repetitive outputs, so intervention magnitude must preserve underlying language-model utility.
  • Hyperparameter Analysis: τ=0.1 consistently achieves the highest jailbreak-defense success rate across models, while larger τ values move performance toward the pre-steering baseline.The pattern is attributed to adaptive scaling that controls how many routing logits are modified.
  • Utility Evaluation: 4.1% average benchmark accuracy decline occurs after MASCing, without catastrophic forgetting or logical collapse on MMLU and GSM8K.The models retain factual recall, domain reasoning, and coherent multi-step mathematical generation.
  • Defensive Capability Comparison: 83.9% defense success for MASCing exceeds SteerMoE’s 58.4% average across evaluated multi-turn jailbreak scenarios.The comparison indicates a substantial advantage for MASCing on complex, multi-turn adversarial interactions.
  • Activation Steering versus Expert Steering: 69.0% success from direct expert steering is substantially below MASCing’s 83.9% defense success.The result supports using continuous routing information rather than relying solely on discrete expert selection.

7 Discussion

MASCing achieves low-overhead inference-time steering, but its effectiveness depends on surrogate fidelity, model capabilities, and the target distribution. Future work proposes dynamic, input-dependent masks and broader task applications.

  • Computational Cost and Overhead of MASCing: MASCing avoids MoE weight modification and costly full-parameter or LoRA fine-tuning, with negligible added latency.Its only training overhead is the lightweight LSTM surrogate model.
  • Limitations: The LSTM surrogate may produce sub-optimal mask optimizations when routing behavior is highly non-linear or unusually deep.The limitation arises because the surrogate approximates complex cross-layer routing dynamics.
  • Limitations: Routing steering assumes the model’s existing experts can generate safe responses, so it cannot synthesize safety when experts are poisoned or lack alignment data.MASCing changes routing without altering expert weights.
  • Limitations: Static steering masks may be less resilient to sophisticated out-of-distribution, zero-day jailbreaks that shift the model’s activation space.The masks generalize well to unseen prompts within the target distribution.
  • Future Work: Future work could use lightweight real-time classifiers to select steering configurations per prompt or token according to threat level.This would replace the current static steering matrix with dynamic, input-dependent masks.
  • Future Work: The task-agnostic framework could also be explored for domain adaptation, hallucination reduction, and conversational personality control.The paper gives medical and legal specialization as examples of domain adaptation.

8 Related Work

Related work studies safety through activation features, architectural components, and MoE expert activations. MASCing is situated among MoE attacks and defenses that localize or manipulate safety-relevant experts.

  • General LLM Safety Mechanisms: Prior work identifies safety-relevant activation directions, components, layers, attention heads, and neurons for analyzing or intervening on refusal behavior.These approaches range from representation engineering to targeted fine-tuning and component ablation.
  • MoE Expert Activations: MoE-specific research includes inference-time attacks that disable safety experts and compromise model alignment.GateBreaker uses a three-stage attack framework to identify and localize safety structures before disabling them.
  • MoE Defenses: SafeX localizes safety-concentrated experts and applies safety patches, but requires computationally expensive fine-tuning and leaves routing-based shortcuts unexamined.This positions SafeX as a training-based MoE defense.
  • MoE Defenses: SteerMoE operates at inference time without fine-tuning and controls behaviors by selectively activating or deactivating experts.It targets behaviors such as faithfulness rather than exclusively safety configuration.

9 Conclusions

MASCing is presented as a lightweight, training-free framework that uses surrogate-modeled routing dependencies and sparse gate-logit masks to configure MoE safety behavior. Across seven open-source models, it improves both defensive and permissive objectives with negligible overhead while preserving general utility.

  • Conclusion: MASCing maps continuous routing logits to behavioral circuits with an LSTM surrogate, then applies static sparse masks to routing gate logits.The masks override expert selection to enhance or suppress specific behaviors without retraining or inference delays.
  • Conclusion: MASCing provides a practical and adaptable mechanism for configuring MoE safety and alignment across diverse deployment scenarios.The reported scope includes both tightening security boundaries and selectively relaxing them.
  • Conclusion: Across seven open-source MoE models, MASCing improves multi-turn jailbreak defense success from 52.5% to 83.9% and adult-content generation success from 52.6% to 82.0%.The evaluation reports negligible computational overhead while preserving general language capabilities and utility.

A LSTM Training Results

The appendix reports validation accuracy for LSTM surrogates trained on datasets generated by the paper’s Section 5.1 method. The results indicate that the LSTM predicts behavior from routing logits with extremely high accuracy.

  • LSTM Training Results: The trained LSTM models achieve extremely high validation accuracy on predicting whether routing logits produce certain behaviors.These models generate the main results and use datasets created with the method described in Section 5.1.

B Additional Figures on Hyperparameter Analysis

This section reports additional hyperparameter-analysis results, including figures that compare success rates across α and λ settings and a table of LSTM validation accuracy.

  • Additional hyperparameter-analysis results are reported in Figures 6 and 7.
  • Table 7 reports validation accuracy for trained hierarchical and flat LSTMs.
  • Figure 6 plots jailbreak-refusal success rate against α across four MoE models and multiple λ penalty weights, with the pre-MASCing baseline shown for comparison.
  • Figure 7 plots adult-content-generation success rate against α across four MoE models and multiple λ penalty weights, with the pre-MASCing baseline shown for comparison.
Loading 2604.27818v1…