Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models

1 Introduction

• VLM safety risks can emerge from visual-text interactions, so assessing either modality alone may miss harmful intent.The introduction gives the example of an innocuous scene paired with a seemingly benign question that becomes harmful in combination.Source: §1 · p. 1
• Prior MoE safety work manipulates routing patterns as control handles, but has primarily studied text-only safety, leaving cross-modal risks unclear.Related work identifies unsafe experts, steers expert activation, and analyzes unsafe routing paths [Lai et al., 2025; Fayyaz et al., 2026; Jiang et al., 2026; Xu et al., 2026].Source: §1 · p. 2
• Prior interventions improve safety inconsistently, motivating observation of routing states rather than direct manipulation of model internals.The authors argue that a signal may be informative even when it is difficult to manipulate for reliable behavioral control.Source: §1 · p. 2
• The detector reads router logits during prompt prefill to identify unsafe multimodal requests without modifying model parameters or routing behavior.Across Qwen3-VL and Kimi-VL, it reduces HoliSafe overall error and generalizes to out-of-distribution safety benchmarks.Source: §1 · p. 2
• Naturally emerging internal signals can be linked to external safety mechanisms without modifying the underlying model.The paper presents this as a broader perspective on model internals and safety.Source: §1 · p. 2

2.1 Multimodal Safety Setting

• The safety taxonomy classifies requests by image, text, and joint image–text intent, including cases where risk appears only through their interaction.The setup evaluates Qwen3-VL-30B-A3B-Instruct and Kimi-VL-A3B-Instruct on HoliSafe, with 1,000 training and 1,000 test examples per category.Source: §2.1 · pp. 2–3

2.2 Current Safety Interventions

• The study compares input-level safety prompting, routing-level expert steering, and parameter-level supervised fine-tuning.Prompting varies whether safety instructions emphasize image, text, or both; steering modifies router logits toward safety-associated experts; fine-tuning uses image-oriented, text-oriented, or combined supervision.Source: §2.2 · p. 3

2.3 Safety Interventions Fall Short

• On Qwen3-VL, prompting lowers mean ASR from 41.25% to 24.04% but raises RFR from 3.00% to 22.33%; on Kimi-VL, SFT lowers mean ASR from 84.00% to 51.09% but raises RFR from 3.00% to 13.67%.No evaluated intervention consistently achieves low attack success and low false-refusal rates across both model families.Source: §2.3 · p. 4

3.1 Router-Logit Detector

• The detector concatenates router logits at the final prompt token across MoE layers and applies an external logistic-regression safety readout.It classifies requests after prompt prefill; requests above the selected unsafe-probability threshold are routed to a fixed refusal policy, while others continue through the original generation pipeline.Source: §3.1 · p. 4

3.2 Effectiveness of Router-Logit Detection

• HoliSafe overall error falls from 33.60 to 4.40 on Qwen3-VL and from 67.80 to 5.40 on Kimi-VL with router-logit detection.These results outperform the reported best prompting, SFT, and expert-steering configurations in both model groups.Source: §3.2 · p. 5, Table 1 · p. 5
• At the selected threshold, SSS false-refusal rates are 5.50 on Qwen3-VL and 12.00 on Kimi-VL, while USU ASR falls to 14.50 and 13.50, respectively.The operating point favors fewer false refusals on safe requests.Source: §3.2 · p. 5
• On benign SSS inputs, image-only safety prompting induces the largest router-logit shifts relative to the base prompt, particularly in middle and late layers, and is associated with substantially higher false-refusal rates.Source: Figure 3 · p. 6
• Without retraining, the detector reduces ASR to 0.00 on MISHard and MM-SafetyBench for both models.The base models’ ASRs range from 23.00 to 97.00 across those benchmarks and models.Source: §3.2 · p. 6

4.1 Safety-Relevant Routing Signals Are Easier to Read Out than Control

• SSU false-refusal rate is 56.5% with image-oriented prompting, versus 6.0% with text-oriented and 4.5% with combined prompting on benign inputs.The image-oriented prompt also induces the strongest routing shifts, particularly in middle and late layers.Source: §4.1 · p. 6
• Steering scores correlate only weakly with detector coefficient magnitudes across Qwen3-VL and Kimi-VL, indicating that readout-relevant routing features differ from steering features.The analysis characterizes safety readout as relying on distributed routing patterns rather than a small set of individually safety-associated experts.Source: Figure 4 · p. 6, §4.1 · p. 6

4.2 Safety Information Is Distributed, Redundant, and Composition-Dependent

• SSU ASR stays between 2.00% and 2.50% and SUU ASR at or below 1.50% across individual layer groups, while SSS RFR falls from 14.50% early to 5.50% late.No single layer group uniformly dominates detection; all-layer features yield 5.50% SSS RFR, with ASRs of 2.00%, 0.00%, and 14.50% on SSU, SUU, and USU.Source: §4.2 · p. 6, Table 2 · p. 7
• At k = 64, random expert subsets obtain 6.2 ± 0.7% SSS RFR and 14.4±0.7% USU ASR, close to selected subsets, consistent with partially redundant routing features.The detector does not depend on a small number of uniquely critical experts.Source: Table 3 · p. 7, §4.2 · p. 7
• With SSU held out, its ASR rises from 2.00% to 32.00%; with USU held out, its ASR rises from 14.50% to 83.40%.Other unsafe compositions vary in transfer, and USU ASR also rises to 20.00–21.40% when other compositions are excluded, indicating composition-specific as well as transferable signals.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3 Safety Information Is Not Unique to Router Logits

• Router logits achieve 4.40% overall error versus 5.30% for hidden states, and 5.50% SSS RFR versus 9.50%, although safety information is also represented in hidden activations.The router-logit detector uses 6,145 parameters for Qwen3-VL and 1,729 for Kimi-VL.Source: §4.3 · pp. 7–8, Table 4 · p. 8

4.4 Safety Trade-off across Thresholds

• Increasing the threshold reduces SSS false refusals but allows more unsafe requests through, with a stronger trade-off for USU than for SSU, SUU, or UUU.At τ = 0.4, SSS RFR is 5.50% on Qwen3-VL and 12.00% on Kimi-VL; the threshold can be adjusted without retraining the model or detector.Source: Figure 6 · p. 8, §4.4 · p. 8

5 Conclusion

• The router-logit detector reads routing features across MoE layers to identify unsafe requests before generation, without modifying model parameters or routing behavior.The conclusion contrasts this readout approach with prompting, supervised fine-tuning, and expert steering, whose safety improvements are described as inconsistent.Source: §5 · p. 8
• Across two MoE VLMs, the detector substantially reduces unsafe responses across compositional risk categories and out-of-distribution benchmarks.The conclusion also reports that predictive signals are distributed across layers and routing dimensions and only weakly align with features emphasized for expert steering.Source: §5 · p. 8
• The conclusion argues that naturally emerging internal signals can support safeguards when read out by lightweight external mechanisms, even if internal representations are not reliable control handles.Source: §5 · p. 8

Limitations

• The evaluation primarily uses string matching to identify refusals and unsafe responses, which may miss nuanced cases not determined by surface patterns alone.Source: Limitations · pp. 8–9
• Safety and harmfulness judgments follow benchmark definitions and should not be treated as universally applicable across cultural, social, and geographic contexts.Source: Limitations · p. 9
• Experiments cover two MoE VLM families and evaluated dataset scales, so broader architectures, model scales, and safety datasets remain to be tested.Source: Limitations · p. 9

Ethical Statement

• The ethical statement says unsafe multimodal interactions are constructed only for research and evaluation, with experiments in controlled settings using public benchmarks.Source: Ethical Statement · p. 9
• The authors acknowledge risks from studying unsafe interactions and frame the work as supporting the characterization of MoE VLM safety risks and safeguards.Source: Ethical Statement · p. 9

A Related Work

• Prior multimodal safety work studies interaction-based risks, benchmarks, jailbreaks, and alignment; HoliSafe organizes image–text safety compositions.Source: A Related Work · p. 10
• This work introduces neither a new benchmark nor a tuning objective, instead examining MoE routing as both a control interface and diagnostic signal.Source: A Related Work · p. 11
• In multimodal MoE models, safety-associated experts offer limited selective control, while predictive safety information is distributed across layers and redundantly encoded across routing dimensions.Source: A Related Work · p. 11
• Unlike prior internal-state detectors, this work reads router logits, compares them with dense hidden-state features, and analyzes safety information across layers and routing dimensions.Source: A Related Work · p. 11

B Dataset Details

• HoliSafe labels image, text, and joint safety in five categories: SSS, SSU, SUU, USU, and UUU, with S and U denoting safe and unsafe.Source: B Dataset Details · p. 11
• Each category has 1000 training, 200 validation, and 1000 test examples; training also adds 200 VQAv2 examples, and splits support learning, calibration, and final evaluation.Source: B Dataset Details · p. 11
• Out-of-distribution safety tests use MISHard SSU and Stable Diffusion MM-SafetyBench examples, while MMMU examples assess general multimodal capabilities.Source: B Dataset Details · p. 11

C.1 Generation Details

• Generation uses temperature 1.0, top-p 0.9, and a 128-token response limit across models and evaluation settings unless specified otherwise.Source: §C.1 · p. 11

C.2 System Prompting Details

• The three fixed safety prompts assess the text alone, image alone, or joint image–text request, and remain unchanged across models and evaluation settings.Source: §C.2 · pp. 11–12

C.3 Expert Steering Details

• The method selects top-k positively scored pairs, weights their sparse prior by refusal signal, and modifies router logits during inference; α controls steering strength.Source: §C.3 · p. 12
• Steering scores layer–expert pairs using refusal association on unsafe inputs and routing perturbation on benign inputs under safety prompting.Source: §C.3 · p. 12

C.4 Supervised Fine-tuning Details

• SFT trains Text, Image, and Combined LoRA variants for three epochs on shared image–text examples, using helpful targets for benign inputs and refusal-style targets for unsafe inputs.The learning rate is 1 × 10−5, with LoRA rank 8 and α = 16.Source: §C.4 · p. 12

C.5 Router-Logit Detector Details

• The detector concatenates final-prompt-token router logits across MoE layers, trains a standardized L2-regularized logistic regression, and forces refusal when unsafe probability exceeds a threshold.Source: §C.5 · p. 13

• Refusal-associated and benign-prompt-shift signals vary substantially across layers and experts in both Qwen3-VL and Kimi-VL.Source: D Additional Results · p. 13
• Figure 7 maps the layer–expert routing signals used in the steering analysis for Qwen3-VL and Kimi-VL.Source: Figure 7 · p. 13
• The distributed safety-associated routing patterns provide a basis for constructing sparse expert-steering priors.Source: Figure 7 · p. 13, D Additional Results · p. 13

• Across seven examples spanning all five HoliSafe compositions, outcomes include benign responses, successful refusals, a false refusal, and a missed detection.Steering uses α = 5.0 for both models, and detector outputs use τ = 0.4.Source: E Qualitative Examples on HoliSafe · p. 13
• For SSS, the detector accepts a benign gardening question but unnecessarily refuses a house-identification question.Source: Figure 8 · p. 14
• In SSU and SUU examples, the detector refuses neutral-caption and disruption prompts, while some baseline captions introduce disability-related stereotypes.Source: Figure 9 · p. 14
• The USU examples pair a benchmark-label-consistent refusal with a missed unsafe image context; label agreement does not establish that every baseline response is harmful.Source: Figure 10 · p. 15
• In the UUU example, the detector refuses a workplace-harassment request.Source: Figure 11 · p. 15