PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Textual Refusal Directions for Multimodal Safety

Moreno D'Incà, Nicu Sebe, Massimiliano Mancini

arXiv:2606.31876v3cs.AIcs.CVcs.LG

TL;DR

Unsafe multimodal safety data is difficult to collect, must be gathered anew across modalities, and raises ethical and legal concerns. The paper introduces MARS, which uses textual refusal directions for training-free multimodal steering; experiments report safety gains across modalities, including video jailbreaks.

  • Problem

    Unsafe multimodal training data is difficult to collect for each modality and raises ethical and legal concerns, despite the need for safety alignment in MLLMs.

    Source: §1 · p. 1

  • Method

    MARS re-centers activations, adaptively scales steering within a geometrically defined trust region, and selects an intervention layer to use textual refusal directions without multimodal safety data.

    Source: §1 · p. 2, §4.2 · p. 6, §7 · p. 10

  • Results

    Across five models and seven benchmarks spanning text, images, and videos, MARS improves safety without multimodal supervision or retraining; QWEN3-VL gains +59.4% refusal on video jailbreaking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Textual refusal directions transfer across modalities, suggesting MLLMs share safety-relevant structure that can be used without multimodal safety data.

    Source: §1 · p. 2

    MARS assumes refusal directions are present in the LLM backbone, that modality misalignment can be mitigated linearly by centering, and retains residual over-refusal on safe inputs.

    Source: §7 · p. 10

Abstract

from arXiv · show

Safety in Multimodal Large Language Models (MLLMs) depends on unsafe multimodal data for post-training alignment or activation steering. However, such data is hard to collect, must be gathered anew for each modality, and its collection and distribution raise ethical and legal concerns. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a lightweight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak, MARS achieves consistent safety gains while preserving utility up to over-refusal. MARS reveals that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.

1 Introduction

Multimodal safety alignment is constrained by the difficulty and ethical concerns of collecting unsafe multimodal data. This paper tests whether textual refusal directions transfer across modalities and introduces MARS to use them for training-free safety steering.

1 Introduction

  • Unsafe multimodal training data is difficult to collect for each modality and raises ethical and legal concerns, motivating safety methods that do not require it.Such data must capture meaningful cross-modal interactions, and vulnerabilities of post-training strategies may be amplified in multimodal settings.Source: §1 · p. 1
  • Textual refusal directions generalize to multimodal inputs, suggesting that safety-relevant structure is shared across modalities.The paper projects activations onto a refusal direction to estimate implicit safety belief and steer activations toward refusal.Source: §1 · p. 2
  • Transfer is not universal: steering efficacy depends on layer and strength, while modality misalignment can spuriously steer safe inputs toward refusal.Modality-specific components may align with the refusal direction regardless of input safety.Source: §1 · p. 2
  • Across five models and seven benchmarks, MARS improves safety across text, image, and video without multimodal supervision or retraining.On QWEN3-VL video jailbreaking, it yields +59.4% refusal.Source: §1 · p. 2
  • MARS addresses these issues through activation re-centering, dynamic steering, and layer-importance estimation.It is a lightweight, training-free approach.Source: §1 · p. 2

2 Preliminaries

The preliminaries define MLLM safety as refusing unsafe queries while correctly answering safe ones, then introduce refusal directions and activation steering as the paper’s setup.

2.1 Problem formulation

  • The paper defines safety as refusing unsafe queries while correctly answering safe ones.Its ideal model minimizes refusal on safe queries and maximizes it on unsafe queries.Source: §2.1 · pp. 2–3
  • An MLLM maps text and visual inputs to textual output through a vision encoder, projection module, and decoder-only LLM.Source: §2.1 · p. 2
  • Increasing α improves refusal on unsafe inputs but induces over-refusal on safe inputs and degrades utility.Source: Figure 1 · p. 3

2.2 Re-purposing refusal directions for safety

  • Refusal directions are extracted by contrasting hidden states associated with acceptance and refusal across decoder layers.The described activations are taken at the assistant-token position.Source: §2.2 · p. 3
  • Steering adds a scaled component along the normalized refusal direction to amplify refusal when the input aligns with it.The steering strength is controlled by α.Source: §2.2 · p. 3

3 Do textual refusal directions transfer to images?

Textual refusal directions transfer to image inputs, but their effectiveness depends on layer and steering strength. Modality-specific activation alignment can blur safety distinctions; centering with neutral images recovers safety-relevant structure.

3 Do textual refusal directions transfer to images?

  • Textual refusal directions transfer to image inputs, but layer-specific peaks can also over-refuse safe queries, including 96.4% at GEMMA3 layer 10.Unsafe refusal peaks at GEMMA3 layer 31, QWEN3-VL layer 16, and INTERNVL 3.5 layer 19.Source: §3 · pp. 3–4
  • Increasing steering strength improves unsafe-input refusal but reduces safe-input acceptance and utility, with GEMMA3 safety already degrading at α=2.0.The effect is evaluated at the layer with the largest safe/unsafe refusal gap.Source: §3 · p. 4

3.1 Investigating failure modes: the activation space misalignment

  • Raw image activations align with the refusal direction regardless of safety semantics, entangling modality-specific and safety-relevant features.The raw-space separation score is low across QWEN3-VL, GEMMA3, and INTERNVL 3.5; GEMMA3 shows the strongest misalignment.Source: §3.1 · pp. 4–5
  • Centering with neutral, randomly colored images shifts unsafe activations toward rejection and safe activations toward acceptance, increasing separation across layers and models.These images capture visual structure without semantic meaning or safety-related content.Source: Figure 2 · p. 4, §3.1 · p. 5

4 Modality-Agnostic Refusal Steering (MARS)

MARS steers multimodal activations at inference time using centered activations and a textual refusal direction, without additional training or multimodal safety data. It gates updates to avoid intervening on inputs estimated as safe, adapts their strength within a trust region, and selects a layer using text-based scores.

4 Modality-Agnostic Refusal Steering (MARS)

  • MARS confines inference-time steering to estimated unsafe inputs and the local activation geometry, without training or additional forward passes.The method addresses steering strength and layer selection using centered activations.Source: §4 · p. 5

4.1 ReLU-Gated Traversal

  • ReLU-gated traversal gives no update to activations negatively aligned with the refusal direction, preserving the model’s original behavior on inputs estimated as safe.The method uses activation alignment rather than an explicit classification step.Source: §4.1 · p. 5

4.2 Adaptive Steering Strength

  • MARS derives steering strength from the distance to the nearest accept or reject centroid and constrains the update within that local trust region.The steered activation is applied to all token positions, with steering performed only on the first generated token during decoding.Source: §4.2 · p. 6

4.3 Layer Selection

  • Layer selection combines direction consistency, context separability, and safe-text margin scores, then selects the layer with the highest summed normalized score.The scoring procedure uses textual prompts and safe validation text, without multimodal safety data or safety-data hyperparameter tuning.Source: §4.3 · p. 6

5 Experiments

MARS improves refusal across image, cross-modal, and video safety evaluations while generally preserving utility; ablations identify layer selection, centering, and ReLU gating as important to that trade-off.

5.1 Experimental protocol

  • The evaluation covers five MLLMs, image and video safety, cross-modal interactions, utility, and video jailbreaks, using refusal rate, safety score, and benchmark accuracy.The protocol compares four baselines: AdaSteer, ECSO, SASA, and AutoSteer; datasets include VISU, HADES, SIUO, HOLISAFE, MMMU, MMMUPRO, and VIDEOSAFETYBENCH.Source: §5 · p. 6, §5.1 · p. 7

5.2 Safety on Images

  • MARS improves image refusal across all models, from +25.1% on GEMMA3 to +67.9% on MOLMO2, with controlled over-refusal.It outperforms AutoSteer on refusal across all models and beats SASA on refusal with lower over-refusal on MOLMO2 and QWEN3-VL.Source: Table 1 · p. 7, §5.2 · p. 8
  • On HADES image jailbreaks, MARS restores refusal across models, reaching +78.6% on MOLMO2 and matching SASA on three models.ECSO’s self-assessment omits images, leaving it blind to visually carried jailbreak signals.Source: Table 2 · p. 7, §5.2 · p. 8
  • For unsafe requests formed by individually safe text and images, MARS improves refusal across both SIUO and HOLISAFE, with gains up to +58% on MOLMO2 on SIUO.It outperforms training-free baselines and remains competitive with training-based methods.Source: Table 3 · p. 8, §5.2 · p. 8
  • MARS generally preserves utility on MMMU and MMMUPRO, matching zero-shot on four models while QWEN3.5 shows modest over-refusal.It yields the best safety-utility trade-off overall; SASA degrades accuracy by -10% on INTERNVL 3.5 and -20% on QWEN3.5.Source: §5.2 · pp. 8–9, Table 4 · p. 9
  • Removing layer selection collapses unsafe refusal on MOLMO2 and GEMMA3; removing centering causes 99.2% over-refusal on safe MOLMO2 inputs.Removing ReLU gating also harms utility, while the full ablation concludes that layer selection preserves integrity, centering isolates refusal, and gating limits steering to unsafe activations.Source: Table 5 · p. 9, §5.2 · p. 9

5.3 Safety on Videos

  • +59.4% refusal on QWEN3-VL; MARS achieves the best video-jailbreak refusal among training-free methods and transfers textual refusal directions without video data.The video evaluation also reports gains of +52% on QWEN3.5 and +39% on MOLMO2; Fig. 3 shows MARS steering MOLMO2 toward a safe answer.Source: §5.3 · p. 9

6 Related Work

Prior work uses activation steering for controllable behavior and has explored safety interventions in language and multimodal models. MARS complements these approaches by transferring text-only refusal directions to multimodal refusal.

6 Related Work

  • Activation steering has been used to control behaviors and probe latent knowledge, including safety-related interventions and textual refusal directions [1] [6] [18] [37] [48] [15] [32].Prior findings include distinct subspaces for harmfulness and refusal [46], and AdaSteer [48] applies activation steering to improve safety.Source: §6 · pp. 9–10
  • Multimodal safety methods include training on multimodal data [7] [19], activation-based interventions [38] [40] [43], and ECSO’s iterative self-assessment [12].These approaches respectively require multimodal data, train an activation classifier or prober, or use multiple forward passes.Source: §6 · p. 10
  • MARS takes an orthogonal approach, using text-only safety data to transfer refusal directions to multimodal inputs without multimodal safety data [1].The paper positions AutoSteer [43] and SASA [40] as data-dependent training baselines, and ECSO [12] as a data-free comparison.Source: §6 · p. 10

7 Conclusion

The conclusion presents MARS as a training-free way to transfer textual refusal directions across image and video inputs, while identifying assumptions and residual over-refusal as limitations. The appendix provides further results and analyses.

7 Conclusion

  • MARS re-centers activations, adaptively scales steering within a trust region, and selects an intervention layer to transfer textual refusal directions across images and videos.Across multiple MLLM families and benchmarks, the paper reports consistent safety gains, often matching or surpassing training-based approaches, with negligible computational overhead.Source: §7 · p. 10
  • MARS assumes refusal directions exist in the backbone and that centering can linearly mitigate modality misalignment; residual over-refusal on safe inputs remains.Performance may degrade when refusal signals are weak or absent, and linear mitigation may not hold in highly nonlinear activation spaces.Source: §7 · p. 10

Appendix

  • The appendix reports additional jailbreak and cross-modal results, utility results, ablations, implementation details, activation-space analyses, and weak-alignment evaluation.It also discusses potential positive and negative societal impacts and provides additional quantitative and qualitative results.Source: Appendix · p. 14

A Additional Safety Results

Across multimodal jailbreak benchmarks, MARS improves refusal, including where neutral prompts or unsafe visual content challenge existing defenses. Its gains are consistent across several settings, though not every model benefits equally.

A Additional Safety Results

  • MARS substantially improves refusal across all models on MMSAFETYBENCH, where neutral prompts paired with unsafe images create multimodal jailbreaks.The benchmark contains 6,720 samples, including 5,040 image–text pairs and 1,680 text-only samples.Source: Table 7 · p. 14, A Additional Safety Results · p. 14
  • On HADES, neutral jailbreaking prompts lower refusal by 31.2% on QWEN3.5 to 72.2% on MOLMO2, while MARS consistently recovers refusal.The neutral prompt is more effective than the benchmark’s original unsafe prompt at suppressing refusal.Source: Table 8 · p. 14, A Additional Safety Results · p. 14
  • MARS improves refusal in both VLGUARD settings and outperforms training-based AutoSteer on every reported setting.VLGUARD contrasts unsafe text paired with safe images (US) and unsafe images (UU); MARS also outperforms AutoSteer on all models except GEMMA3 on HOLISAFE.Source: Table 9 · p. 15, A Additional Safety Results · p. 15, Table 10 · p. 15
  • On HOLISAFE, MARS improves refusal by 43.3% on QWEN3-VL and 42.6% on MOLMO2, but all training-free methods struggle on GEMMA3.MARS outperforms AutoSteer on every model except GEMMA3 and outperforms SASA on MOLMO2.Source: Table 10 · p. 15, A Additional Safety Results · p. 15

B Additional Utility Results

MARS largely preserves visual-question answering accuracy on MMSTAR, with over-refusal reducing accuracy on QWEN3.5.

B Additional Utility Results

C Additional Ablation Studies

Additional studies examine MARS’s component ablations, implementation, activation-space analysis, and societal implications. They report that centering and layer selection matter for utility and over-refusal, while neutral-image estimates remain stable across sample sizes.

C Additional Ablation Studies

  • Removing layer selection severely degrades utility, while removing centering can cause safe inputs to be spuriously steered toward refusal.Removing ReLU gating slightly reduces utility; individual layer-score components affect outcomes when their removal changes the selected layer.Source: C Additional Ablation Studies · p. 16, Table 12 · p. 17
  • Neutral-image mean estimates remain stable across sample sizes, but omitting centering can degrade utility or cause over-refusal.Safety drops are minor on MOLMO2 at n=8 and n=24; randomly colored images serve as an annotation-free proxy for modality bias.Source: C Additional Ablation Studies · p. 16, Table 13 · p. 17

D Additional Details

  • MARS runs at approximately the models’ original inference time, with reported experiments taking 30 minutes to a few hours.Experiments use deterministic decoding and run on a single NVIDIA A100 GPU with FP16 automatic mixed precision.Source: D Additional Details · p. 17

E Additional Activation Space Analysis

  • The activation-space analysis identifies safe images being projected toward the refusal subspace, leading to over-refusal.The paper reports further qualitative and quantitative analysis in Figures 6–9.Source: E Additional Activation Space Analysis · p. 17

F Societal Impact

  • MARS is presented as a lightweight post-hoc intervention that uses text-based safety data rather than labeled multimodal examples or retraining.The authors say this reduces annotation burden and exposure to disturbing content, and extends to video; Figure 5 reports layer-selection scores across models.Source: F Societal Impact · p. 18, Figure 5 · p. 19
  • The authors flag dual-use risk: knowledge of activation steering could motivate attacks that steer models toward acceptance regardless of safety.They note such attacks are already known and studied [1] [30].Source: F Societal Impact · p. 18

G Robustness to Weak Textual Refusal

MARS improves refusal on HADES and VISU for LLAVA 1.5 despite the model’s weak textual refusal signal. The results support the presence of latent safety representations even in weakly aligned or outdated models.

G Robustness to Weak Textual Refusal

H Additional Results

Additional qualitative video examples contrast zero-shot responses that provide harmful instructions with MARS refusals or safer redirections. The examples cover animal abuse, child endangerment, violence, and political content.

H Additional Results

  • The additional qualitative VIDEOSAFETYBENCH [24] figures cover MOLMO2 8B, QWEN3-VL 8B, QWEN3.5 9B, and INTERNVL 3.5 8B.Source: H Additional Results · p. 19, Figure 10 · p. 21, Figure 13 · p. 24
  • For requests to replicate hunting a bird or uncomfortable workplace behavior, MARS refuses, whereas zero-shot responses provide instructions or scene-setting advice.Source: H Additional Results · pp. 22, 26
  • MARS refuses or redirects requests about mistreating a cat, restraining it with a chain, or killing a fish, unlike zero-shot responses that give instructions.These examples are among the additional VIDEOSAFETYBENCH [24] results.Source: H Additional Results · pp. 19, 22, 23
  • MARS declines to infer accusations from a staged political scene and redirects protest-planning guidance toward peaceful, lawful alternatives.The corresponding zero-shot responses infer possible corruption and outline protest-organizing steps.Source: H Additional Results · pp. 24–25
  • For requests involving children, MARS refuses to help publish adult-child themes or create a scene of a trapped child, while zero-shot responses offer guidance.MARS also refuses a request about recreating a threatening scene at a religious gathering.Source: H Additional Results · pp. 24, 27, 29

Related papers

Loading 2606.31876v3…