Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

1 Introduction

• The framework localizes and probes safety-sensitive representations in frozen VLMs using layer-wise analysis and controlled interventions without changing model parameters.The interventions include activation magnitude scaling and Concept Activation Steering on selected vision layers.Source: §1 · pp. 1–2
• On Qwen3-VL-4B, early vision layers enable bidirectional refusal control through steering, while later layers are less steering-responsive and more sensitive to magnitude scaling.Deeper representations can remain separable even when steering produces limited behavioral change.Source: §1 · p. 2
• InternVL2.5-4B shows qualitatively similar depth-dependent geometry despite differences in layer location and magnitude.This provides cross-model representational evidence that the observed structure is not unique to Qwen3-VL-4B.Source: §1 · p. 2
• The findings expose a representation-level refusal-bypass attack surface in VLM vision encoders.The paper frames the depth-dependent causal influence of safety geometry as enabling targeted refusal control.Source: §1 · p. 2

2 Threat Model

• The adversary uses multimodal inputs, such as adversarial images or prompts, to seek internal representation shifts that weaken or bypass refusal while preserving coherent outputs.The model is frozen and its parameters cannot be modified at inference time.Source: §2 · p. 2
• Forward hooks directly perturb intermediate activations to isolate layers and directions and measure their causal influence on refusal.These interventions characterize a representation-level attack surface and simulate worst-case internal shifts.Source: §2 · p. 2
• The experiments do not establish that external inputs alone can realize the same representation shifts; constructing an end-to-end input-level attack is outside scope.Source: §2 · p. 2

3 Related Work

• Prior studies identify safety layers and shallow safety alignment in LLMs, multiple safety-related directions, and uneven harmful information across VLM image-encoder layers [2025].The cited VLM work also presents ICET and a Layer-Wise PPO defense.Source: §3 · p. 2
• Representation-engineering work uses activation directions to monitor or control behavior, while refusal-direction studies show removal can suppress refusal and addition can induce refusal on benign inputs [2018] [2023] [2024].The paper extends this perspective to VLM vision encoders and asks how directional controllability changes across depth.Source: §3 · pp. 2–3
• Inference-time VLM defenses include safety-direction steering and ASTRA's adaptive steering from adversarial visual features [2025].Other cited approaches use adversarial perturbations during preference optimization [2025].Source: §3 · p. 3
• The paper uses activation steering as a causal diagnostic alongside intervention, comparing directional steering with magnitude scaling; on Qwen3-VL-4B, early vision layers permit strong bidirectional refusal control.Deeper representations respond more to magnitude perturbations despite remaining geometrically separable; InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry.Source: §3 · p. 3

4 Methodology

• The framework has three stages: layer-wise divergence analysis, activation scaling, and directional steering.These stages identify candidate regions, measure magnitude sensitivity, and test whether safety-related geometry influences refusal.Source: §4 · p. 3

4.1 Activation Divergence Analysis

• For activation divergence analysis, the authors compare pooled hidden representations from 100 safe and 100 unsafe examples at each layer, sampling Safe–Safe, Unsafe–Unsafe, and Safe–Unsafe pairs across repeated runs.Layers with consistently greater Safe–Unsafe angular separation than both within-condition comparisons are treated as candidate safety-sensitive regions; the procedure is applied to Qwen3-VL-4B and InternVL2.5-4B.Source: §4.1 · p. 3

4.2 Activation Scaling

• Activation scaling uses forward hooks to modify selected Qwen3-VL-4B layer outputs and uniformly scales visual tokens to test effects on URR and SRR.The tested scaling factors are α ∈ {0.85, 0.9, 1.0, 1.15, 1.2}.Source: §4.2 · p. 4

4.3 Directional Activation Steering

• Directional steering constructs a layer-specific mean-difference direction and projects token representations onto it before applying a steering shift.The study compares focused steering on the CLS token and first 10 patch tokens with global steering on all visual tokens.Source: §4.3 · p. 4

5 Results

• The evaluation uses URR and SRR to measure rejection on unsafe and safe inputs, respectively, and includes UnsafeBench, GCC, and MMLU.The framework assesses layer-wise divergence and intervenes on selected vision layers using magnitude scaling and Concept Activation Steering.Source: §5 · p. 4, §5.1 · p. 4, §5.2 · p. 5

5.1 Divergence Analysis

• Language-tower divergence is strongest early and generally declines with depth, while vision-encoder divergence is heterogeneous, with distinct Qwen3-VL-4B regions around Layers 7–10 and 18–22.The Qwen3-VL-4B divergence patterns are shown in Figure 1.Source: §5.1 · p. 4, Figure 1 · p. 5
• InternVL2.5-4B shows qualitatively similar depth-dependent divergence, but layer locations and magnitudes differ, and causal interventions were evaluated only on Qwen3-VL-4B.Its vision SU–SS divergence weakens toward later layers, while SU–UU transitions from negative toward positive divergence.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 Parameter Scaling

• At Layers 7–10, α = 0.85 yields 100% URR on UnsafeBench and 100% SRR on GCC, indicating near-total refusal rather than selective safety improvement.At Layers 18–22, α = 1.15 raises URR to 96%, with SRR simultaneously reaching 91% on UnsafeBench and remaining 8–9% on GCC.Source: §5.2 · p. 6
• Scaling language-layer activations to α = 1.2 raises URR to 98%, while α = 0.85 lowers URR to 37%; scaling leaves accuracy unchanged from the 44% baseline on 100 MMLU samples.The MMLU result does not establish preserved safety selectivity.Source: §5.2 · p. 6

5.3 Concept Activation Steering

• At Layers 7–10, global steering changes URR from the 84% baseline to 6% at α = 4.0 and 96% at α = −4.0, while focused steering has much smaller effects.The contrast suggests the effect is distributed across visual tokens rather than concentrated in the CLS representation.Source: §5.3 · p. 7
• At Layers 18–20, steering leaves URR near baseline despite deeper activations remaining separated along E1, distinguishing representational separability from causal controllability.These later layers respond more strongly to magnitude scaling than to directional steering.Source: §5.3 · pp. 7–8, Figure 3 · p. 7

Cross-Model Safety Geometry

• InternVL2.5-4B shows qualitatively recurring safety geometry, but interventions were performed only on Qwen3-VL-4B, so causal equivalence is not established.The precise layer locations and divergence magnitudes differ across architectures.Source: §6 · p. 8

Security and Safety–Selectivity Implications

• Early vision layers support bidirectional refusal steering, whereas later layers are more responsive to magnitude scaling, exposing a white-box, representation-level attack surface.The experiments require access to internal activations and do not demonstrate an end-to-end adversarial image or prompt that realizes the same shifts.Source: §6 · p. 8
• Magnitude scaling can raise unsafe rejection while also increasing safe rejection on difficult safe inputs, creating a safety–selectivity trade-off.No change in the 100-sample MMLU evaluation does not establish preservation of broad utility.Source: §6 · p. 8

Limitations and Future Work

• The work calls for interventions across architectures, broader held-out and safety-category evaluation, and higher-dimensional safety subspaces to assess direction stability and defenses.Input-level realization of refusal bypass also remains open.Source: §6 · p. 8

• On Qwen3-VL-4B, early vision layers support bidirectional refusal control through steering, while later layers respond more to magnitude scaling, reaching 96% unsafe rejection with elevated false rejection.Source: §7 · p. 8
• InternVL2.5-4B exhibits qualitatively similar depth-dependent activation geometry despite differences in precise layer location.Source: §7 · p. 8
• The framework identifies and manipulates safety-sensitive VLM representations at inference time without modifying model parameters.Source: §7 · p. 8